Video transmission method, related equipment, system and storage medium

Abstract

The invention discloses a video transmission method, related equipment, a video transmission system and a storage medium, the video transmission method is applied to video sending equipment, and the video transmission method comprises the following steps: the video sending equipment comprises multiple groups of video data acquired by different video acquisition components; determining a sending sequence of each group of video data based on a data transmission time delay condition between the video sending equipment and the video receiving equipment; and multiplexing a communication link between the video sending device and the video receiving device according to the sending sequence, and sending each group of video data to the video receiving device in sequence. According to the scheme, the synchronization effect of the video receiving equipment from the video data corresponding to each video acquisition component can be improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a video transmission method, related equipment, system and storage medium. Background Art

[0002] At present, the main application scenario of multi-eye wireless cameras on the market is generally that the multi-eye wireless video acquisition device is connected to the public cloud platform via Wi-Fi, and the mobile terminal can access the cloud platform via Wi-Fi or mobile network to view the real-time video and recordings collected by the multi-eye wireless video acquisition device in real time.

[0003] by Figure 1 Taking the video transmission of a binocular camera as an example, currently the video data in different video acquisition components of the binocular camera are first transmitted to the public cloud platform through communication link 1 and communication link 2 respectively, and then the public cloud platform uses communication link 3 and communication link 4 to send the video data collected by different video acquisition components to the terminal device. After the video data collected by different video acquisition components pass through different network transmission paths, the time sequence of multiple videos arriving at the mobile terminal may be inconsistent, which is prone to the problem of asynchronous video playback. Summary of the invention

[0004] The present application at least provides a video transmission method, related equipment, system and storage medium.

[0005] The present application provides a video transmission method, which is applied to a video sending device. The video transmission method includes: the video sending device includes multiple groups of video data collected by different video acquisition components; based on the data transmission delay between the video sending device and the video receiving device, the sending order of each group of video data is determined; according to the sending order, the communication link between the video sending device and the video receiving device is multiplexed, and each group of video data is sent to the video receiving device in sequence.

[0006] The present application provides a video transmission method, which is applied to a video receiving device. The video transmission method includes: the video receiving device receives multiple groups of video data sent by a video sending device based on the above-mentioned video transmission method; and caches each received video data.

[0007] The present application provides a video transmission device, including: a video acquisition module, a sequence determination module and a data sending module; the video acquisition module is used in a video sending device to include multiple groups of video data collected by different video acquisition components; the sequence determination module is used to determine the sending order of each group of video data based on the data transmission delay between the video sending device and the video receiving device; the data sending module is used to multiplex the communication link between the video sending device and the video receiving device according to the sending order, and send each group of video data to the video receiving device in sequence.

[0008] The present application provides a video transmission device, including: a data receiving module and a cache module; the data receiving module is used for a video receiving device to receive multiple groups of video data sent by a video sending device based on the above-mentioned video transmission method; and the cache module is used to cache each received video data.

[0009] The present application provides a video sending device, including a first communication module, a first memory and a first processor, the first processor is used to execute program instructions stored in the first memory to determine the sending order of each group of video data, the first communication module is used to send each group of video data to a video receiving device, wherein the first communication module, the first memory and the first processor cooperate to implement the above-mentioned first video transmission method.

[0010] The present application provides a video receiving device, including a second communication module, a second memory and a second processor, wherein the second communication module is used to receive multiple groups of video data sent by a video sending device, and the second processor is used to execute program instructions stored in the second memory and cache the received video data, wherein the second communication module, the second memory and the second processor cooperate to implement the above-mentioned second video transmission method.

[0011] The present application provides a video transmission system, which includes the video sending device and the video receiving device as described above, and the video sending device and the video receiving device are communicatively connected.

[0012] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, any of the above-mentioned video transmission methods is implemented.

[0013] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0016] Figure 1 is a schematic diagram of a conventional multi-channel video transmission method provided by the present application;

[0017] Figure 2 It is a flowchart of an embodiment of a video transmission method of the present application applied to a video sending device;

[0018] Figure 3 is another flow chart of an embodiment of a video transmission method of the present application applied to a video sending device;

[0019] Figure 4 yes Figure 2 Schematic diagram of the sub-process of step S12;

[0020] Figure 5 It is a schematic diagram of the structure of a data frame sent by a video sending device to a video receiving device of the present application;

[0021] Figure 6 It is a flowchart of an embodiment of a video transmission method of the present application applied to a video receiving device;

[0022] Figure 7 It is a schematic diagram of the structure of the data frame fed back by the video receiving device of the present application to the video sending device;

[0023] Figure 8 It is a schematic diagram of video transmission between multiple devices of the present application;

[0024] Fig. 9 This is another flowchart of an embodiment of the video transmission method of the present application;

[0025] Fig.10 It is a structural schematic diagram of an embodiment of a video transmission device of the present application;

[0026] Fig.11 is a structural schematic diagram of another video transmission device according to an embodiment of the present application;

[0027] Fig.12 It is a structural diagram of an embodiment of a video sending device of the present application;

[0028] Fig.13 It is a structural diagram of an embodiment of a video receiving device of the present application;

[0029] Fig.14 It is a structural diagram of an embodiment of a video transmission system of the present application;

[0030] Fig.15 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0031] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0032] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0033] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] The present application provides a video transmission method. In the present application, the execution subject for implementing the video transmission method described in the present application may be a video transmission device, a communication terminal, etc. For example, the video transmission device may be set in a communication terminal or a server or other processing device, wherein the terminal device may be an electronic device, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the video transmission method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0035] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a video transmission method of the present application applied to a video sending device. The video transmission method provided in this embodiment is applied to a video sending device. Figure 2 As shown, the video transmission method provided by the embodiment of the present disclosure may include the following steps:

[0036] Step S11: the video sending device includes multiple groups of video data collected by different video collection components.

[0037] In some application scenarios, the video sending device may be a video acquisition device with different video acquisition components, and the video receiving device may be a communication terminal or a cloud platform that establishes a communication connection with the video acquisition device. In some application scenarios, the video sending device may be a cloud platform or a communication terminal that caches multiple groups of video data sent by a video acquisition device with different video acquisition components, and the video receiving device may be another cloud platform or another communication terminal connected to the cloud platform or the communication terminal. That is, the video transmission method between the video acquisition device and the cloud platform may be the same as the video transmission method between the cloud platform and the communication terminal. The embodiment of the present application takes one of the stages as an example, and the other stage is executed by reference.

[0038] Exemplarily, the video sending device may be a multi-eye camera, and the different video acquisition components may be different camera components in the multi-eye camera, and each camera component acquires a set of video data. For example, the multi-eye camera is a binocular camera, and there are two camera components in the binocular camera, and the two camera components can acquire two sets of video data. The number of video data contained in each set of video data may be one or more, which is not specifically limited here.

[0039] Step S12: Based on the data transmission delay between the video sending device and the video receiving device, determine the sending order of each group of video data.

[0040] In some application scenarios, the data transmission delay situation can be based on the mean value of the transmission delay of each group of video data transmitted by the video sending device and the video receiving device within a preset historical time period, or other types of mathematical statistics. The transmission order of video data of different data sizes can be dynamically adjusted according to the data delay situation. For example, if the current data transmission delay is large, it can be chosen to give priority to sending the video data with larger data in each group of video data. Or in order to ensure that the video receiving device can understand the situation of the captured scene in a timely manner, it can be chosen to give priority to transmitting video data with smaller data packets. That is, the data volume of each video data in different groups of video data may be different.

[0041] In some application scenarios, the corresponding transmission delays of the video data corresponding to different video acquisition components can be determined and counted to obtain multiple data transmission delay situations, thereby dynamically adjusting the sending order between different groups of video data. Exemplarily, the video acquisition components include video acquisition component a and video acquisition component b. Video acquisition component a sends i video data within a preset historical time period, and video acquisition component b sends j video data within a preset time period. Then, a data transmission delay situation about video acquisition component a can be determined based on the transmission delay of i video data, and a data transmission delay situation about video acquisition component b can be determined based on the transmission delay of j video data. Then, the sending order relationship between the video data collected by video acquisition component a and video acquisition component b is dynamically adjusted based on the data transmission delay of video acquisition component a and the data transmission delay of video acquisition component b.

[0042] Step S13: multiplexing the communication link between the video sending device and the video receiving device according to the sending order, and sending each set of video data to the video receiving device in sequence.

[0043] Exemplarily, each group of video data may be sent in sequence using the communication link in the order of each group of video data. After receiving each group of video data from the communication link, the video receiving device may cache the video data collected by different video collection components respectively.

[0044] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0045] In some embodiments, the video transmission method further comprises the following steps:

[0046] Receive the video delay fed back by the video receiving device. The video delay is used to characterize the synchronization of the video data sent by different video stream channels received by the video receiving device. Exemplarily, the video delay can be the timestamps between the timestamps of the videos. Exemplarily, the timestamp of the first video data received by the video receiving device is 11:20, and the timestamp of the second video data is 11:10, then the video delay between the first video data and the second video data is 10 seconds. If the first video data and the second video data are displayed directly on the display interface of the video receiving device, the displayed screen will be seriously out of sync, and the time difference between the two screens may even reach 10 seconds.

[0047] In some application scenarios, in response to the video delay being greater than or equal to the preset delay, the step of determining the sending order of each group of video data based on the data transmission delay between the video sending device and the video receiving device is performed. If the video delay between different groups of video data is large, it means that the video is seriously out of sync and the order of video data transmission needs to be adjusted. In other application scenarios, in response to the video delay being less than the preset delay, the sending order of each video data between the current video stream channels is maintained. If the video delay is small, it means that the video data are relatively synchronized and there is no need to adjust the sending order. The current video sending order can continue to be used to continue to transmit the video data collected by each video acquisition component in the video sending device.

[0048] In some embodiments, different groups of video data are in different video stream channels in the video sending device. Figure 3 The video transmission method may further include the following steps:

[0049] Step S21: sending first handshake information to the video receiving device, so that the video receiving device determines second handshake information based on the first handshake information.

[0050] The first handshake information includes the first identification information of each video stream channel in the video sending device that can be used to send video data, and the second handshake information includes the second identification information of each video stream channel in the video receiving device that can be used to receive video data. Exemplarily, the first handshake information includes a list of video stream channel identifications supported by the video sending device, and the second handshake information includes a list of video stream channel identifications supported by the video receiving device. In some application scenarios, the video sending device sends the first handshake information by means of reliable UDP, and the first handshake information may also include at least one of the ability to inform the video receiving device that the local end supports a multi-stream multiplexing communication link, the transmission delay of each video stream channel, and a preset maximum delay difference (synchronization delay threshold). The transmission delay of each video stream channel can be determined based on the transmission status of other handshake information sent between the video sending device and the video receiving device. The transmission delay can be used to determine the sending order of each video stream channel to send the first batch of video data.

[0051] Step S22: receiving the second handshake information sent by the video receiving device.

[0052] Step S23: Based on the second identification information, determine the video stream channel used by the video sending device to send each video data.

[0053] Exemplarily, the video stream channel for sending each video data can be determined by selecting the same identifier in the first handshake information and the second handshake information. That is, multiple video stream channels can be created from the video stream identifier list of the local end to send the video. In addition, the video receiving device can also create multiple video stream channels for receiving each group of video data.

[0054] In some embodiments, see Figure 4 , the above step S12 may include the following steps:

[0055] Step S121: obtaining the transmission delay of video data sent by different video stream channels through the communication link within a preset historical time period.

[0056] Optionally, step S121 may be implemented by feeding back a reception timestamp after receiving the video data sent by the video stream channel according to the video receiving device, and then the video sending device determines the transmission delay according to the transmission timestamp and the reception timestamp of each video data. The transmission delays of the video data in different video stream channels are calculated independently, that is, if there are x video stream channels in the video sending device, x transmission delays are calculated. Specifically, an average value or other mathematical statistical value may be determined based on the transmission delay of each video data sent by each video stream channel within a preset historical time as the transmission delay of each video stream channel.

[0057] Step S122: Based on the transmission delay of each video stream channel, determine the sending order of each video data between the video stream channels.

[0058] The video data in the video stream channel with high transmission delay is sent before the video data in the video stream channel with low transmission delay. That is, the video data in the video stream channel with large transmission delay can be sent first. The video data between different video stream channels can be sent alternately, or after sending multiple video data in one video stream channel, the video data in another video stream channel can be sent.

[0059] Optionally, the video transmission method further includes the following steps:

[0060] Obtain the delay difference between the transmission delays of each video stream channel. The delay difference greater than or equal to the preset maximum delay difference is the target delay difference. In response to the existence of the target delay difference, suspend the transmission of video data in the video stream channel corresponding to the smaller delay difference in the target delay difference within a preset time period. Optionally, in response to the absence of the target delay difference, execute step S122.

[0061] Because the communication link may be subject to different external interference at different times, the transmission delays generated by transmitting video data in different video stream channels at different times may also be different due to different interference. In some application scenarios, if the transmission delay difference in different video stream channels is too large, the transmission of video data in the video stream channel with a smaller transmission delay can be suspended, and the video data in the video stream channel with a larger transmission delay can be sent intensively. Exemplarily, the pause time can be pre-set, or it can be determined based on the size relationship between the delay difference between the transmission delays of each video stream channel and the preset maximum delay difference. Exemplarily, if the pause time corresponding to the preset maximum delay difference is t seconds, and the transmission delay difference of each video stream channel is z times the preset maximum delay difference, the pause time can be t / z seconds.

[0062] Optionally, the video transmission method further comprises the following steps: determining the amount of video data to be transmitted each time in each video stream channel based on the transmission delay of each video stream channel. For each video stream channel, determining the secondary transmission order between each video data in the video stream channel.

[0063] Optionally, the amount of video data that needs to be sent each time in a video stream channel with a larger transmission delay is greater than the amount of video data that needs to be sent each time in a video stream channel with a smaller transmission delay, and the larger and smaller may be the relative size relationship of the transmission delay between the two video stream channels. Exemplarily, the amount of video data that needs to be sent each time in each channel can be determined based on the delay difference between the transmission delays of each video stream channel. For example, if the transmission delay of the first video stream channel is larger and much larger than the transmission delay of the second video stream channel, you can choose to send multiple video data in the first video stream channel at one time, and then send the video data in the second video stream channel. The secondary sending order of each video data in each video stream channel is determined separately, and the secondary sending order is also the sending order between different video data in the same video channel. Exemplarily, please refer to Figure 5 The video data encapsulation format in the nth video stream channel in the video sending device shown in FIG. Figure 5 As shown, each video data includes a connector, a header of video stream channel n, and a payload of video stream channel n. Video data of different video stream channels can be distinguished according to the connector, and the headers of different video stream channels n can distinguish different video data in video stream channel n. Among them, two sequence numbers can be encapsulated in the encapsulation format. The sequence number in the connector is relative to the order of other video stream channels, and the sequence number in the header of video stream channel n is relative to the order of other video data in the video stream channel. The video data in different video stream channels are sent alternately, so the corresponding sequence number needs to be set in the connector, and the data of a single video stream channel also needs to be sent in sequence, so the second sequence number can be set in the header.

[0064] In some application scenarios, the order of the connectors in the video data that needs to be sent each time in the same video stream channel can be the same. For example, in a certain video stream channel, two video data need to be sent at one time. The order of one video data is r_m, and the order of the other video data is r_m+1, where r represents the sequence number in the connector, and m and m+1 represent the sequence numbers in the header, the same below. In another video stream channel, two video data need to be sent this time, one of which is sent in the order of r+1_t, and the other is sent in the order of r+1_t+1. The order of sending these four videos is r_m, r_m+1, r+1_t, and then r+1_t+1.

[0065] In some application scenarios, for the connector, the video data sent by different video stream channels need to be numbered consecutively, and the serial numbers in the connectors of different video data are different. For example, there are 2 video data to be sent in a certain video stream channel, one of which has the order r_m, and the other has the order r+1_m+1. There are 2 video data to be sent in another video stream channel, one of which has the order r+2_t, and the other has the order r+3_t+1. The order of sending these four videos is r_m, r+1_m+1, r+2_t, and then r+3_t+1.

[0066] Optionally, the video transmission method further comprises the following steps: in response to receiving packet loss feedback information sent by the video receiving device, determining the data to be retransmitted and the target video stream channel corresponding to the data to be retransmitted. Then, the sending order of the data to be retransmitted in the target video stream channel is placed before other video data in the target video stream channel, and the other video data is non-retransmitted data.

[0067] That is, if the packet loss feedback information sent by the video receiving device is received, the video data to be retransmitted and the video stream channel to which the video data belongs can be determined based on the packet loss feedback information. The priority of sending the data to be retransmitted in the video stream channel is higher than the video data transmitted for the first time, that is, if there is data to be retransmitted in a video stream channel, the data to be retransmitted will be transmitted first.

[0068] In some embodiments, before executing the above step S13, the following steps may also be performed:

[0069] The network quality information of the communication link is obtained. Based on the network quality information, it is determined whether the communication link allows data to be sent. Then, in response to the communication link allowing data to be sent, the communication link between the video sending device and the video receiving device is multiplexed in a sending order, and each group of video data is sent to the video receiving device in sequence.

[0070] The network quality information of the communication link may include, but is not limited to, congestion window information, network RTT information, and / or packet loss rate. If the network quality information shows that the network quality of the communication link is poor, data transmission is not allowed. If the network quality information shows that the network quality of the communication link is good, data transmission is allowed. Exemplarily, if the packet loss rate of the communication link is large and exceeds a reasonable range, the communication link is not allowed to be used to send video data. Optionally, in this case, a prompt message of poor network quality can be continuously sent to the video receiving device to replace the communication link or repair the communication link.

[0071] See also Figure 6 , Figure 6 1 is a flow chart of an embodiment of a video transmission method of the present application applied to a video receiving device. The video transmission method provided in this embodiment is applied to a video receiving device. Figure 6 As shown, the video transmission method provided by the embodiment of the present disclosure may include the following steps:

[0072] Step S31: the video receiving device receives multiple groups of video data sent by the video sending device.

[0073] The video data is the video data sent by the video sending device based on the video transmission method applied to the video sending device. The video sending device sends the video data based on the video transmission method applied to the video sending device, which can be referred to the above embodiment and will not be described in detail here.

[0074] Step S32: Buffer the received video data.

[0075] Optionally, the received video data collected by different video collection components can be stored separately and independently. In some embodiments, each video data can also be displayed on the display interface of the video receiving device. The video data collected by different video collection components can be displayed separately.

[0076] In some embodiments, the video transmission method may further include the step of storing the received multiple groups of video data in a video stream channel corresponding to each group of video data in the video receiving device.

[0077] In some application scenarios, the video transmission method may further include the following steps: sending packet loss feedback information to the video sending device, so that the video sending device resends the corresponding video data based on the packet loss feedback information.

[0078] In some application scenarios, a first handshake message sent by a video sending device is received and a second handshake message is sent to the video sending device, so that the video sending device determines a video stream channel for sending each video data based on the second handshake message. The first handshake message includes first identification information of each video stream channel in the video sending device that can be used to send video data, and the second handshake message includes second identification information of each video stream channel in the video receiving device that can be used to receive video data. Exemplarily, the first handshake message includes a list of video stream channel identifiers supported by the video sending device, and the second handshake message includes a list of video stream channel identifiers supported by the video receiving device.

[0079] In some application scenarios, a video delay is sent to a video sending device so that the video sending device determines the order of sending video data between each video stream channel in the video sending device based on the video delay. The video delay is used to characterize the synchronization of video data sent by different video stream channels received by a video receiving device. The video delay is used to characterize the synchronization of video data sent by different video stream channels received by a video receiving device. Exemplarily, the video delay can be the timestamps between the timestamps of the videos. Exemplarily, the timestamp of the first video data received by the video receiving device is 11:20, and the timestamp of the second video data is 11:10. The video delay between the first video data and the second video data is 10 seconds. If the first video data and the second video data are displayed directly on the display interface of the video receiving device, the displayed screen will be seriously out of sync, and the time difference between the two screens may even reach 10 seconds.

[0080] In some embodiments, a video stream channel is created for each group of video streams in the video sending device, and each video stream channel independently caches the video data to be sent. Among them, the video sending device also includes a packet timing scheduling module, which can dynamically adjust the sending order between different video stream channels and / or determine the number of video data that needs to be sent each time in each video stream channel according to the transmission delay and network quality of each video stream channel. In some embodiments, the video sending device may also include a network perception and congestion control module, which can be used to determine the network quality of the communication link and determine whether the current communication link can transmit video data. In addition, the video sending device can send the video data in each video stream channel to the video receiving device through a communication link. The video receiving device can create a video stream channel corresponding to the video sending device for each group of video data, and each video stream channel independently caches the received video data. The video receiving device may also include a packet loss feedback module and a packet delay feedback module, the packet loss feedback module notifies the video sending device of the currently lost data, and the packet delay feedback module feedbacks the delay of the data packet through the stream, for example, feedbacks the above-mentioned video delay.

[0081] The feedback frame encapsulation results of each video stream channel in the video receiving device can be referred to Figure 7 The feedback frame encapsulation of each video stream channel may include a connector, a header of video stream channel n, and a status of video stream channel n. The video data of different video stream channels can be distinguished according to the header or connector of video stream channel n. The video data in each video stream channel in the video sending device is encapsulated separately. The encapsulation structure of the video data in each video stream channel can be referred to Figure 5 , may include a connection header, a header of video stream channel n, and a payload of video stream channel n. Video data of different video stream channels may be distinguished according to the header or the connection header of video stream channel n.

[0082] To better understand the video transmission method provided by this application, please refer to Figure 8 and Fig. 9 , Figure 8 It shows that the device side can transmit video with the cloud platform through the second communication link (the device side acts as a video sending device, and the cloud platform acts as a video receiving device), and the cloud platform can transmit video with the playback side through the first communication link (the cloud platform acts as a video sending device, and the playback side acts as a video receiving device). In the process of the video on the device side reaching the playback side, the device side needs to first send the video data to the cloud platform, and then the cloud platform sends the video data to the playback side. In this process, the cloud platform can be both a video sending device and a video receiving device. The video transmission between the device side and the cloud platform, and the video transmission between the cloud platform and the receiving side can be the same. The following embodiments take the video transmission between the device side and the cloud platform as an example. Fig. 9 As shown, the video transmission method may further include the following steps:

[0083] Step S41: the playback end requests the cloud platform to transmit the video data of the device end.

[0084] Step S42: The application layer at the playback end sets the synchronization delay threshold T1 of the first communication link according to the video synchronization delay requirement.

[0085] The synchronization delay threshold T1 may be the above-mentioned preset maximum delay difference.

[0086] Step S43: The reliable UDP layer at the playback end and the platform end creates a stream channel for each video encoder according to the number of video stream groups.

[0087] For the convenience of description, the video stream channel is abbreviated as stream channel below. The number of video stream groups is the same as the number of video acquisition components. Videos are generally encoded before transmission, and the number of video encoders can be the same as the number of stream channels, or in other embodiments, the video encoders can also encode the video data in each stream channel separately.

[0088] Step S44: The platform sets the synchronization delay threshold T1 of the second communication link according to the video synchronization delay requirement.

[0089] That is, the synchronization delay threshold of the first communication link may be the same as the synchronization delay threshold of the second communication link. In other embodiments, the synchronization delay thresholds of the two communication links may also be different.

[0090] Step S45: The reliable UDP layer on the device side and the platform side creates multiple stream channels according to the number of video streams.

[0091] Step S46: The video encoder n at the device end encodes the video data of the transport stream channel n.

[0092] Step S47: Whether the current congestion window allows sending.

[0093] If sending is allowed, execute step S48, otherwise continue to execute step S47.

[0094] Step S48: Different stream channels of the reliable UDP layer on the device side are sent through the same second communication link and the stream channel transmission delay is calculated periodically.

[0095] Step S49: Obtain the difference between the transmission delays of each stream channel in the device end.

[0096] The method for obtaining the transmission delay can be referred to above and will not be described in detail here.

[0097] Step S49: Determine whether the difference exceeds the synchronization delay threshold T1.

[0098] When the difference exceeds the synchronization delay threshold T1, step S50 is executed; otherwise, step S51 is executed.

[0099] Step S50: The device side suspends sending video data of the video stream with a smaller transmission delay value.

[0100] Exemplarily, if the transmission delay of a certain video stream channel is relatively smaller, the transmission of video data in the video stream channel is suspended, and the video data in the video stream channel with a larger transmission delay is continuously sent.

[0101] Step S51: The device side preferentially selects the stream channels with the larger transmission delay to send first.

[0102] That is, the video data in the video stream channel with a larger transmission delay is sent first.

[0103] Step S52: the device obtains the retransmission queue of the selected stream channel to be sent.

[0104] If the video data in the first video stream channel is to be sent, it is necessary to first obtain the retransmission queue in the video stream channel, which may or may not contain data to be retransmitted.

[0105] Step S53: Determine whether there is data to be retransmitted in the retransmission queue.

[0106] If there is data to be retransmitted in the retransmission queue, execute step S54; otherwise, execute step S55.

[0107] Step S54: the device side preferentially selects the data to be retransmitted in the stream channel and sends it first.

[0108] Step S55: The device side preferentially selects the data to be sent in the stream channel and sends it first.

[0109] In some embodiments, the reliable UDP synchronous transmission protocol negotiates transmission parameters before data transmission. The specific negotiation process is as follows:

[0110] The reliable UDP video sending device informs the other end through the reliable UDP handshake packet that the local end supports the multi-stream multiplexing channel transmission capability and the list of stream channel identifiers to be used subsequently, and provides the transmission delay and synchronization delay threshold of each stream channel in the stream channel identifier list. The reliable UDP video receiving device informs the other end through the reliable UDP handshake packet that the local end supports the multi-stream multiplexing channel transmission capability and the list of stream channel identifiers to be used subsequently, and provides the transmission delay and synchronization delay threshold of each stream channel in the stream channel identifier list.

[0111] The video sending device selects to create multiple reliable stream channels from the identification list of the reliable transmission stream channels on the local end for transmission. For the video stream channel, it needs to wait for the data response confirmation from the other end and retransmit the lost data. The video receiving device creates multiple reliable stream channels to receive the corresponding stream data, responds to the stream channels in the identification list of the other end's reliable stream channels, and feeds back the video delay of the current stream channel. In addition, the video receiving device feeds back the timestamp of the received video data for each stream channel. This can be done through the timestamps and other information corresponding to each video stream channel in the video receiving device, or a total video stream channel can be constructed, and the packet loss feedback information and receiving timestamp and other information of each video stream channel can be fed back through this total video stream channel.

[0112] The video sending device calculates network quality information such as the current network congestion window, network RTT, and packet loss, and calculates information such as video frame transmission delay and delay difference. The video sending device evaluates the amount of video data to be sent next time based on the congestion window, selects the next video stream channel to be sent based on the transmission delay of each stream channel, and sends retransmission packets first based on the packet loss information of the stream channel. The video receiving device ensures that the stream data is received completely based on the information of each stream channel, and notifies the business layer to collect the complete data in a timely manner.

[0113] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0114] Multiple video stream channels are multiplexed and transmitted through a reliable UDP connection. Multiple video streams are carried in one link for unified management and scheduling of congestion window, video delay and business logic. Compared with the traditional method of allocating a communication link to each video stream, this solution can better ensure the synchronization of multi-channel video transmission.

[0115] See also Fig.10 , Fig.10 It is a structural diagram of an embodiment of a video transmission device of the present application. The video transmission device 60 can execute the video transmission method provided by the embodiment of the video transmission method applied to the video sending device. The video transmission device 60 includes a video acquisition module 61, a sequence determination module 62 and a data sending module 63; the video acquisition module 61 is used for the video sending device to include multiple groups of video data collected by different video acquisition components; the sequence determination module 62 is used to determine the sending order of each group of video data based on the data transmission delay between the video sending device and the video receiving device; the data sending module 63 is used to multiplex the communication link between the video sending device and the video receiving device according to the sending order, and send each group of video data to the video receiving device in sequence.

[0116] Among them, the sequence determination module 62 can be the above-mentioned packet timing scheduling module or a collection of the above-mentioned packet scheduling module and the network perception and congestion control module.

[0117] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0118] The functions of each module may be found in the embodiment of the video transmission method applied to the video sending device, and will not be described in detail here.

[0119] See also Fig.11 , Fig.11 1 is a schematic diagram of the structure of another embodiment of a video transmission device of the present application. Another video transmission device 70 can execute the video transmission method provided by the above-mentioned video transmission method embodiment applied to a video receiving device. Another video transmission device 70 includes a data receiving module 71 and a cache module 72; the data receiving module 71 is used for the video receiving device to receive multiple groups of video data sent by the video sending device based on the above-mentioned video transmission method; the cache module 72 is used to cache each received video data.

[0120] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0121] The functions of each module may refer to the above-mentioned video transmission method embodiment applied to a video receiving device, and will not be described in detail here.

[0122] See also Fig.12 , Fig.12 1 is a schematic diagram of the structure of an embodiment of a video sending device of the present application. The video sending device 80 includes a first memory 81, a first processor 82 and a first communication module 83, the first processor 82 is used to execute the program instructions stored in the first memory 81, determine the sending order of each group of video data, and the first communication module 83 is used to send each group of video data to the video receiving device, wherein the first communication module 83, the first memory 81 and the first processor 82 cooperate to implement the above-mentioned video transmission method applied to the video sending device.

[0123] In a specific implementation scenario, the video sending device 80 may include but is not limited to: a microcomputer, a server, and in addition, the video sending device 80 may also include a carrier device such as a laptop computer and a tablet computer, which is not limited here.

[0124] The first communication module 83 can be any module with communication function, which is not specifically limited here. Specifically, the first processor 82 is used to control itself and the first memory 81 to implement the steps in any of the above-mentioned video transmission method embodiments. The first processor 82 can also be called a CPU (Central Processing Unit). The first processor 82 may be an integrated circuit chip with signal processing capabilities. The first processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the first processor 82 can be implemented by an integrated circuit chip.

[0125] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0126] See also Fig.13 , Fig.13 1 is a schematic diagram of the structure of an embodiment of a video receiving device of the present application. The video receiving device 90 includes a second memory 91, a second processor 92 and a second communication module 93, the second communication module 93 is used to receive multiple groups of video data sent by the video sending device, the second processor 92 is used to execute the program instructions stored in the second memory 91, and cache the received video data, wherein the second communication module 93, the second memory 91 and the second processor 92 cooperate to implement the above-mentioned video transmission method applied to the video receiving device.

[0127] In a specific implementation scenario, the video receiving device 90 may include but is not limited to: an Internet of Things device, a microcomputer, and a server. In addition, the video receiving device 90 may also include a laptop computer, a tablet computer, and other carrier devices, which are not limited here.

[0128] The second communication module 93 can be any module with communication function, which is not specifically limited here. Specifically, the second processor 92 is used to control itself and the second memory 91 to implement the steps in any of the above-mentioned video transmission method embodiments. The second processor 92 can also be called a CPU (Central Processing Unit). The second processor 92 may be an integrated circuit chip with signal processing capabilities. The second processor 92 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the second processor 92 can be implemented by an integrated circuit chip.

[0129] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0130] See also Fig.14 , Fig.14 The video transmission system 10 includes a video transmitting device 80 and a video receiving device 90, and the video transmitting device 80 and the video receiving device 90 are in communication connection.

[0131] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0132] See also Fig.15 , Fig.15 The computer-readable storage medium 100 stores program instructions 101, which implement the steps in any of the above-mentioned video transmission method embodiments when executed by a processor.

[0133] The above scheme first determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then reuses a communication link to send each group of video data to the video receiving device in sequence according to the sending order. Compared with different groups of video data using different communication links for communication transmission, this scheme can make the groups of video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0134] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0135] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0137] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A video transmission method, characterized in that: The video transmission method is applied to a video sending device, and the video transmission method comprises: The video sending device includes multiple groups of video data collected by different video acquisition components; Determining a sending order of each group of the video data based on a data transmission delay between the video sending device and the video receiving device; The communication link between the video sending device and the video receiving device is multiplexed according to the sending order, and each group of the video data is sent to the video receiving device in sequence.

2. The method according to claim 1, characterized in that Different groups of video data are in different video stream channels in the video sending device, and determining the sending order of each group of video data based on the data transmission delay between the video sending device and the video receiving device includes: Obtaining the transmission delay of video data sent by different video stream channels through the communication link within a preset historical time period; Based on the transmission delay of each of the video stream channels, the sending order of each of the video data between the video stream channels is determined, wherein the sending order of the video data in the video stream channel with high transmission delay is before the video data in the video stream channel with low transmission delay.

3. The method according to claim 2, characterized in that The method further comprises: Receiving a video delay fed back by the video receiving device, where the video delay is used to characterize synchronization of video data sent by different video stream channels received by the video receiving device; In response to the video delay being greater than or equal to the preset delay, performing the step of determining the sending order of each group of the video data based on the data transmission delay between the video sending device and the video receiving device; And / or, in response to the video delay being less than the preset delay, maintaining the sending order of the video data between the current video stream channels.

4. The method according to claim 2, characterized in that: The method further comprises: Obtaining a delay difference between transmission delays of each of the video stream channels, wherein a delay difference that is greater than or equal to a preset maximum delay difference is a target delay difference; In response to the existence of the target delay difference, the sending of video data in the video stream channel corresponding to the smaller delay difference among the target delay differences is suspended within a preset time period.

5. The method according to claim 2, characterized in that: The method further comprises: Determining the amount of video data that needs to be sent each time in each video stream channel based on the transmission delay of each video stream channel; For each of the video stream channels, a secondary sending order between each video data in the video stream channel is determined.

6. The method according to claim 2, characterized in that The method further comprises: Sending first handshake information to the video receiving device, so that the video receiving device determines second handshake information based on the first handshake information, wherein the first handshake information includes first identification information of each video stream channel in the video sending device that can be used to send video data, and the second handshake information includes second identification information of each video stream channel in the video receiving device that can be used to receive video data; Receiving second handshake information sent by the video receiving device; Based on the second identification information, a video stream channel used by the video sending device to send each of the video data is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In response to receiving the packet loss feedback information sent by the video receiving device, determining the data to be retransmitted and the target video stream channel corresponding to the data to be retransmitted; The sending order of the data to be retransmitted in the target video stream channel is placed before other video data in the target video stream channel, and the other video data is non-retransmitted data.

8. The method according to any one of claims 1 to 6, characterized in that Before multiplexing the communication link between the video sending device and the video receiving device in the sending order and sequentially sending each group of the video data to the video receiving device, the method further includes: Acquiring network quality information of the communication link; Determining whether the communication link allows data to be sent based on the network quality information; In response to the communication link allowing data to be sent, the step of multiplexing the communication link between the video sending device and the video receiving device according to the sending order and sending each group of the video data to the video receiving device in sequence is performed.

9. A video transmission method, characterized in that: The video transmission method is applied to a video receiving device, and the video transmission method comprises: The video receiving device receives a plurality of groups of video data sent by the video sending device based on the video transmission method according to any one of claims 1 to 8; The received video data are cached.

10. The method according to claim 9, characterized in that The method further comprises at least one of the following steps: storing the received multiple groups of video data into the video stream channel corresponding to each group of the video data in the video receiving device; Sending packet loss feedback information to the video sending device so that the video sending device resends corresponding video data based on the packet loss feedback information; receiving first handshake information sent by the video sending device and sending second handshake information to the video sending device, so that the video sending device determines a video stream channel for sending each of the video data based on the second handshake information; The video delay is sent to the video sending device so that the video sending device determines the sending order of video data between the video stream channels in the video sending device based on the video delay. The video delay is used to characterize the synchronization of video data sent by different video stream channels received by the video receiving device.

11. A video sending device, characterized in that: It includes a first communication module, a first memory and a first processor, the first processor is used to execute program instructions stored in the first memory to determine the sending order of each group of video data, the first communication module is used to send each group of video data to a video receiving device, wherein the first communication module, the first memory and the first processor cooperate to implement the video transmission method described in any one of claims 1 to 8.

12. A video receiving device, characterized in that: It includes a second communication module, a second memory and a second processor, the second communication module is used to receive multiple groups of video data sent by a video sending device, the second processor is used to execute program instructions stored in the second memory, and cache the received video data, wherein the second communication module, the second memory and the second processor cooperate to implement the video transmission method described in claim 9 or 10.

13. A video transmission system, characterized in that: The video transmission system comprises the video sending device as claimed in claim 11 and the video receiving device as claimed in claim 12, and the video sending device and the video receiving device are communicatively connected.

14. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method described in any one of claims 1 to 8 or the method described in claim 9 or 10 is implemented.

Images