Video transmission method, related device, system, and storage medium

Abstract

The application discloses a video transmission method, related equipment, a system and a storage medium. The video transmission method is applied to a video sending device. The video transmission method comprises the following steps: a plurality of groups of video data collected by different video collection components are included in the video sending device; the sending sequence of the video data is determined based on the data transmission delay between the video sending device and a video receiving device; and the communication link between the video sending device and the video receiving device is multiplexed according to the sending sequence, and the video data is sequentially sent to the video receiving device. The above scheme can improve the synchronization effect of the video receiving device on the video data corresponding to each video collection component.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a video transmission method, related equipment, system and storage medium. Background Art

[0002] Currently, the main application scenario of multi-eye wireless cameras on the market is generally that the multi-eye wireless video acquisition device is connected to the public cloud platform via Wi-Fi, and the mobile terminal can access the cloud platform through Wi-Fi or mobile network to view the real-time video and recordings collected by the multi-eye wireless video acquisition device in real time.

[0003] by Figure 1 Taking the video transmission of a binocular camera as an example, the video data in different video acquisition components of the binocular camera are currently transmitted to the public cloud platform through communication link 1 and communication link 2 respectively, and then the public cloud platform uses communication link 3 and communication link 4 to send the video data collected by different video acquisition components to the terminal device. After the video data collected by different video acquisition components pass through different network transmission paths, the time sequence of multiple videos arriving at the mobile terminal may be inconsistent, which may easily cause the problem of asynchronous video playback. Summary of the Invention

[0004] This application at least provides a video transmission method, related equipment, system and storage medium.

[0005] The present application provides a video transmission method, which is applied to a video sending device. The video transmission method includes: the video sending device includes multiple groups of video data collected by different video acquisition components; based on the data transmission delay between the video sending device and the video receiving device, the sending order of each group of video data is determined; the communication link between the video sending device and the video receiving device is multiplexed according to the sending order, and each group of video data is sent to the video receiving device in sequence.

[0006] The present application provides a video transmission method, which is applied to a video receiving device. The video transmission method includes: the video receiving device receives multiple groups of video data sent by a video sending device based on the above-mentioned video transmission method; and caches the received video data.

[0007] The application provides a video transmission device, comprising a video acquisition module, a sequence determination module and a data sending module; the video acquisition module is used for including multiple groups of video data collected by different video acquisition components in a video sending device; the sequence determination module is used for determining the sending sequence of each group of video data based on the data transmission delay between the video sending device and a video receiving device; and the data sending module is used for multiplexing the communication link between the video sending device and the video receiving device according to the sending sequence, and sequentially sending each group of video data to the video receiving device.

[0008] The application provides a video transmission device, comprising a data receiving module and a buffer module; the data receiving module is used for receiving multiple groups of video data sent by a video sending device based on the above-mentioned video transmission method by a video receiving device; and the buffer module is used for buffering each received video data.

[0009] The application provides a video sending device, comprising a first communication module, a first memory and a first processor; the first processor is used for executing the program instructions stored in the first memory to determine the sending sequence of each group of video data; and the first communication module is used for sending each group of video data to a video receiving device; wherein the first communication module, the first memory and the first processor cooperate to realize the above-mentioned first video transmission method.

[0010] The application provides a video receiving device, comprising a second communication module, a second memory and a second processor; the second communication module is used for receiving multiple groups of video data sent by a video sending device; the second processor is used for executing the program instructions stored in the second memory to buffer each received video data; wherein the second communication module, the second memory and the second processor cooperate to realize the above-mentioned second video transmission method.

[0011] The application provides a video transmission system, comprising the above-mentioned video sending device and the above-mentioned video receiving device; the video sending device and the video receiving device are communicatively connected.

[0012] The application provides a computer readable storage medium, which stores program instructions; when the program instructions are executed by a processor, any one of the above-mentioned video transmission methods is realized.

[0013] The above-mentioned scheme determines the sending sequence of each group of video data according to the data delay between the video sending device and the video receiving device, and then multiplexes a communication link to sequentially send each group of video data to the video receiving device according to the sending sequence; compared with using different communication links to communicate and transmit different groups of video data, the scheme can make each group of video data received by the video receiving device more synchronized, that is, the synchronization effect between each group of video data is improved.

[0014] It should be understood that the general description above and the detailed description below are only exemplary and explanatory, not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0015] The drawings incorporated in the specification include exemplary embodiments in accordance with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0016] Figure 1 is a schematic diagram of a conventional multi-path video transmission method provided by the present application;

[0017] Figure 2 is a flowchart of an embodiment of a video transmission method applied to a video sending device of the present application;

[0018] Figure 3 is another flowchart of an embodiment of a video transmission method applied to a video sending device of the present application;

[0019] Figure 4 is a sub-flowchart of step S12 in Figure 2

[0020] Figure 5 is a structural diagram of a data frame sent by a video sending device to a video receiving device of the present application;

[0021] Figure 6 is a flowchart of an embodiment of a video transmission method applied to a video receiving device of the present application;

[0022] Figure 7 is a structural diagram of a data frame fed back by a video receiving device to a video sending device of the present application;

[0023] Figure 8 is a schematic diagram of video transmission between multiple devices of the present application;

[0024] Figure 9 is still another flowchart of an embodiment of a video transmission method of the present application;

[0025] Figure 10 is a structural diagram of an embodiment of a video transmission device of the present application;

[0026] Figure 11 is a structural diagram of another embodiment of a video transmission device of the present application;

[0027] Figure 12 is a structural diagram of an embodiment of a video sending device of the present application;

[0028] Figure 13 is a structural diagram of an embodiment of a video receiving device of the present application;​

[0029] Figure 14 is a structural schematic diagram of an embodiment of a video transmission system of the present application.

[0030] Figure 15 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application. DETAILED DESCRIPTION

[0031] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0032] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, but not for the sake of limitation.

[0033] The term "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents that the front and rear associated objects are in an "or" relationship. In addition, "multiple" herein means two or more than two. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] The present application provides a video transmission method. In the present application, the execution subject for realizing the video transmission method described in the present application can be a video transmission device, a communication terminal, etc. For example, the video transmission device can be arranged in a communication terminal or a server or other processing equipment, wherein the terminal equipment can be an electronic device, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the video transmission method can be realized by a processor calling computer readable instructions stored in a memory.

[0035] Please refer to Figure 2 , Figure 2 is a flowchart of an embodiment of a video transmission method applied to a video sending device of the present application. The embodiment provided by the present application provides a video transmission method applied to a video sending device. As shown in Figure 2 , the video transmission method provided by the present application can include the following steps:

[0036] Step S11: The video sending device includes multiple groups of video data collected by different video collection components.

[0037] In some application scenarios, the video sending device may be a video acquisition device with different video acquisition components, and the video receiving device may be a communication terminal or a cloud platform that establishes a communication connection with the video acquisition device. In some application scenarios, the video sending device may be a cloud platform or a communication terminal that caches multiple sets of video data sent by a video acquisition device with different video acquisition components, and the video receiving device may be another cloud platform or another communication terminal connected to the cloud platform or the communication terminal. That is, the video transmission method between the video acquisition device and the cloud platform may be the same as the video transmission method between the cloud platform and the communication terminal. The embodiment of the present application takes one of the stages as an example, and the other stage is executed for reference.

[0038] For example, the video transmitting device may be a multi-camera, and the different video capture components may be different camera components within the multi-camera. Each camera component captures a set of video data. For example, if the multi-camera is a binocular camera, the binocular camera has two camera components, and the two camera components can capture two sets of video data. The number of video data items contained in each set of video data may be one or more, and is not specifically limited herein.

[0039] Step S12: Based on the data transmission delay between the video sending device and the video receiving device, determine the sending order of each group of video data.

[0040] In some application scenarios, the data transmission delay can be determined based on the average transmission delay of each set of video data transmitted between the video transmitting device and the video receiving device within a preset historical time period, or other mathematical statistics. The order in which video data of different data sizes are transmitted can be dynamically adjusted based on the data delay. For example, if the current data transmission delay is large, the larger video data among the groups of video data can be prioritized. Alternatively, to ensure that the video receiving device can promptly understand the captured scene, the smaller video data can be prioritized. In other words, the data size of each video data set in different groups may vary.

[0041] In some application scenarios, the transmission delay of the video data corresponding to different video collection components can be determined to obtain multiple data transmission delay conditions, so as to dynamically adjust the sending order between different groups of video data. For example, the video collection components include a video collection component a and a video collection component b. The video collection component a sends i video data in a preset historical time period, and the video collection component b sends j video data in a preset time period. Then, the data transmission delay condition of the video collection component a can be determined according to the transmission delay of the i video data, and the data transmission delay condition of the video collection component b can be determined according to the transmission delay of the j video data. Then, the sending relationship between the video data collected by the video collection component a and the video collection component b is dynamically adjusted according to the data transmission delay condition of the video collection component a and the data transmission delay condition of the video collection component b.

[0042] Step S13: multiplexing the communication link between the video sending device and the video receiving device according to the sending order, and sequentially sending each group of video data to the video receiving device.

[0043] For example, each group of video data can be sent in sequence using the communication link according to the sending order. After the video receiving device receives each group of video data from the communication link, the video data collected by different video collection components can be buffered respectively.

[0044] The above scheme determines the sending order of each group of video data according to the data transmission delay condition between the video sending device and the video receiving device, and then sequentially sends each group of video data to the video receiving device according to the sending order by multiplexing a communication link. Compared with using different communication links to communicate and transmit different groups of video data, the present scheme can make the video data received by the video receiving device more synchronized, that is, the synchronization effect between each group of video data is improved.

[0045] In some embodiments, the video transmission method further comprises the following steps:

[0046] Receiving the video delay feedback by the video receiving device. The video delay is used to represent the synchronization condition of the video data sent by different video stream channels received by the video receiving device. For example, the video delay can be the time stamp between the time stamps of the video. For example, the time stamp of the first video data received by the video receiving device is 11:20, and the time stamp of the second video data is 11:10. The video delay between the first video data and the second video data is 10 seconds. If the first video data and the second video data are directly displayed on the display interface of the video receiving device, the display screen will be seriously out of synchronization, and the time difference between the two screens can reach 10 seconds.

[0047] In some application scenarios, in response to the video latency being greater than or equal to the preset latency, the step of determining the sending order of the groups of video data based on the data transmission latency between the video sending device and the video receiving device is performed. If the video latency between different groups of video data is large, it indicates that the videos are seriously out of synchronization, and the sending order of the video data transmission needs to be adjusted. In other application scenarios, in response to the video latency being less than the preset latency, the sending order of the video data between the current video stream channels is maintained. If the video latency is small, it indicates that the video data is relatively synchronized, and the sending order does not need to be adjusted, and the current video sending order can be used to continue transmitting the video data collected by the video capture components in the video sending device.

[0048] In some embodiments, the different groups of video data are in different video stream channels in the video sending device. Please refer to Figure 3 The video transmission method can further include the following steps:

[0049] Step S21: sending first handshake information to the video receiving device, so that the video receiving device determines second handshake information based on the first handshake information.

[0050] The first handshake information includes first identification information of each video stream channel in the video sending device that can be used to send video data, and the second handshake information includes second identification information of each video stream channel in the video receiving device that can be used to receive video data. For example, the first handshake information includes a list of video stream channel identifications supported by the video sending device, and the second handshake information includes a list of video stream channel identifications supported by the video receiving device. In some application scenarios, the video sending device sends the first handshake information in the form of reliable UDP, and the first handshake information can further include at least one of the following: the capability of the local end of the video receiving device to support a multi-stream multiplexing communication link, the transmission latency of each video stream channel, and a preset maximum latency difference (synchronization latency threshold). The transmission latency of each video stream channel can be determined according to the transmission of other handshake information sent between the video sending device and the video receiving device. The transmission latency can be used to determine the sending order of the first batch of video data sent by each video stream channel.

[0051] Step S22: receiving the second handshake information sent by the video receiving device.

[0052] Step S23: determining the video stream channel used by the video sending device to send each group of video data based on the second identification information.

[0053] Exemplarily, the video stream channels for transmitting the video data can be determined according to the same identifier selected from the first handshake information and the second handshake information. That is, a plurality of video stream channels can be created from the video stream identifier list of the local end for transmitting the video. In addition, the video receiving device can also create a plurality of video stream channels for receiving the video data of each group.

[0054] In some embodiments, referring to Figure 4 The step S12 can include the following steps:

[0055] Step S121: Obtain the transmission delay of the video data transmitted by the different video stream channels through the communication link in a preset historical time period.

[0056] Optionally, the implementation of the step S121 can be that the receiving time stamp fed back by the video receiving device after receiving the video data transmitted by the video stream channel, and then the video sending device determines the transmission delay according to the transmission time stamp and the receiving time stamp of each video data. The transmission delay of the video data in different video stream channels is calculated independently, that is, if there are x video stream channels in the video sending device, x transmission delays are calculated. Specifically, the average value or other mathematical statistical value of the transmission delay of each video data transmitted by each video stream channel in a preset historical time can be determined as the transmission delay of each video stream channel.

[0057] Step S122: Determine the transmission order of the video data between the video stream channels based on the transmission delay of each video stream channel.

[0058] Among them, the transmission order of the video data in the video stream channel with high transmission delay is before the video data in the video stream channel with low transmission delay. That is, the video data in the video stream channel with large transmission delay can be transmitted preferentially. The video data between different video stream channels can be transmitted alternately, or the video data in another video stream channel is transmitted after transmitting a plurality of video data in one video stream channel.

[0059] Optionally, the video transmission method further includes the following steps:

[0060] Obtain the delay difference between the transmission delays of each video stream channel. Among them, the delay difference greater than or equal to the preset maximum delay difference is the target delay difference. In response to the existence of the target delay difference, the video data in the video stream channel corresponding to the smaller delay difference in the target delay difference is suspended for transmission in a preset time period. Optionally, in response to the absence of the target delay difference, the step S122 is executed.

[0061] Because the communication link may be subject to different external interference conditions at different times, the transmission delays generated by transmitting video data in different video stream channels at different times may also be different due to different interferences. In some application scenarios, if the transmission delay difference in different video stream channels is too large, the transmission of video data in the video stream channel with a smaller transmission delay can be suspended, and the video data in the video stream channel with a larger transmission delay can be sent intensively. Exemplarily, the pause time can be pre-set, or it can be determined based on the relationship between the delay difference between the transmission delays of each video stream channel and the preset maximum delay difference. Exemplarily, if the pause time corresponding to the preset maximum delay difference is t seconds, and the transmission delay difference of each video stream channel is z times the preset maximum delay difference, the pause time can be t / z seconds.

[0062] Optionally, the video transmission method further comprises the following steps: determining the amount of video data to be transmitted each time in each video stream channel based on the transmission delay of each video stream channel; and determining, for each video stream channel, a secondary transmission order between each video data in the video stream channel.

[0063] Optionally, the amount of video data that needs to be sent each time in a video stream channel with a larger transmission delay is greater than the amount of video data that needs to be sent each time in a video stream channel with a smaller transmission delay. Larger and smaller can be the relative size relationship of the transmission delay between the two video stream channels. Exemplarily, the amount of video data that needs to be sent each time in each channel can be determined based on the delay difference between the transmission delays of each video stream channel. For example, if the transmission delay of the first video stream channel is larger and much larger than the transmission delay of the second video stream channel, you can choose to send multiple video data in the first video stream channel at one time, and then send the video data in the second video stream channel. The secondary sending order of each video data in each video stream channel is determined separately, and the secondary sending order is also the sending order between different video data in the same video channel. For example, please refer to Figure 5 The video data encapsulation format in the nth video stream channel in the video sending device shown is as follows: Figure 5 As shown, each video data includes a connector, a header of video stream channel n, and a payload of video stream channel n. Video data of different video stream channels can be distinguished based on the connector, and the headers of different video stream channels n can distinguish different video data in video stream channel n. Among them, two sequence numbers can be encapsulated in the encapsulation format. The sequence number in the connector is the order relative to other video stream channels, and the sequence number in the header of video stream channel n is the order relative to other video data in the video stream channel. Video data in different video stream channels are sent alternately, so the corresponding sequence number needs to be set in the connector. The data of a single video stream channel also needs to be sent sequentially, so the second sequence number can be set in the header.

[0064] In some application scenarios, the order of each connector in the video data to be sent each time in the same video stream channel can be the same. For example, in a video stream channel, two video data to be sent at one time, one of which has an order of r_m, and the other has an order of r_m+1, where r represents the serial number of the connector, and m and m+1 represent the serial number of the header, and the same applies hereinafter. In another video stream channel, the video data to be sent at one time includes two, one of which has an order of r+1_t, and the other has an order of r+1_t+1, and the order of the four video sending orders is r_m, r_m+1, r+1_t, and r+1_t+1.

[0065] In some application scenarios, for the connector, the video data sent by different video stream channels needs to be numbered continuously, and the serial numbers of different video data in the connector are different. For example, in a video stream channel, there are two video data to be sent, one of which has an order of r_m, and the other has an order of r+1_m+1. In another video stream channel, there are two video data to be sent, one of which has an order of r+2_t, and the other has an order of r+3_t+1, and the order of the four video sending orders is r_m, r+1_m+1, r+2_t, and r+3_t+1.

[0066] Optionally, the video transmission method further comprises the following steps: in response to receiving the packet loss feedback information sent by the video receiving device, determining the data to be retransmitted and the target video stream channel corresponding to the data to be retransmitted. Then, the sending order of the data to be retransmitted in the target video stream channel is placed before other video data in the target video stream channel, and the other video data is non-retransmission data.

[0067] That is, if the packet loss feedback information sent by the video receiving device is received, the video data to be retransmitted and the video stream channel to which the video data belongs can be determined according to the packet loss feedback information. The sending priority of the data to be retransmitted in the video stream channel is higher than that of the video data transmitted for the first time, that is, if there is data to be retransmitted in a video stream channel, the data to be retransmitted is transmitted preferentially.

[0068] In some embodiments, before performing the above step S13, the following step can also be performed:

[0069] The network quality information of the communication link is obtained. Based on the network quality information, it is determined whether the communication link allows data to be sent. Then, in response to the communication link allowing data to be sent, the step of multiplexing the communication link between the video sending device and the video receiving device according to the sending order and sequentially sending each group of video data to the video receiving device is performed.

[0070] The network quality information of the communication link can include, but is not limited to, congestion window information, network RTT information, and / or packet loss rate information. If the network quality information shows that the network quality of the communication link is poor, the sending of data is not allowed, and if the network quality information shows that the network quality of the communication link is good, the sending of data is allowed. For example, if the packet loss rate of the communication link is large and exceeds a reasonable range, the communication link is not allowed to be used to send video data. Optionally, in this case, the video receiving device can be continuously sent prompt information that the network quality is poor to replace the communication link or repair the communication link.

[0071] Referring to Figure 6 , Figure 6 is a flowchart of an embodiment of a video transmission method applied to a video receiving device. The video transmission method provided by the embodiment is applied to a video receiving device. As shown in Figure 6 , the video transmission method provided by the embodiment can include the following steps:

[0072] Step S31: The video receiving device receives a plurality of groups of video data sent by a video sending device.

[0073] The video data is video data sent by the video sending device based on the above-mentioned video transmission method applied to the video sending device. The manner in which the video sending device sends the video data based on the above-mentioned video transmission method applied to the video sending device can refer to the above-mentioned embodiments, which will not be described here.

[0074] Step S32: Cache each video data received.

[0075] Optionally, the received video data collected by different video collection components can be stored separately. In some embodiments, each video data can also be displayed on the display interface of the video receiving device. The video data collected by different video collection components can be displayed separately.

[0076] In some embodiments, the video transmission method can further include the step of storing the received plurality of groups of video data in the video stream channel corresponding to each group of video data in the video receiving device.

[0077] In some application scenarios, the video transmission method can further include the step of sending packet loss feedback information to the video sending device, so that the video sending device re-sends the corresponding video data based on the packet loss feedback information.

[0078] In some application scenarios, the video receiving device sends first handshake information to the video sending device and sends second handshake information to the video sending device, so that the video sending device determines the video stream channel used to send the video data based on the second handshake information. The first handshake information includes first identification information of each video stream channel in the video sending device that can be used to send the video data, and the second handshake information includes second identification information of each video stream channel in the video receiving device that can be used to receive the video data. For example, the first handshake information includes a list of video stream channel identifications supported by the video sending device, and the second handshake information includes a list of video stream channel identifications supported by the video receiving device.

[0079] In some application scenarios, the video sending device sends a video time delay to the video receiving device, so that the video sending device determines the sending order of the video data between the video stream channels in the video sending device based on the video time delay. The video time delay is used to represent the synchronization of the video data sent by different video stream channels received by the video receiving device. The video time delay is used to represent the synchronization of the video data sent by different video stream channels received by the video receiving device. For example, the video time delay can be the time stamp between the time stamps of the video. For example, the time stamp of the first video data received by the video receiving device is 11:20, and the time stamp of the second video data is 11:10, and the video time delay between the first video data and the second video data is 10 seconds. If the first video data and the second video data are directly displayed on the display interface of the video receiving device, it will cause the display screen to be seriously out of synchronization, and even the time difference between the two screens reaches 10 seconds.

[0080] In some embodiments, the video sending device creates a video stream channel for each group of video streams, and each video stream channel independently buffers the video data to be sent. The video sending device further includes a packet timing scheduling module, which can dynamically adjust the sending order between different video stream channels and / or determine the number of video data to be sent each time in each video stream channel according to the transmission delay and network quality of each video stream channel. In some embodiments, the video sending device can further include a network awareness and congestion control module, which can be used to determine the network quality of the communication link and determine whether the current communication link can perform video data transmission. In addition, the video sending device can send the video data in each video stream channel to the video receiving device through a communication link. The video receiving device can create a video stream channel corresponding to the video sending device for each group of video data, and each video stream channel independently buffers the received video data. The video receiving device can further include a packet loss feedback module and a packet delay feedback module, the packet loss feedback module notifies the video sending device of the currently lost data, and the packet delay feedback module feeds back the delay of the stream data packet, for example, the above-mentioned video time delay.

[0081] The feedback frame packaging result of each video stream channel in the video receiving device can refer to Figure 7 The feedback frame packaging of each video stream channel can include a connection header, a header of the video stream channel n, and a state of the video stream channel n. The video data of different video stream channels can be distinguished according to the header of the video stream channel n or the connection header. The video data in each video stream channel in the video sending device is packaged respectively, and the packaging structure of the video data in each video stream channel can refer to Figure 5 The video data in each video stream channel in the video sending device is packaged respectively, and the packaging structure of the video data in each video stream channel can refer to

[0082] For better understanding of the video transmission method provided in the present application, please refer to Figure 8 and Figure 9 , Figure 8 It is shown that the device end can perform video transmission with the cloud platform through the second communication link (the device end as a video sending device, and the cloud platform as a video receiving device), and the cloud platform can perform video transmission with the playing end through the first communication link (the cloud platform as a video sending device, and the playing end as a video receiving device). In the process of the video of the device end reaching the playing end, the device end needs to send the video data to the cloud platform first, and then the cloud platform sends the video data to the playing end. In this process, the cloud platform can be a video sending device or a video receiving device. The video transmission between the device end and the cloud platform and the video transmission between the cloud platform and the receiving end can be the same, and the following embodiments take the video transmission between the device end and the cloud platform as an example. As shown in Figure 9 The video transmission method can further include the following steps:

[0083] Step S41: The playing end requests the cloud platform to transmit the video data of the device end.

[0084] Step S42: The playing end application layer sets a synchronization delay threshold T1 of the first communication link according to the video synchronization delay requirement.

[0085] The synchronization delay threshold T1 can be the above-mentioned preset maximum delay difference.

[0086] Step S43: The playing end and platform reliable UDP layer create a stream channel for each video encoder according to the group number of the video stream.

[0087] For convenience of description, the video stream channel is simply written as a stream channel below. The group number of the video stream is the same as the number of the video acquisition component. The video is generally encoded before being transmitted, and the number of the video encoder can be the same as the number of the stream channel, or in some other embodiments, the video encoder can also encode the video data in each stream channel respectively.

[0088] Step S44: The platform end sets the synchronization delay threshold T1 of the second communication link according to the video synchronization delay requirement.

[0089] That is, the synchronization delay threshold of the first communication link and the synchronization delay threshold of the second communication link can be the same, and in other embodiments, the synchronization delay thresholds of the two communication links can also be different.

[0090] Step S45: The device end and the platform end reliable UDP layer create multiple stream channels according to the number of video streams.

[0091] Step S46: The video encoder n of the device end encodes the video data of the transmission stream channel n.

[0092] Step S47: Whether the current congestion window allows sending.

[0093] If sending is allowed, step S48 is executed, otherwise step S47 can be continued.

[0094] Step S48: Different stream channels of the device end reliable UDP layer are sent through the same second communication link and the transmission delay of the stream channel is calculated.

[0095] Step S49: The difference between the transmission delays of each stream channel in the device end is obtained.

[0096] The way to obtain the transmission delay can refer to the above, which is not repeated here.

[0097] Step S49: Determine whether the difference exceeds the synchronization delay threshold T1.

[0098] In the case where the difference exceeds the synchronization delay threshold T1, step S50 is executed, otherwise step S51 is executed.

[0099] Step S50: The device end suspends the sending of the video data of the video stream with the smaller transmission delay.

[0100] For example, if the transmission delay of a certain video stream channel is relatively small, the transmission of the video data in the video stream channel is suspended, and the video data in the video stream channel with the larger transmission delay is continued to be sent.

[0101] Step S51: The device end preferentially selects the stream channel with the larger transmission delay for sending.

[0102] That is, the video data in the video stream channel with the larger transmission delay is preferentially sent.

[0103] Step S52: The device end obtains the retransmission queue of the selected to-be-sent stream channel.

[0104] If the video data in the first video stream channel is about to be sent, the retransmission queue in the video stream channel needs to be obtained first. The retransmission queue can have data to be retransmitted or can have no data to be retransmitted.

[0105] Step S53: It is judged whether the retransmission queue has data to be retransmitted.

[0106] If the retransmission queue has data to be retransmitted, step S54 is performed, otherwise step S55 is performed.

[0107] Step S54: The device end preferentially selects the data to be retransmitted in the stream channel for preferential sending.

[0108] Step S55: The device end preferentially selects the data to be sent in the stream channel for preferential sending.

[0109] In some embodiments, the reliable UDP synchronization transmission protocol can negotiate transmission parameters before data transmission. The specific negotiation process is as follows:

[0110] The video sending device of the reliable UDP notifies the opposite end that the local end supports the multi-stream multiplexing channel transmission capability and the identification list of the stream channels to be used subsequently through the handshake packet of the reliable UDP, and provides the transmission delay and synchronization delay threshold of each stream channel in the identification list of the stream channels. The video receiving device of the reliable UDP notifies the opposite end that the local end supports the multi-stream multiplexing channel transmission capability and the identification list of the stream channels to be used subsequently through the handshake packet of the reliable UDP, and provides the transmission delay and synchronization delay threshold of each stream channel in the identification list of the stream channels.

[0111] The video sending device selects and creates multiple reliable stream channels for sending from the identification list of the reliable transmission stream channels of the local end, needs to wait for the data response confirmation of the opposite end for the video stream channel, and performs retransmission for the lost data. The video receiving device creates multiple reliable stream channels for corresponding stream data reception, performs data response confirmation for the stream channels in the identification list of the reliable stream channels of the opposite end, and feeds back the video delay of the current stream channel. In addition, the video receiving device feeds back the time stamp of the received video data for each stream channel, which can be realized through the time stamp and other information of each video stream channel in the video receiving device, or a total video stream channel can be constructed to feed back the packet loss feedback information and the receiving time stamp and other information of each video stream channel through the total video stream channel.

[0112] The video sending device calculates network quality information such as a current network congestion window, a network RTT, and packet loss, and calculates information such as a video frame transmission delay and a delay difference. The video sending device evaluates the amount of video data to be sent next according to the congestion window, selects a video stream channel to be sent next according to the transmission delay of each stream channel, and preferentially sends retransmission packets according to stream channel packet loss information. The video receiving device ensures complete reception of stream data according to the information of each stream channel, and notifies the service layer to collect complete data in a timely manner.

[0113] The above scheme determines the sending order of each group of video data according to the data transmission delay between the video sending device and the video receiving device, and then multiplexes a communication link to sequentially send each group of video data to the video receiving device. Compared with using different communication links for communication transmission of different groups of video data, the scheme can make the video receiving device receive more synchronized groups of video data, that is, improve the synchronization effect between groups of video data.

[0114] Multiple video stream channels are multiplexed and transmitted through a reliable UDP connection, and multiple video streams are carried in one link for unified congestion window, video delay, and service logic management and scheduling. Compared with the traditional method of allocating one communication link for each video stream, the scheme can better guarantee the synchronization of multiple video transmissions.

[0115] Please refer to Figure 10 , Figure 10 is a structural schematic diagram of an embodiment of the video transmission device of the present application. The video transmission device 60 can perform the video transmission method provided by the above-mentioned video transmission method embodiment applied to the video sending device. The video transmission device 60 includes a video acquisition module 61, a sequence determination module 62, and a data sending module 63; the video acquisition module 61 is configured to acquire multiple groups of video data collected by different video acquisition components in the video sending device; the sequence determination module 62 is configured to determine the sending order of each group of video data based on the data transmission delay between the video sending device and the video receiving device; and the data sending module 63 is configured to multiplex a communication link between the video sending device and the video receiving device according to the sending order, and sequentially send each group of video data to the video receiving device.

[0116] The sequence determination module 62 can be the above-mentioned packet time sequence scheduling module, or a combination of the above-mentioned packet scheduling module and network perception and congestion control module.

[0117] The above scheme can make the video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0118] The functions of each module can be referred to the video transmission method applied to the video sending device, which will not be repeated here.

[0119] Please refer to Figure 11 , Figure 11 is a structural schematic diagram of another embodiment of a video transmission device of the present application. The other video transmission device 70 can perform the video transmission method provided by the above-mentioned video transmission method applied to the video receiving device. The other video transmission device 70 comprises a data receiving module 71 and a buffer module 72; the data receiving module 71 is configured to receive, by the video receiving device, a plurality of groups of video data transmitted by the video sending device based on the above-mentioned video transmission method; and the buffer module 72 is configured to buffer each group of video data received.

[0120] The above scheme can make the video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0121] The functions of each module can be referred to the above-mentioned video transmission method applied to the video receiving device, which will not be repeated here.

[0122] Please refer to Figure 12 , Figure 12 is a structural schematic diagram of another embodiment of a video transmission device of the present application. The other video transmission device 70 can perform the video transmission method provided by the above-mentioned video transmission method applied to the video receiving device. The other video transmission device 70 comprises a data receiving module 71 and a buffer module 72; the data receiving module 71 is configured to receive, by the video receiving device, a plurality of groups of video data transmitted by the video sending device based on the above-mentioned video transmission method; and the buffer module 72 is configured to buffer each group of video data received.

[0123] In a specific implementation scenario, the video sending device 80 can include, but is not limited to, a microcomputer, a server, and in addition, a notebook computer, a tablet computer, and the like carrying device, which are not limited herein.

[0124] The first communication module 83 can be any module with a communication function, which is not specifically limited herein. Specifically, the first processor 82 is configured to control itself and the first memory 81 to implement the steps in any of the above video transmission method embodiments. The first processor 82 can also be referred to as a CPU (Central Processing Unit). The first processor 82 can be an integrated circuit chip with signal processing capability. The first processor 82 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the first processor 82 can be implemented by an integrated circuit chip together.

[0125] The above scheme determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then multiplexes a communication link to send each group of video data to the video receiving device in the sending order. Compared with using different communication links to communicate and transmit different groups of video data, the scheme can make the video receiving device receive more synchronized groups of video data, that is, improve the synchronization effect between the groups of video data.

[0126] Please refer to Figure 13 , Figure 13 is a structural schematic diagram of an embodiment of the video receiving device of the present application. The video receiving device 90 includes a second memory 91, a second processor 92, and a second communication module 93. The second communication module 93 is configured to receive multiple groups of video data sent by the video sending device. The second processor 92 is configured to execute program instructions stored in the second memory 91 and buffer the received video data. The second communication module 93, the second memory 91, and the second processor 92 cooperate to implement the above video transmission method applied to the video receiving device.

[0127] In one specific implementation scenario, the video receiving device 90 can include, but is not limited to, an Internet of Things device, a microcomputer, a server, and in addition, the video receiving device 90 can also include a notebook computer, a tablet computer and the like carrying device, which are not limited herein.

[0128] The second communication module 93 can be any module with a communication function, which is not specifically limited herein. Specifically, the second processor 92 is configured to control itself and the second memory 91 to implement the steps in any of the above video transmission method embodiments. The second processor 92 can also be referred to as a CPU (Central Processing Unit). The second processor 92 can be an integrated circuit chip with signal processing capability. The second processor 92 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the second processor 92 can be implemented by an integrated circuit chip together.

[0129] The above scheme determines the sending order of each group of video data according to the data delay between the video sending device and the video receiving device, and then sends each group of video data to the video receiving device in the sending order according to one communication link. Compared with using different communication links to communicate and transmit different groups of video data, the scheme can make the video receiving device receive more synchronized groups of video data, that is, improve the synchronization effect between the groups of video data.

[0130] Please refer to Figure 14 , Figure 14 is a structural schematic diagram of an embodiment of the video transmission system of the present application. The video transmission system 10 includes a video sending device 80 and a video receiving device 90, and the video sending device 80 and the video receiving device 90 are communicatively connected.

[0131] The above scheme can make the video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0132] Please refer to Figure 15 , Figure 15 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 100 has program instructions 101 stored thereon, and the program instructions 101 are executed by a processor to implement the steps in any of the above video transmission method embodiments.

[0133] The above scheme can make the video data received by the video receiving device more synchronized, that is, improve the synchronization effect between the groups of video data.

[0134] In some embodiments, the device provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, they will not be repeated here.

[0135] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For brevity, they will not be repeated here.

[0136] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the above-described device implementation is only schematic. For example, the division of modules or units is only a logical function division, and actual implementation can have another division manner. For example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the coupling or direct coupling or communication connection between the displayed or discussed mutual can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0137] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit. When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the method of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A video transmission method, characterized in that: The video transmission method is applied to a video sending device, and the video transmission method includes: The video sending device includes multiple sets of video data collected by different video acquisition components; Determining a sending order of each group of video data based on a data transmission delay between the video sending device and the video receiving device; Multiplexing the communication link between the video sending device and the video receiving device in the sending order, and sequentially sending each group of the video data to the video receiving device; The video data of different groups are in different video stream channels of the video sending device, and determining the sending order of the video data of each group based on the data transmission delay between the video sending device and the video receiving device includes: Obtaining transmission delays of video data sent by different video stream channels through the communication link within a preset historical time period; Determining, based on the transmission delay of each video stream channel, a sending order of each video data between the video stream channels, wherein the sending order of the video data in the video stream channel with a higher transmission delay is before the video data in the video stream channel with a lower transmission delay; Obtaining a delay difference between transmission delays of each of the video stream channels, wherein a delay difference greater than or equal to a preset maximum delay difference is a target delay difference; the video stream channels include video stream channels corresponding to smaller delay differences and video stream channels corresponding to larger delay differences, and the delay difference of the video stream channels corresponding to the larger delay difference is greater than the delay difference of the video stream channels corresponding to the smaller delay difference; In response to the existence of the target delay difference, the sending of video data in the video stream channel corresponding to the smaller delay difference among the target delay differences is suspended within a preset time period.

2. The method according to claim 1, characterized in that The method further comprises: receiving a video delay fed back by the video receiving device, where the video delay is used to characterize synchronization of video data sent by different video stream channels received by the video receiving device; In response to the video delay being greater than or equal to a preset delay, performing the step of determining a sending order of each group of the video data based on a data transmission delay between the video sending device and the video receiving device; And / or, in response to the video delay being less than the preset delay, maintaining the sending order of the video data between the current video stream channels.

3. The method according to claim 1, characterized in that The method further comprises: Determining the amount of video data that needs to be sent each time in each video stream channel based on the transmission delay of each video stream channel; For each of the video stream channels, a secondary sending order between each video data in the video stream channel is determined.

4. The method according to claim 1, wherein The method further comprises: Sending first handshake information to the video receiving device, so that the video receiving device determines second handshake information based on the first handshake information, wherein the first handshake information includes first identification information of each video stream channel in the video sending device that can be used to send video data, and the second handshake information includes second identification information of each video stream channel in the video receiving device that can be used to receive video data; receiving second handshake information sent by the video receiving device; Based on the second identification information, a video stream channel used by the video sending device to send each of the video data is determined.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: In response to receiving packet loss feedback information sent by the video receiving device, determining data to be retransmitted and a target video stream channel corresponding to the data to be retransmitted; The sending order of the data to be retransmitted in the target video stream channel is placed before other video data in the target video stream channel, and the other video data is non-retransmitted data.

6. The method according to any one of claims 1 to 4, characterized in that Before multiplexing the communication link between the video sending device and the video receiving device in the sending order and sequentially sending each group of the video data to the video receiving device, the method further includes: Acquiring network quality information of the communication link; determining whether the communication link allows data transmission based on the network quality information; In response to the communication link allowing data transmission, the step of multiplexing the communication link between the video transmitting device and the video receiving device according to the transmission order and sequentially transmitting each group of the video data to the video receiving device is performed.

7. A video transmission method, characterized in that: The video transmission method is applied to a video receiving device, and the video transmission method includes: The video receiving device receives multiple sets of video data sent by the video sending device based on the video transmission method according to any one of claims 1 to 6; The received video data are buffered.

8. The method according to claim 7, characterized in that The method further comprises at least one of the following steps: Storing the received multiple groups of video data in the video stream channel corresponding to each group of the video data in the video receiving device; Sending packet loss feedback information to the video sending device, so that the video sending device resends corresponding video data based on the packet loss feedback information; receiving first handshake information sent by the video sending device and sending second handshake information to the video sending device, so that the video sending device determines a video stream channel for sending each video data based on the second handshake information; The video delay is sent to the video sending device so that the video sending device determines the sending order of video data between the video stream channels in the video sending device based on the video delay. The video delay is used to characterize the synchronization of the video data sent by different video stream channels received by the video receiving device.

9. A video sending device, characterized in that: It includes a first communication module, a first memory and a first processor, the first processor is used to execute program instructions stored in the first memory to determine the sending order of each group of video data, the first communication module is used to send each group of video data to a video receiving device, wherein the first communication module, the first memory and the first processor cooperate to implement the video transmission method described in any one of claims 1 to 6.

10. A video receiving device, characterized in that: It includes a second communication module, a second memory and a second processor, the second communication module is used to receive multiple groups of video data sent by a video sending device, the second processor is used to execute program instructions stored in the second memory, and cache the received video data, wherein the second communication module, the second memory and the second processor cooperate to implement the video transmission method described in claim 7 or 8.

11. A video transmission system, characterized in that: The video transmission system includes the video sending device according to claim 9 and the video receiving device according to claim 10, and the video sending device and the video receiving device are communicatively connected.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 6 or the method according to claim 7 or 8 is implemented.

Images