Video transmission scheduling method and device, electronic device and storage medium

Abstract

The application relates to a video transmission scheduling method and device, an electronic device and a storage medium, wherein the video transmission scheduling method comprises the following steps: acquiring a reference I-frame interval duration of a reference to-be-scheduled video channel, determining a target I-frame interval duration according to the reference I-frame interval duration, acquiring a first target I-frame generation time of a target to-be-scheduled video channel, determining a second target I-frame generation time according to the first target I-frame generation time and the target I-frame interval duration, transmitting a plurality of target video frames of the target to-be-scheduled video channel to a target video transmission channel for transmission from the second target I-frame generation time, acquiring to-be-transmitted data packets and target scheduling time stamps of the to-be-transmitted data packets, and performing transmission scheduling according to the target scheduling time stamps, so that the transmission competition problem caused by the superposition of I-frame peak values of multiple channels during transmission between video streams of multiple video channels of the same device is solved, the peak value of the video air interface network card flow of the multi-objective device is reduced, the utilization rate of the wireless air interface bandwidth is improved, and the transmission time consumption and the playing lag are reduced.

Description

Technical Field

[0001] This application relates to the field of network transmission, and in particular to video transmission scheduling methods, apparatus, electronic devices, and storage media. Background Technology

[0002] With technological advancements, scenarios utilizing video data are increasingly common. Devices typically stream video in real-time, periodically sending video frames at a certain frequency. Key I-frames are significantly larger than regular P-frames (usually more than 10 times larger). For example, with a video frame rate of 25 frames per second (FPS) and a Group of Pictures (GOP) of 50 in the video encoding, the network traffic characteristics are as follows: one video frame is generated every 40ms, and one key I-frame is generated every 25 × 50 × 40 = 2000 milliseconds. In practical applications, the Maximum Transmission Unit (MTU) is used to inform the other party of the maximum size of the data service unit that can be accepted, indicating the size of the payload that the sender can accept. If the MTU is too large, the data will be rejected by routers. Therefore, due to MTU limitations, the payload data to be sent needs to be segmented before transmission. For example, at 2Mbps and 15fps H.265 encoding, one video I-frame is approximately 217,917 bytes, and one video P-frame is approximately 13,930 bytes. Using the common 1200-byte packet format, one I-frame is divided into approximately 182 video data packets, and one P-frame is divided into approximately 12 video data packets. A single video frame can only be decoded and played after all the segmented video data packets have arrived.

[0003] Taking the mainstream 4G wireless cameras (IPCs) on the market as an example, wireless IPCs typically connect to a public cloud platform via cellular network base stations. Users' mobile phones access the cloud platform via Wi-Fi or cellular networks to view real-time video and alarm recordings of the monitored area. Due to the limitations on uplink bandwidth imposed by cellular network standards such as 4G (e.g., 4G CAT1 limits the maximum uplink bandwidth to 5Mbps), the multi-channel video stream transmission of multi-camera systems (multiple cameras mounted on one camera) requires very high bandwidth utilization. Simultaneously, the access of video terminal devices to wireless networks is affected by factors such as user installation location, wireless interference, network latency, and wireless resource contention, causing network layer bandwidth fluctuations, further exacerbating problems such as stuttering, frame drops, latency, and desynchronization in video playback.

[0004] Therefore, when multiple video I-frames from a multi-view network camera are transmitted simultaneously, their peak values ​​will overlap, putting great pressure on the air interface bandwidth. This leads to increased packet loss and latency in the wireless air interface, affecting the utilization rate of the air interface network bandwidth. Furthermore, the transmission of multiple video I-frames will also compete with each other, resulting in increased video transmission latency and scheduling chaos in the transport layer, triggering invalid timeout retransmissions (not network packet loss) in the transport layer, further exacerbating the pressure on wireless air interface video transmission.

[0005] Currently, no effective solution has been proposed for the transmission contention problem caused by the superposition of I-frame peaks from multiple channels during the transmission of video streams from multiple video channels on the same device. Summary of the Invention

[0006] This application provides a video transmission scheduling method, apparatus, electronic device, and storage medium to at least solve the transmission contention problem caused by the superposition of I-frame peaks from multiple channels during transmission of video streams from multiple video channels on the same device in related technologies.

[0007] In a first aspect, embodiments of this application provide a video transmission scheduling method.

[0008] In some embodiments, the video transmission scheduling method includes:

[0009] Obtain the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration.

[0010] Obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the interval duration of the target I-frame.

[0011] Starting from the moment the second target I-frame is generated, multiple target video frames corresponding to the target video channel to be scheduled are sequentially sent to the corresponding target video transmission channel for transmission.

[0012] Obtain the data packet to be transmitted corresponding to the target video frame, determine the target scheduling timestamp of the data packet to be transmitted, and schedule the transmission of the data packet to be transmitted according to the target scheduling timestamp.

[0013] In some embodiments, obtaining the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determining the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration, includes:

[0014] Obtain the baseline I-frame interval duration, the number of video channels to be scheduled, and the encoding unit duration corresponding to the baseline video channel to be scheduled, as well as the channel identifier of the target video channel to be scheduled;

[0015] The target I-frame interval duration corresponding to the target video channel to be scheduled is calculated based on the baseline I-frame interval duration, the number of video channels to be scheduled, the encoding unit duration, and the channel identifier to be scheduled.

[0016] In some embodiments, obtaining the data packet to be transmitted corresponding to the target video frame, determining the target scheduling timestamp of the data packet to be transmitted, and scheduling the transmission of the data packet to be transmitted according to the target scheduling timestamp includes:

[0017] Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame. Determine the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, the target encoding timestamp and the channel identifier to be scheduled.

[0018] The data packets to be transmitted are scheduled for transmission based on the target scheduling timestamp.

[0019] In some embodiments, obtaining the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame, and determining the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, the target encoding timestamp, and the channel identifier to be scheduled includes:

[0020] Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame;

[0021] Obtain the video frame to be compared in the video channel to be compared, and the channel identifier to be compared corresponding to the video channel to be compared, and determine the video frame type and the encoding timestamp to be compared corresponding to the video frame to be compared;

[0022] The target scheduling timestamp of the data packet to be transmitted is determined based on the target video frame type, the target encoding timestamp, the channel identifier to be scheduled, the video frame type to be compared, the encoding timestamp to be compared, and the channel identifier to be compared.

[0023] In some embodiments, obtaining the data packet to be transmitted corresponding to the target video frame, determining the target scheduling timestamp of the data packet to be transmitted, and scheduling the transmission of the data packet to be transmitted according to the target scheduling timestamp includes:

[0024] Obtain the data packet to be transmitted corresponding to the target video frame, and determine the target scheduling timestamp of the data packet to be transmitted, the data packet type of the data packet to be transmitted, the current allowed data transmission volume, the total number of data packets to be transmitted in the target video transmission channel, and the total number of data packets to be retransmitted in the target video transmission channel.

[0025] The data packets to be transmitted are scheduled for transmission based on the data packet type, the current allowed data volume, the total number of data packets to be sent, the total number of data packets to be retransmitted, and the target scheduling timestamp.

[0026] In some embodiments, the step of scheduling the transmission of the data packets to be transmitted based on the data packet type, the currently allowed data transmission volume, the total number of data packets to be transmitted, the total number of data packets to be retransmitted, and the target scheduling timestamp includes:

[0027] If the data packet type is a non-retransmission packet type, the total number of accumulated data packets to be sent is greater than zero and / or the total number of accumulated data packets to be retransmitted is greater than zero, and the current allowed data transmission amount is greater than zero, then the data packet to be transmitted is determined to be a non-retransmission data packet.

[0028] Based on the current allowed data transmission volume and the target scheduling timestamp, the non-retransmitted data packet is transmitted, and the number of non-retransmitted data packets corresponding to the transmitted non-retransmitted data packet is obtained. Based on the number of non-retransmitted data packets, the total number of accumulated data packets to be sent and the current allowed data transmission volume are updated.

[0029] In some embodiments, the step of scheduling the transmission of the data packets to be transmitted based on the data packet type, the currently allowed data transmission volume, the total number of data packets to be transmitted, the total number of data packets to be retransmitted, and the target scheduling timestamp includes:

[0030] If the data packet type is a retransmission packet type, the total number of accumulated data packets to be sent is greater than zero and / or the total number of accumulated data packets to be retransmitted is greater than zero, and the current allowed data transmission amount is greater than zero, then the data packet to be transmitted is determined to be a retransmission data packet.

[0031] The retransmitted data packet is transmitted according to the target scheduling timestamp, and the number of retransmitted data packets corresponding to the transmitted retransmitted data packet is obtained. The total number of accumulated data packets to be retransmitted is updated according to the number of retransmitted data packets.

[0032] Secondly, embodiments of this application provide a video transmission scheduling device.

[0033] In some embodiments, the video transmission scheduling device includes an interval duration determination module, a transmission time determination module, a target data transmission module, and a transmission data scheduling module:

[0034] The interval duration determination module is used to obtain the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration.

[0035] The transmission time determination module is used to obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the interval duration of the target I-frame.

[0036] The target data sending module is used to send multiple target video frames corresponding to the target video channel to be scheduled to the corresponding target video transmission channel for transmission, starting from the time the second target I-frame is generated.

[0037] The data transmission scheduling module is used to obtain the data packet to be transmitted corresponding to the target video frame, determine the target scheduling timestamp of the data packet to be transmitted, and perform transmission scheduling on the data packet to be transmitted according to the target scheduling timestamp.

[0038] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video transmission scheduling method as described in the first aspect above.

[0039] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the video transmission scheduling method as described in the first aspect above.

[0040] Compared to related technologies, the video transmission scheduling method, apparatus, electronic device, and storage medium provided in this application, by obtaining the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determining the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration; obtaining the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determining the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the target I-frame interval duration; starting from the generation time of the second target I-frame, sequentially sending multiple target video frames corresponding to the target video channel to be scheduled to the corresponding target video transmission channel for transmission; obtaining the data packet to be transmitted corresponding to the target video frame, and determining the target scheduling timestamp of the data packet to be transmitted, and performing transmission scheduling of the data packet to be transmitted based on the target scheduling timestamp, solves the transmission contention problem caused by the superposition of I-frame peaks of multiple channels during transmission between video streams of multiple video channels on the same device in related technologies. It can reduce the peak traffic of the video air interface network card of multi-camera devices, reduce the transmission time of multiple video I-frames, improve the utilization rate of wireless air interface bandwidth, and reduce video playback stuttering.

[0041] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0043] Figure 1 This is a hardware structure block diagram of a terminal for a video transmission scheduling method according to an embodiment of this application;

[0044] Figure 2 This is a flowchart of a video transmission scheduling method according to an embodiment of this application;

[0045] Figure 3 This is a flowchart of a video transmission scheduling method according to a preferred embodiment of this application;

[0046] Figure 4 This is a structural block diagram of a video transmission scheduling device according to an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal for the video transmission scheduling method according to an embodiment of the present invention. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0051] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the video transmission scheduling method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0053] This embodiment provides a video transmission scheduling method. Figure 2 This is a flowchart of a video transmission scheduling method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0054] Step S201: Obtain the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration.

[0055] The device in this embodiment includes multiple video channels to be scheduled. When a user previews a request to play video from different channels, the video encoding module sequentially starts video encoding for different channels to generate the first I-frame according to the user's request. In order to quickly output the image, no peak-shifting control is performed on the first I-frame of each channel. That is, taking the total number of video channels as N as an example, the device's video encoding module will generate the first I-frames corresponding to N channels almost simultaneously.

[0056] Select a baseline video channel from multiple video channels to be scheduled and determine its corresponding baseline I-frame interval duration. Taking a total of N video channels as an example, number the N video channels and determine their channel identifiers n (n=1, 2, 3...N). The video channel with channel identifier 1 is designated as the baseline video channel to be scheduled, and the time interval between the generation time of its second I-frame and the generation time of its first I-frame is determined as the baseline I-frame interval duration T1. The video channel with channel identifier i is designated as the target video channel to be scheduled, and based on the baseline I-frame interval duration T1, the target I-frame interval duration Ti corresponding to the target video channel to be scheduled is determined.

[0057] Step S202: Obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the target I-frame interval duration.

[0058] This application embodiment performs multi-channel I-frame timing control and adjustment at the source of video generation through a video encoding module. Specifically, it obtains the generation time of the first target I-frame corresponding to the target video channel to be scheduled and starts the second target I-frame timer corresponding to the target video channel to be scheduled. The timer duration is set to the target I-frame interval duration. When the second target I-frame timer expires, the video encoding of the target video channel to be scheduled generates the second video I-frame (second target I-frame) of that channel, and this time is determined as the generation time of the second target I-frame corresponding to the target video channel to be scheduled. It is worth mentioning that in this application embodiment, the target video channel to be scheduled can be one of the N video channels included in the device, or it can include multiple channels. Therefore, in some embodiments, the target I-frame interval duration of multiple target video channels to be scheduled is calculated respectively, and the corresponding second target I-frame timer is started according to the multiple target I-frame interval durations. When the second target I-frame timer expires, video encoding is performed on the corresponding target video channel to be scheduled, and the second video I-frame (second target I-frame) is generated at this time, and this time is determined as the generation time of the second target I-frame corresponding to that video channel. If the timer for the second target I-frame has not expired, non-I-frames will continue to be generated according to the regular video encoding.

[0059] Step S203: Starting from the time the second target I-frame is generated, multiple target video frames corresponding to the target video channel to be scheduled are sequentially sent to the corresponding target video transmission channel for transmission.

[0060] After the second target I-frame of the target video channel to be scheduled is generated, the target video channel to be scheduled generates multiple subsequent video frames according to conventional encoding. These multiple target video frames include the second target I-frame and the subsequent multiple video frames. Starting from the time the second target I-frame is generated, the multiple target video frames corresponding to the target video channel to be scheduled are sequentially sent to the corresponding target video transmission channels for transmission. In some embodiments that include multiple target video channels to be scheduled, when the timer for generating the second target I-frame of each video channel reaches its expiration, the generation time of the second target I-frame for each channel is determined, and the generation time of the second target I-frame for each channel is taken as the start time of its respective I-frame. The second target I-frame and subsequent video frames obtained by video encoding of different video channels are then sequentially sent to the corresponding transmission channels for network transmission.

[0061] Step S204: Obtain the data packet to be transmitted corresponding to the target video frame, determine the target scheduling timestamp of the data packet to be transmitted, and schedule the transmission of the data packet to be transmitted according to the target scheduling timestamp.

[0062] This application embodiment performs transmission timing scheduling on video frames from different channels. A target video frame is divided into multiple data packets to be transmitted. The data packets to be transmitted corresponding to the target video frame are obtained, and the target scheduling timestamp of the data packets to be transmitted is determined based on the target video frame type and target encoding timestamp. The target scheduling timestamp corresponding to each data packet to be transmitted is used to determine the time when the data packet to be transmitted is sent. Therefore, the transmission order of multiple data packets to be transmitted can be scheduled based on the target scheduling timestamp, thereby avoiding excessive transmission contention caused by peak overlap.

[0063] Through the above steps, the reference I-frame interval duration corresponding to the reference video channel to be scheduled is obtained. Based on the reference I-frame interval duration, the target I-frame interval duration corresponding to the target video channel to be scheduled is determined. The generation time of the first target I-frame corresponding to the target video channel to be scheduled is obtained. Based on the generation time of the first target I-frame and the target I-frame interval duration, the generation time of the second target I-frame corresponding to the target video channel to be scheduled is determined. In this way, the generation times of the corresponding non-first I-frames of each channel are different, realizing peak-shifting control of the video frame generation source. Furthermore, starting from the generation time of the second target I-frame, multiple target video frames corresponding to the target video channel to be scheduled are sequentially sent to the corresponding target video transmission channels. The system performs transmission by acquiring the data packet to be transmitted corresponding to the target video frame and determining the target scheduling timestamp of the data packet. Based on the target scheduling timestamp, the transmission of the data packet is scheduled. Since peak-shaving control is performed during video frame generation, the target scheduling timestamp of the data packet to be transmitted will also be staggered, thereby achieving transmission timing scheduling at the multi-channel video frame level and video packet level. This solves the transmission contention problem caused by the superposition of peak values ​​of multiple channel I-frames during transmission between multiple video channels of the same device in related technologies. It can reduce the peak traffic of the video air interface network card of multi-view devices, reduce the transmission time of multiple video I-frames, improve the utilization of wireless air interface bandwidth, and reduce video playback stuttering.

[0064] In some embodiments, step S201 includes:

[0065] Step S2011: Obtain the reference I-frame interval duration, the number of video channels to be scheduled, and the encoding unit duration corresponding to the reference video channel to be scheduled, as well as the channel identifier of the target video channel to be scheduled.

[0066] Furthermore, in this embodiment, in addition to obtaining the baseline I-frame interval duration corresponding to the baseline video channel to be scheduled, the number of video channels to be scheduled, the coding unit duration (GOP), and the channel identifier of the target video channel to be scheduled are also obtained. Continuing from the above, taking the example of a total of N video channels, determining the video channel with channel identifier 1 as the baseline video channel to be scheduled, and determining the baseline I-frame interval duration T1, the number of video channels to be scheduled N, the coding unit duration (GOP), and the channel identifier i of the target video channel to be scheduled are obtained.

[0067] Step S2012: Calculate the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration, the number of video channels to be scheduled, the encoding unit duration, and the channel identifier to be scheduled.

[0068] Furthermore, continuing with the above examples, the target I-frame interval duration Ti corresponding to the target video channel to be scheduled can be calculated using the following formula based on the baseline I-frame interval duration T1, the number of video channels to be scheduled N, the coding unit duration GOP, and the channel identifier i to be scheduled.

[0069] Ti=T1+(i-1)×GOP / N(i=2,3,…N)

[0070] Taking a binocular camera (N=2) device and GOP=4 seconds as an example, then T2=T1+2 seconds;

[0071] Taking a three-lens camera (N=3) device and GOP=4 seconds as an example, then T2= T1+1.3 seconds and T3=T1+2.6 seconds.

[0072] Through the above steps, this application provides a method for accurately calculating the target I-frame interval duration corresponding to the target video channel to be scheduled based on the baseline I-frame interval duration, the number of video channels to be scheduled, the encoding unit duration, and the channel identifier to be scheduled. This method is accurate and feasible, further improving the orderly transmission of video frames during off-peak periods and reducing transmission time.

[0073] In some embodiments, step S204 includes:

[0074] Step S2041: Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame. Determine the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, target encoding timestamp and the channel identifier to be scheduled.

[0075] This application embodiment reads the video frame type and encoding timestamp of video frames in all transmission channels, and marks the scheduling timestamp of the data packet corresponding to the video frame based on the video frame type and encoding timestamp. For example, for video frames of different channels, this application embodiment re-marks the scheduling timestamp corresponding to the video frame data packet according to the video frame type and encoding timestamp size (smaller timestamps correspond to earlier times). The re-marked scheduling timestamps also prioritize smaller timestamps, and the re-marked scheduling timestamps are unique within the transmission module. Further, it obtains the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame. Based on the target video frame type, target encoding timestamp, and the channel identifier to be scheduled, it determines the target scheduling timestamp of the data packet to be transmitted.

[0076] Step S2042: Perform transmission scheduling on the data packets to be transmitted based on the target scheduling timestamp.

[0077] The scheduling timestamp can be used to determine the time of video frame transmission. Therefore, when multiple video frames exist, the transmission order of each video frame / data packet can be determined based on the scheduling timestamp, i.e., the transmission of video frames / data packets is scheduled. Furthermore, in this embodiment, the transmission of the data packet to be transmitted is scheduled based on the target scheduling timestamp.

[0078] Through the above steps, this application provides a specific method for determining the scheduling timestamp, thereby determining the time and order of transmission of multiple channel video frames based on the scheduling timestamp, effectively avoiding transmission conflicts between video frames, improving the utilization rate of wireless air interface bandwidth, and reducing video playback stuttering.

[0079] In some embodiments, step S2041 includes:

[0080] Step S2141: Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame.

[0081] Identify the target video channel to be scheduled, and obtain the data packet to be transmitted corresponding to the target video frame of the channel, as well as the target video frame type and target encoding timestamp.

[0082] Step S2241: Obtain the video frame to be compared in the video channel to be compared, and the channel identifier to be compared corresponding to the video channel to be compared, and determine the video frame type and the encoding timestamp to be compared corresponding to the video frame to be compared.

[0083] Furthermore, a video channel to be compared that is different from the target video channel to be scheduled is identified, and the video frames to be compared in the video channel to be compared, as well as the identification of the video channel to be compared, are obtained. The video frame type and the encoding timestamp to be compared are then determined.

[0084] Step S2341: Determine the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, target encoding timestamp, channel identifier to be scheduled, video frame type to be compared, encoding timestamp to be compared, and channel identifier to be compared.

[0085] In this embodiment, when the target encoding timestamp and the comparison encoding timestamp are the same, further determination is made based on the target video frame type and the comparison video frame type. Specifically, the scheduling timestamp of the data packet corresponding to an I-frame is less than the scheduling timestamp of the data packet corresponding to a non-I-frame. That is: if the target video frame type is I-frame and the comparison video frame type is non-I-frame, then the target scheduling timestamp of the data packet to be transmitted is determined to be less than the scheduling timestamp of the data packet corresponding to the comparison video frame; if the target video frame type is non-I-frame and the comparison video frame type is I-frame, then the scheduling timestamp of the data packet corresponding to the comparison video frame is determined to be less than the target scheduling timestamp of the data packet to be transmitted. When the target video frame type and the comparison video frame type are the same, and the target encoding timestamp and the comparison encoding timestamp are the same, then the following determination is made based on the channel identifier to be scheduled and the channel identifier to be compared: if the channel identifier to be scheduled is less than the channel identifier to be compared, then the target scheduling timestamp of the data packet to be transmitted is determined to be less than the scheduling timestamp of the data packet corresponding to the comparison video frame; if the channel identifier to be scheduled is greater than the channel identifier to be compared, then the target scheduling timestamp of the data packet to be transmitted is greater than the scheduling timestamp of the data packet corresponding to the comparison video frame.

[0086] Through the above steps, this application embodiment marks video frames of different channels with scheduling timestamps based on video frame type and encoding timestamp. By scheduling timestamps, priority transmission of I-frames and orderly transmission are achieved, further reducing the transmission time of multiple video I-frames and improving transmission reliability.

[0087] In some embodiments, step S204 includes:

[0088] Step S2043: Obtain the data packet to be transmitted corresponding to the target video frame, and determine the target scheduling timestamp of the data packet to be transmitted, the data packet type of the data packet to be transmitted, the current allowed data transmission volume, the total number of data packets to be transmitted in the target video transmission channel, and the total number of data packets to be retransmitted in the channel.

[0089] In this embodiment of the application, the data packet to be transmitted corresponding to the target video frame is obtained, and the target scheduling timestamp and data packet type of the data packet to be transmitted are determined. Furthermore, the current allowed data transmission volume L, the total number of data packets to be transmitted in the target video transmission channel D, and the total number of data packets to be retransmitted in the target video transmission channel R are obtained.

[0090] Step S2044: Based on the data packet type, the current allowed data transmission volume, the total number of data packets to be sent, the total number of data packets to be retransmitted, and the target scheduling timestamp, perform transmission scheduling for the data packets to be transmitted.

[0091] The data packet types include retransmission packet types and non-retransmission packet types. The currently allowed data transmission volume is used to characterize the current network transmission capacity and can be determined through real-time calculation. Based on the data packet type, the currently allowed data transmission volume, the total number of data packets to be sent, the total number of data packets to be retransmitted, and the target scheduling timestamp, the transmission scheduling of the data packets to be transmitted is performed.

[0092] Through the above steps, this application embodiment performs transmission scheduling on the data packets to be transmitted based on the data packet type, the current allowed amount of data to be sent, the total number of data packets to be sent, the total number of data packets to be retransmitted, and the target scheduling timestamp. It comprehensively considers the characteristics of the transmitted data and the network carrying capacity, realizes the overall transmission scheduling optimization of the system, further improves the utilization rate of wireless air interface bandwidth, and reduces video playback stuttering.

[0093] In some embodiments, step S2044 includes:

[0094] Step S2144: If the data packet type is a non-retransmission packet type, the total number of data packets to be sent is greater than zero and / or the total number of data packets to be retransmitted is greater than zero, and the current allowed amount of data to be sent is greater than zero, then the data packet to be transmitted is determined to be a non-retransmission data packet.

[0095] In this embodiment of the application, when the total number of data packets to be sent, D, is greater than zero and / or the total number of data packets to be retransmitted, R, is greater than zero, and the current allowed amount of data to be sent, L, is greater than zero, and the data packet type is a non-retransmission packet type, the data packet to be transmitted is determined to be a non-retransmission data packet.

[0096] Step S2244: Based on the current allowed data transmission volume and the target scheduling timestamp, transmit non-retransmitted data packets, obtain the number of non-retransmitted data packets corresponding to the transmitted non-retransmitted data packets, and update the total number of data packets to be transmitted and the current allowed data transmission volume based on the number of non-retransmitted data packets.

[0097] Further, the embodiments of the present application sequentially schedule non-retransmitted data packets according to the sequence of the currently allowable data transmission amount L and the target scheduling timestamp. In specific scheduling, when L is greater than zero, a non-retransmitted data packet with a smaller corresponding target scheduling timestamp is transmitted earlier. The number of non-retransmitted data packets corresponding to the transmitted non-retransmitted data packets is acquired, and meanwhile, the total number of data packets to be transmitted D and the currently allowable data transmission amount L are synchronously reduced according to the actual number of transmitted non-retransmitted data packets. For example, if k (k<D and k<L) non-retransmitted data packets are actually transmitted, D is decreased by k and L is decreased by k in real time.

[0098] Through the above steps, the embodiments of the present application provide a specific transmission scheduling scheme when the data packet type is a non-retransmitted packet type, which has high feasibility and real-time performance and saves resources.

[0099] In some embodiments, step S2044 comprises:

[0100] Step S2344: In a case where the data packet type is a retransmitted packet type, the accumulated total number of data packets to be transmitted is greater than zero and / or the accumulated total number of data packets to be retransmitted is greater than zero, and the currently allowable data transmission amount is greater than zero, determining that the data packet to be transmitted is a retransmitted data packet.

[0101] In the embodiments of the present application, when the accumulated total number of data packets to be transmitted D is greater than zero and / or the accumulated total number of data packets to be retransmitted R is greater than zero, the currently allowable data transmission amount L is greater than zero, and the data packet type is a retransmitted packet type, it is determined that the data packet to be transmitted is a retransmitted data packet.

[0102] Step S2444: Transmit the retransmitted data packets according to the target scheduling timestamp, acquire the number of retransmitted data packets corresponding to the transmitted retransmitted data packets, and update the accumulated total number of data packets to be retransmitted according to the number of retransmitted data packets.

[0103] Further, the embodiments of the present application sequentially schedule the retransmitted data packets according to the sequence of the target scheduling timestamps. In specific scheduling, a retransmitted data packet with a smaller corresponding target scheduling timestamp is transmitted earlier. The number of retransmitted data packets corresponding to the transmitted retransmitted data packets is acquired, and meanwhile, the accumulated total number of data packets to be retransmitted R is synchronously reduced according to the actual number of transmitted retransmitted data packets. For example, if m (m<R) retransmitted data packets are actually transmitted, R is decreased by m in real time. In addition, when (D=0 and R=0) or L=0, and transmission on all channels is completed, the multi-channel video transmission is stopped.

[0104] Through the above steps, the embodiments of the present application provide a specific transmission scheduling scheme when the data packet type is a retransmitted packet type, which has high feasibility and real-time performance and saves resources.

[0105] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0106] Figure 3 This is a flowchart of a video transmission scheduling method according to a preferred embodiment of this application. Figure 3 As shown, the video transmission scheduling method includes the following steps:

[0107] Step S301: Obtain the reference I-frame interval duration, the number of video channels to be scheduled, and the encoding unit duration corresponding to the reference video channel to be scheduled, as well as the channel identifier of the target video channel to be scheduled;

[0108] Step S302: Calculate the target I-frame interval duration corresponding to the target video channel to be scheduled based on the baseline I-frame interval duration, the number of video channels to be scheduled, the encoding unit duration, and the channel identifier to be scheduled.

[0109] Step S303: Obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled; and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the target I-frame interval duration.

[0110] Step S304: Starting from the time the second target I-frame is generated, multiple target video frames corresponding to the target video channel to be scheduled are sequentially sent to the corresponding target video transmission channel for transmission.

[0111] Step S305: Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame; obtain the video frame to be compared in the video channel to be compared, as well as the channel identifier to be compared corresponding to the video channel to be compared, and determine the video frame type to be compared and the encoding timestamp to be compared corresponding to the video frame to be compared.

[0112] Step S306: Determine the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, target encoding timestamp, channel identifier to be scheduled, video frame type to be compared, encoding timestamp to be compared, and channel identifier to be compared.

[0113] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0114] This embodiment also provides a video transmission scheduling device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0115] Figure 4 This is a structural block diagram of a video transmission scheduling device according to an embodiment of this application, such as... Figure 4 As shown, the device includes an interval duration determination module 10, a transmission time determination module 20, a target data transmission module 30, and a transmission data scheduling module 40.

[0116] The interval duration determination module 10 is used to obtain the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration.

[0117] The transmission time determination module 20 is used to obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the interval duration of the target I-frame.

[0118] The target data sending module 30 is used to send multiple target video frames corresponding to the target video channel to be scheduled to the corresponding target video transmission channel for transmission, starting from the time the second target I-frame is generated.

[0119] The data transmission scheduling module 40 is used to obtain the data packet to be transmitted corresponding to the target video frame, determine the target scheduling timestamp of the data packet to be transmitted, and perform transmission scheduling on the data packet to be transmitted according to the target scheduling timestamp.

[0120] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0121] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0122] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0123] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0124] Obtain the baseline I-frame interval duration corresponding to the baseline video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the baseline I-frame interval duration.

[0125] Obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the target I-frame interval duration.

[0126] Starting from the moment the second target I-frame is generated, multiple target video frames corresponding to the target video channel to be scheduled will be sent sequentially to the corresponding target video transmission channel for transmission.

[0127] The system acquires the data packet to be transmitted corresponding to the target video frame, determines the target scheduling timestamp of the data packet, and schedules the transmission of the data packet according to the target scheduling timestamp. It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0128] Furthermore, in conjunction with the video transmission scheduling methods in the above embodiments, this application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the video transmission scheduling methods in the above embodiments.

[0129] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video transmission scheduling method, characterized in that, Includes the following steps: Obtain the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration. Obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the interval duration of the target I-frame. Starting from the moment the second target I-frame is generated, multiple target video frames corresponding to the target video channel to be scheduled are sequentially sent to the corresponding target video transmission channel for transmission. Obtain the data packet to be transmitted corresponding to the target video frame, determine the target scheduling timestamp of the data packet to be transmitted, and schedule the transmission of the data packet to be transmitted according to the target scheduling timestamp.

2. The video transmission scheduling method according to claim 1, characterized in that, The process of obtaining the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determining the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration, includes: Obtain the baseline I-frame interval duration, the number of video channels to be scheduled, and the encoding unit duration corresponding to the baseline video channel to be scheduled, as well as the channel identifier of the target video channel to be scheduled; The target I-frame interval duration corresponding to the target video channel to be scheduled is calculated based on the baseline I-frame interval duration, the number of video channels to be scheduled, the encoding unit duration, and the channel identifier to be scheduled.

3. The video transmission scheduling method according to claim 2, characterized in that, The step of obtaining the data packet to be transmitted corresponding to the target video frame, determining the target scheduling timestamp of the data packet to be transmitted, and scheduling the transmission of the data packet to be transmitted according to the target scheduling timestamp includes: Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame. Determine the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, the target encoding timestamp and the channel identifier to be scheduled. The data packets to be transmitted are scheduled for transmission based on the target scheduling timestamp.

4. The video transmission scheduling method according to claim 3, characterized in that, The step of obtaining the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame, and determining the target scheduling timestamp of the data packet to be transmitted based on the target video frame type, the target encoding timestamp, and the channel identifier to be scheduled includes: Obtain the data packet to be transmitted corresponding to the target video frame, as well as the target video frame type and target encoding timestamp corresponding to the target video frame; Obtain the video frame to be compared in the video channel to be compared, and the channel identifier to be compared corresponding to the video channel to be compared, and determine the video frame type and the encoding timestamp to be compared corresponding to the video frame to be compared; The target scheduling timestamp of the data packet to be transmitted is determined based on the target video frame type, the target encoding timestamp, the channel identifier to be scheduled, the video frame type to be compared, the encoding timestamp to be compared, and the channel identifier to be compared.

5. The video transmission scheduling method according to any one of claims 1 to 4, characterized in that, The step of obtaining the data packet to be transmitted corresponding to the target video frame, determining the target scheduling timestamp of the data packet to be transmitted, and scheduling the transmission of the data packet to be transmitted according to the target scheduling timestamp includes: Obtain the data packet to be transmitted corresponding to the target video frame, and determine the target scheduling timestamp of the data packet to be transmitted, the data packet type of the data packet to be transmitted, the current allowed data transmission volume, the total number of data packets to be transmitted in the target video transmission channel, and the total number of data packets to be retransmitted in the target video transmission channel. The data packets to be transmitted are scheduled for transmission based on the data packet type, the current allowed data volume, the total number of data packets to be sent, the total number of data packets to be retransmitted, and the target scheduling timestamp.

6. The video transmission scheduling method according to claim 5, characterized in that, The step of scheduling the transmission of the data packets to be transmitted based on the data packet type, the currently allowed data transmission volume, the total number of data packets to be transmitted, the total number of data packets to be retransmitted, and the target scheduling timestamp includes: If the data packet type is a non-retransmission packet type, the total number of accumulated data packets to be sent is greater than zero and / or the total number of accumulated data packets to be retransmitted is greater than zero, and the current allowed data transmission amount is greater than zero, then the data packet to be transmitted is determined to be a non-retransmission data packet. Based on the current allowed data transmission volume and the target scheduling timestamp, the non-retransmitted data packet is transmitted, and the number of non-retransmitted data packets corresponding to the transmitted non-retransmitted data packet is obtained. Based on the number of non-retransmitted data packets, the total number of accumulated data packets to be sent and the current allowed data transmission volume are updated.

7. The video transmission scheduling method according to claim 6, characterized in that, The step of scheduling the transmission of the data packets to be transmitted based on the data packet type, the currently allowed data transmission volume, the total number of data packets to be transmitted, the total number of data packets to be retransmitted, and the target scheduling timestamp includes: If the data packet type is a retransmission packet type, the total number of accumulated data packets to be sent is greater than zero and / or the total number of accumulated data packets to be retransmitted is greater than zero, and the current allowed data transmission amount is greater than zero, then the data packet to be transmitted is determined to be a retransmission data packet. The retransmitted data packet is transmitted according to the target scheduling timestamp, and the number of retransmitted data packets corresponding to the transmitted retransmitted data packet is obtained. The total number of accumulated data packets to be retransmitted is updated according to the number of retransmitted data packets.

8. A video transmission scheduling device, characterized in that, It includes an interval duration determination module, a transmission time determination module, a target data transmission module, and a transmission data scheduling module: The interval duration determination module is used to obtain the reference I-frame interval duration corresponding to the reference video channel to be scheduled, and determine the target I-frame interval duration corresponding to the target video channel to be scheduled based on the reference I-frame interval duration. The transmission time determination module is used to obtain the generation time of the first target I-frame corresponding to the target video channel to be scheduled, and determine the generation time of the second target I-frame corresponding to the target video channel to be scheduled based on the generation time of the first target I-frame and the interval duration of the target I-frame. The target data sending module is used to send multiple target video frames corresponding to the target video channel to be scheduled to the corresponding target video transmission channel for transmission, starting from the time the second target I-frame is generated. The data transmission scheduling module is used to obtain the data packet to be transmitted corresponding to the target video frame, determine the target scheduling timestamp of the data packet to be transmitted, and perform transmission scheduling on the data packet to be transmitted according to the target scheduling timestamp.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the video transmission scheduling method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the video transmission scheduling method according to any one of claims 1 to 7 when running.

Images