Audio and video frame sending scheduling method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an audio and video frame sending scheduling method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring an audio frame queue and a video frame queue; the audio frame queue comprises a plurality of audio frames to be sent; the video frame queue comprises a plurality of video frames to be sent; each audio frame and each video frame respectively correspond to timestamps; according to a preset scheduling proportion, frames to be sent are obtained from the audio frame queue and the video frame queue in sequence; and sending the frame to be sent to a receiving end. According to the method, the scheduling proportion of the audio frames and the video frames is preset, so that the situation that normal transmission of the audio frames is blocked due to the fact that the data size of the video frames is large can be avoided, smooth transmission of the audio frames is guaranteed, and the playing continuity of a user is guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device, storage medium and computer program product for scheduling audio and video frame transmission. Background Art

[0002] With the development of Internet technology, various Internet-based applications have emerged in an endless stream, greatly enriching people's daily lives. In particular, the rise of video conferencing and live webcasting has completely changed people's traditional way of communication, making remote communication more convenient and efficient.

[0003] However, the viewing experience of live broadcasts on the Internet also faces many challenges, especially when the network transmission speed is slow or the network environment is unstable. These problems will cause delays in audio and video data, resulting in discontinuity or asynchrony of images and sounds, seriously affecting the audience's overall viewing experience. This delay not only destroys the smoothness of the live broadcast, but may also lead to a decrease in audience satisfaction. Therefore, how to overcome the communication difficulties in a weak network environment has become the key to improving the quality of live broadcasts on the Internet. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, device, electronic device, storage medium, and computer program product for scheduling audio and video frame transmission.

[0005] According to the first aspect of the embodiments of the present disclosure, there is provided a method for scheduling the sending of audio and video frames, including: obtaining an audio frame queue and a video frame queue; the audio frame queue includes a number of audio frames to be sent; the video frame queue includes a number of video frames to be sent; each of the audio frame and video frame has a corresponding time stamp; according to a pre-set scheduling ratio, frames to be sent are obtained from the audio frame queue and the video frame queue in turn; the frames to be sent are the audio frames or the video frames; and the frames to be sent are sent to a receiving end.

[0006] In some exemplary embodiments of the present disclosure, the obtaining of frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio includes: determining an audio scheduling threshold R a and video scheduling threshold R v ; According to the timestamp, obtain R from the audio frame queue in sequence a audio frames as the frames to be sent; in response to obtaining R a audio frames as the frames to be sent, and obtain R from the video frame queue in sequence according to the timestamp. v video frames as the frames to be sent.

[0007] In some exemplary embodiments of the present disclosure, the obtaining of frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio includes: determining an audio scheduling threshold R a , video scheduling threshold R v and count value C; in response to C mod(R a +R v ) <R a , obtain the audio frame with the earliest timestamp from the audio frame queue as the frame to be sent; in response to C mod(R a +R v )≥R a , obtaining the video frame with the earliest timestamp from the video frame queue as the frame to be sent; in response to obtaining the frame to be sent, the count value C is increased by one.

[0008] In some exemplary embodiments of the present disclosure, the method of obtaining the frames to be sent from the audio frame queue and the video frame queue in sequence according to a pre-set scheduling ratio also includes: in response to the audio frame queue having no audio frames to be sent, obtaining the frames to be sent from the video frame queue; in response to the video frame queue having no video frames to be sent, obtaining the frames to be sent from the audio frame queue.

[0009] In some exemplary embodiments of the present disclosure, the obtaining of frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio further includes: obtaining the timestamp T of the earliest audio frame in the audio frame queue. a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the audio frame satisfying T a <T v +D1, determine the audio frame as the frame to be sent; wherein D1 is the first leading threshold.

[0010] In some exemplary embodiments of the present disclosure, the obtaining of frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio further includes: obtaining the timestamp T of the earliest audio frame in the audio frame queue. a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the video frame satisfying T v <T a +D2, determine the video frame as the frame to be sent; wherein D2 is the second leading threshold.

[0011] In some exemplary embodiments of the present disclosure, the method also includes: in response to the number of video frames cached at the receiving end being less than a first cache number, obtaining the frames to be sent from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames; and sending the frames to be sent to the receiving end until the number of video frames cached at the receiving end meets the first cache number.

[0012] In some exemplary embodiments of the present disclosure, the method further includes: in response to being in the target playback phase, obtaining the to-be-sent frames from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames, and sending them to the receiving end; in response to the number of video frames cached by the receiving end satisfying the second cache number, sequentially obtaining the target playback timestamps T from the audio frame queue target The previous audio frame is sent to the receiving end; in response to the target playback timestamp T in the audio frame queue target The previous audio frames are sent to the receiving end, and the target playback timestamps T are obtained from the video frame queue in sequence. target The previous video frame is sent to the receiving end.

[0013] According to the second aspect of the embodiments of the present disclosure, there is provided an audio and video frame sending scheduling device, comprising: a queue acquisition module, configured to acquire an audio frame queue and a video frame queue; the audio frame queue includes a number of audio frames to be sent; the video frame queue includes a number of video frames to be sent; each of the audio frame and video frame has a corresponding time stamp; a scheduling module, configured to acquire frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio; the frames to be sent are the audio frames or the video frames; a sending module, configured to send the frames to be sent to a receiving end.

[0014] According to the third aspect of an embodiment of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement any one of the audio and video frame sending scheduling methods.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the audio and video frame sending scheduling methods.

[0016] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements any one of the audio and video frame sending scheduling methods.

[0017] The audio and video frame transmission scheduling method provided by the embodiment of the present disclosure obtains the frames to be transmitted from the audio frame queue and the video frame queue in sequence according to the preset scheduling ratio, and transmits the frames to be transmitted to the receiving end. Since the scheduling ratio of audio frames to video frames is preset in the method, the situation where the large amount of video frame data blocks the normal transmission of audio frames can be avoided, so as to ensure that the audio frames can be transmitted smoothly, thereby ensuring the continuity of user playback.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A schematic diagram showing an exemplary system architecture to which the method according to an embodiment of the present disclosure can be applied is shown.

[0021] Figure 2 The present invention is a flowchart of a method for scheduling audio and video frame transmission according to an exemplary embodiment.

[0022] Figure 3 The process of the audio and video frame scheduling method according to an example is shown Figure 1 .

[0023] Figure 4 The process of the audio and video frame scheduling method according to an example is shown Figure 2 .

[0024] Figure 5 The present invention is a flowchart of a method for scheduling audio and video frames when cache frames are insufficient according to an example.

[0025] Figure 6 It is a flowchart of a method for scheduling audio and video frames in a target playback phase according to an example.

[0026] Figure 7 The present invention is a block diagram of a device for scheduling audio and video frame transmission according to an exemplary embodiment.

[0027] Figure 8 It is a schematic diagram showing the structure of an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0029] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0030] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0033] Figure 1 A schematic diagram showing an exemplary system architecture to which the method according to an embodiment of the present disclosure can be applied is shown.

[0034] like Figure 1As shown, the system architecture may include a server 101, a network 102, a terminal device 103, a terminal device 104, and a terminal device 105. The network 102 is used to provide a medium for a communication link between the terminal device 103, the terminal device 104, or the terminal device 105 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0035] The server 101 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal device 103, the terminal device 104, or the terminal device 105. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device 103, the terminal device 104, or the terminal device 105.

[0036] Terminal device 103, terminal device 104 and terminal device 105 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a wearable smart device, a virtual reality device, an augmented reality device, etc., but are not limited thereto.

[0037] It should be understood that Figure 1 The number of terminal devices 103, terminal devices 104, terminal devices 105, networks 102 and servers 101 are merely illustrative. Server 101 may be a physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, any number of terminal devices, networks and servers may be provided.

[0038] Below, each step of the method in the exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

[0039] Figure 2 The present invention is a flowchart of a method for scheduling audio and video frame transmission according to an exemplary embodiment. Figure 2 The method provided in the embodiment is applied to the client and can be executed by any electronic device, such as the above Figure 1 The terminal device in Figure 1 Servers in, or Figure 1 The terminal device and the server in the embodiment are jointly executed, but the present disclosure is not limited to this.

[0040] In step S210, an audio frame queue and a video frame queue are obtained; the audio frame queue includes a number of audio frames to be sent; the video frame queue includes a number of video frames to be sent; each of the audio frame and video frame has a corresponding time stamp.

[0041] In the disclosed embodiments, during video conferencing and live webcasting, the live broadcasting party is responsible for generating audio and video signals containing the broadcast content. These signals are processed into a series of video frames and audio frames, each of which is a basic unit of the live broadcast content. Video frames and audio frames represent image and sound information within a specific period of time, respectively. Next, these audio and video frames are encoded and converted into data packets suitable for network transmission. The encoded audio and video frames are sequentially transmitted to the viewing party via the Internet. When the data packets arrive at the viewing party, the receiving party decodes the received audio and video data. The decoded audio and video frames are reassembled and played in the correct time order, thereby providing a smooth and synchronized viewing experience.

[0042] During the network transmission of the audio and video frames, it involves how to schedule the sending order of audio frames and video frames to ensure that the receiver can meet the playback conditions and obtain a smooth viewing experience. This method of sending and scheduling audio and video frames is particularly important in a weak network environment and can significantly improve the viewing experience of live broadcasts. Here, the network transmission process can be the communication between the server and the client, or it can be the communication between two connected servers.

[0043] In the disclosed embodiment, the audio frame queue contains multiple audio frames to be sent. Each audio frame is an audio data block after encoding and represents the sound information within a specific period of time. Each audio frame has a corresponding timestamp. The timestamp is used to identify the specific time point when the audio frame should be played. The audio frame queue arranges the audio frames in the order of the timestamps.

[0044] In the disclosed embodiment, a video frame queue contains multiple video frames to be sent. Each video frame is an image data block after encoding and represents a frame. Each video frame has a corresponding timestamp. The timestamp is used to identify the specific time point when the video frame should be played. The video frame queue arranges the video frames in the order of the timestamps.

[0045] This timestamp is not only used to determine the correct playback time of each frame, but also to achieve accurate synchronization between audio frames and video frames. During the actual transmission process, the system will decide which frame to send first based on the timestamp, thereby ensuring that the audio and video content received by the receiver is synchronized.

[0046] In step S220, frames to be sent are obtained from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio; the frames to be sent are the audio frames or the video frames.

[0047] In the related technology, the current scheduling method commonly used in the industry is to strictly interleave the transmission according to the timestamps of the audio and video frames, that is, the audio and video frames with earlier timestamps are sent first. In this scheduling method, video frames with larger data volumes may block the transmission of audio frames until the video frames themselves are sent. In a weak network environment, this strategy may cause viewers to be unable to receive complete audio or video frames for a long time, thereby causing freezes during viewing.

[0048] In the embodiment of the present disclosure, in order to solve the problems existing in the related art, a scheduling ratio between audio frames and video frames is preset. The scheduling ratio is used to set the ratio of the number of frames that alternate between audio frames and video frames. The scheduling module sequentially obtains the audio frame or video frame with the earliest timestamp in the audio frame queue and the video frame queue as the frame to be sent according to the preset scheduling ratio. The frame to be sent is a frame selected by the scheduling module to be sent, which can be a selected audio frame or a selected video frame.

[0049] In the disclosed embodiment, since the data volume of audio frames is smaller than that of video frames, sending audio and video frames according to the scheduling ratio can avoid the situation where the audio frames are blocked from being sent normally due to the larger data volume of video frames, thereby ensuring that the audio frames can be sent smoothly. Under the premise of transmitting the same amount of data, this method adjusts the order of data sending, minimizing the damage to the audience experience caused by the weak network environment.

[0050] It should be noted that the scheduling method provided by the present disclosure is mainly used in the normal playback stage after the live video broadcast has been completed, that is, the receiving end has cached enough audio and video frames for the start of the broadcast. However, for the start of broadcast, rebroadcast or other situations where the number of audio and video frames cached by the receiving end is insufficient, other scheduling methods can be used for frame scheduling. For example, the interleaved sending method of the prior art is used.

[0051] In an exemplary embodiment, the audio scheduling threshold R may be set for audio frames and video frames respectively. a and video scheduling threshold R v The dispatch ratio is R a : R v , for example, it can be set to 1:1, 2:1 or 3:2. 1:1 means sending one audio frame and one video frame, and so on; 2:1 means sending two audio frames and one video frame, and so on; 3:2 means sending three audio frames and two video frames, and so on.

[0052] In an exemplary embodiment, if the selected queue (audio frame queue or video frame queue) has no data waiting to be sent during the scheduling process, another queue is automatically selected to obtain the frame to be sent. That is, in response to the audio frame queue having no audio frame to be sent, the frame to be sent is obtained from the video frame queue; in response to the video frame queue having no video frame to be sent, the frame to be sent is obtained from the audio frame queue.

[0053] Since the present disclosure adopts a strategy of sending corresponding audio and video frames in sequence according to a preset scheduling ratio. Therefore, it may cause the problem of the audio frame and video frame to be sent being out of sync in timing, that is, the audio frame is sent ahead of the video frame, or the video frame is sent ahead of the audio frame. A certain timing advance is allowed, but if the timing advances too much, it may cause the audio and video content to be out of sync. In order to avoid related situations, the present disclosure is provided with an advance threshold to limit the scheduling of related audio and video frames. When the frame in front of the timing exceeds the advance threshold, this type of frame is no longer selected as the frame to be sent, and the frame in the back of the timing is directly selected as the frame to be sent.

[0054] In an exemplary embodiment, the timestamp T of the earliest audio frame in the audio frame queue is obtained. a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the audio frame satisfying T a <T v +D1, determine the audio frame as the frame to be sent; wherein D1 is the first leading threshold.

[0055] In this example, the audio frame timing exceeds the video frame timing. A first leading threshold D1 is preset. When the timestamp of the audio frame meets T a <T v +D1, then the audio frame is obtained from the audio frame queue as the frame to be sent according to the disclosed strategy; when the timestamp of the audio frame meets T a ≥T v +D1, the audio frames are no longer obtained from the audio frame queue as the frames to be sent, but the video frames are obtained from the video frame queue as the frames to be sent.

[0056] In an exemplary embodiment, the timestamp T of the earliest audio frame in the audio frame queue is obtained. a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the video frame satisfying T v <T a +D2, determine the video frame as the frame to be sent; wherein D2 is the second leading threshold.

[0057] In this example, the video frame timing exceeds the audio frame timing. A second leading threshold D2 is preset. When the timestamp of the video frame meets T v <T a +D2, then according to the disclosed strategy, a video frame is obtained from the video frame queue as a frame to be sent; when the timestamp of the video frame satisfies T v ≥T a +D2, the video frame is no longer obtained from the video frame queue as the frame to be sent, but the audio frame is obtained from the audio frame queue as the frame to be sent.

[0058] In an exemplary embodiment, an audio advance time threshold D is set. audio and the leading time threshold D of both parties both The audio advance time threshold D audio , indicating the timestamp threshold that allows audio to be ahead of video. The time threshold D both , represents the time stamp threshold of both audio and video. Among them, the first advanced threshold D1=max(D audio , D both ), the second leading threshold D2 = D both .

[0059] In step S230, the frame to be sent is sent to the receiving end.

[0060] In the disclosed embodiment, the frames to be sent obtained according to the scheduling in step S220 are sent to the receiving end according to the scheduling order.

[0061] The audio and video frame transmission scheduling method provided by the embodiment of the present disclosure obtains the frames to be transmitted from the audio frame queue and the video frame queue in turn according to the preset scheduling ratio, and transmits the frames to be transmitted to the receiving end. Since the scheduling ratio of audio frames to video frames is preset in this method, the situation where the large amount of video frame data blocks the normal transmission of audio frames can be avoided, so as to ensure that the audio frames can be transmitted smoothly, thereby ensuring the continuity of user playback. At the same time, by setting the advance threshold, the problem of asynchronism in the transmission timing of audio frames and video frames can be avoided.

[0062] Figure 3 The process of the audio and video frame scheduling method according to an example is shown Figure 1 .like Figure 3 As shown, step S220 may include the following steps.

[0063] In step S310, the audio scheduling threshold R is determined a and video scheduling threshold R v .

[0064] In the embodiment of the present disclosure, the preset audio scheduling threshold R is determined respectively. a and the video scheduling threshold Rv The audio scheduling threshold R a and the video scheduling threshold R v are integers, representing the number of audio frames and video frames sent in a scheduling period. That is, the scheduling ratio is R a： R v .

[0065] In step S320, according to the timestamp, R is obtained from the audio frame queue in sequence. a audio frames as the frames to be sent.

[0066] In the embodiment of the present disclosure, audio frames are scheduled first and then video frames are scheduled according to the scheduling ratio in a scheduling cycle. First, according to the timestamps corresponding to each audio frame, R is obtained from the audio frame queue. a The audio frame with the earliest timestamp is determined as the frame to be sent according to the time sequence of each acquired audio frame.

[0067] In step S330, in response to obtaining R a audio frames as the frames to be sent, and obtain R from the video frame queue in sequence according to the timestamp. v video frames as the frames to be sent.

[0068] In the embodiment of the present disclosure, in a scheduling cycle, audio frames are scheduled first and then video frames are scheduled according to the scheduling ratio. After the audio frames are scheduled, R is obtained from the video frame queue according to the timestamps corresponding to each video frame. v The video frame with the earliest timestamp is determined as the frame to be sent according to the time sequence of each acquired video frame.

[0069] In an exemplary embodiment, the scheduling ratio is set to 3:2. In one scheduling cycle, the three audio frames with the earliest timestamps are first obtained from the audio frame queue, and then the two video frames with the earliest timestamps are obtained from the video frame queue as the frames to be sent.

[0070] In an exemplary embodiment, if the selected queue (audio frame queue or video frame queue) has no data waiting to be sent during the scheduling process, another queue is automatically selected to obtain the frame to be sent. That is, in response to the audio frame queue having no audio frame to be sent, the frame to be sent is obtained from the video frame queue; in response to the video frame queue having no video frame to be sent, the frame to be sent is obtained from the audio frame queue.

[0071] Figure 4 The process of the audio and video frame scheduling method according to an example is shown Figure 2 .like Figure 4 As shown, step S220 may include the following steps.

[0072] In step S410, the audio scheduling threshold R is determined a , video scheduling threshold R v and count value C.

[0073] In the embodiment of the present disclosure, the preset audio scheduling threshold R is determined respectively. a and the video scheduling threshold R v The audio scheduling threshold R a and the video scheduling threshold R v are integers, representing the number of audio frames and video frames sent in a scheduling period. That is, the scheduling ratio is R a : R v .

[0074] In the disclosed embodiment, the audio and video frames are scheduled by counting the count value C obtained by the counter. The counter counts according to the number of frames to be sent (including audio frames and video frames). That is, each time a frame to be sent is obtained from the audio frame queue or the video frame queue, a count is performed.

[0075] In an exemplary embodiment, the counter does not count during the start-up or replay phase, but starts counting from the normal play phase.

[0076] In step S420, in response to C mod (R a +R v ) <R a , obtain the audio frame with the earliest timestamp from the audio frame queue as the frame to be sent.

[0077] In the embodiment of the present disclosure, according to the above determined audio scheduling threshold R a , video scheduling threshold R v and count value C, based on a predetermined scheduling decision algorithm, determine the queue to obtain the frames to be sent. The scheduling decision algorithm is based on the following judgment formula:

[0078] C mod(R a +R v ) <R a Audio Frame

[0079] When the count value C satisfies C mod(R a +R v ) <R a , the audio frame with the earliest timestamp in the audio frame queue is used as the frame to be sent.

[0080] In step S430, in response to C mod (R a +R v )≥R a, obtaining the video frame with the earliest timestamp from the video frame queue as the frame to be sent.

[0081] In the embodiment of the present disclosure, according to the audio scheduling threshold R determined above a , video scheduling threshold R v and count value C, based on a predetermined scheduling decision algorithm, determine the queue to obtain the frames to be sent. The scheduling decision algorithm is based on the following judgment formula:

[0082] C mod(R a +R v )≥R a Video Frame

[0083] When the count value C satisfies C mod(R a +R v )≥R a , then the video frame with the earliest timestamp in the video frame queue is used as the frame to be sent.

[0084] In an exemplary embodiment, the audio scheduling threshold R is determined a is 3, the video scheduling threshold R v is 2. When the count value C is 12, 12mod 5<3 is satisfied, then the audio frame with the earliest timestamp in the audio frame queue is used as the frame to be sent. When the count value C is 13, 13mod 5≥3 is satisfied, then the video frame with the earliest timestamp in the video frame queue is used as the frame to be sent.

[0085] In step S440, in response to obtaining the frame to be sent, the count value C is increased by one.

[0086] In the embodiment of the present disclosure, no matter the scheduling module obtains a frame to be sent from the audio frame queue or the video frame queue, the count value C in the counter is increased by 1. After completing the above steps S410 to S440 to obtain a frame to be sent, return to the above step S410, and the next frame to be sent is obtained in this cycle to implement the scheduling process.

[0087] In an exemplary embodiment, if the selected queue (audio frame queue or video frame queue) has no data waiting to be sent during the scheduling process, another queue is automatically selected to obtain the frame to be sent. That is, in response to the audio frame queue having no audio frame to be sent, the frame to be sent is obtained from the video frame queue; in response to the video frame queue having no video frame to be sent, the frame to be sent is obtained from the audio frame queue.

[0088] It should be noted that when there is no data waiting to be sent in the selected queue and a frame to be sent is obtained from another queue, the count value of the counter will count normally and will not stop counting because of obtaining the frame to be sent from another queue.

[0089] The audio and video frame transmission scheduling method provided by the embodiment of the present disclosure provides multiple implementation plans for audio and video frame scheduling according to a preset scheduling ratio. By setting the audio scheduling threshold and the video scheduling threshold, the number of audio frames and video frames scheduled in a scheduling cycle can be directly set. On this basis, the sent frames are further counted based on the counter, which can ensure that when there is no data waiting to be sent in the selected queue, the frames to be sent can be obtained from another queue.

[0090] Figure 5 FIG. 1 is a flowchart of a method for scheduling audio and video frames when cache frames are insufficient according to an example. Figure 5 As shown in the aforementioned Figure 2 Based on the audio and video frame sending scheduling method shown, the method can also include the following steps.

[0091] In step S510, in response to the number of video frames cached at the receiving end being less than the first cache number, the frames to be sent are obtained from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames.

[0092] In some business scenarios, during the normal playback phase of audio and video transmission, there is a minimum requirement for the number of video frames cached by the receiving end. If the receiving end does not meet the minimum number requirement, it may face situations such as interruptions and freezes that affect the user's viewing experience. Figure 2 On the basis of the audio and video frame sending scheduling method shown, a special scheduling algorithm arrangement is also given for this situation to meet the minimum number requirement as soon as possible and avoid the risks of interruption, freeze, etc.

[0093] In an embodiment of the present disclosure, when it is detected that the number of video frames cached by the receiving end is less than the first cache number, the audio and video frames are sent according to the aforementioned interleaved scheduling method. The interleaved scheduling method is to obtain the frames to be sent from the audio frame queue and the video frame queue in the order of the timestamps corresponding to the audio frames and the video frames, that is, the audio and video frames with earlier timestamps are sent first. Among them, the first cache number is the minimum number requirement for the receiving end to cache video frames set in the business scenario. Depending on the needs of different business scenarios, the minimum number requirement may be different. The first cache number and the number of video frames cached by the current receiving end can be obtained in a variety of ways, and the present disclosure does not limit the method of obtaining them.

[0094] In step S520, the to-be-sent frames are sent to the receiving end until the number of video frames cached at the receiving end meets the first cache number.

[0095] In the embodiment of the present disclosure, the audio and video frames are sent to the receiving end according to the above interleaving scheduling method until the number of video frames cached by the receiving end meets the first cache number, and then the above Figure 2 The audio and video frame sending scheduling method shown performs audio and video scheduling.

[0096] Figure 6 is a flow chart of a method for scheduling audio and video frames in a target playback phase according to an example. Figure 6 As shown in the aforementioned Figure 2 Based on the audio and video frame sending scheduling method shown, the method can also include the following steps.

[0097] In step S610, in response to being in the target playback phase, the frames to be sent are obtained from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames, and sent to the receiving end.

[0098] As mentioned above, the scheduling method provided by the present disclosure is mainly used in the normal playback stage after the live video broadcast has been completed, that is, the receiving end has cached enough audio and video frames for the start of the broadcast. However, for the start of broadcast, rebroadcast or other situations where the number of audio and video frames cached by the receiving end is insufficient, other scheduling methods can be used for frame scheduling.

[0099] In the disclosed embodiment, different scheduling strategies are provided for the start and replay stages. First, the current playback stage is determined. In response to being in the target playback stage, the Figure 6 The target playback phase can be the start-up phase or the replay phase. For the start-up and replay phases, the number of video frames cached by the receiving end is often required to meet the minimum number N. v Required, and the target playback timestamp T is cached target The minimum number N v It can be the same as the aforementioned first cache number, or different.

[0100] In the embodiment of the present disclosure, in view of the relevant business requirements of the above-mentioned target playback stage, when it is determined that the current stage is the target playback stage, the audio and video frames are sent according to the above-mentioned interleaving scheduling method. The interleaving scheduling method is to obtain the frames to be sent from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames, that is, the audio and video frames with earlier timestamps are sent first.

[0101] In step S620, in response to the number of video frames cached by the receiving end satisfying the second cache number, the target playback timestamps T target The previous audio frame is sent to the receiving end.

[0102] In the embodiment of the present disclosure, the audio and video frames are sent according to the sending scheduling strategy of step S610 until the number of video frames cached by the receiving end meets the second cache number. The second cache number is the minimum number N of the aforementioned start and replay stages. v Requirements. Depending on the needs of different business scenarios, the minimum number N v The requirements may be different. The second cache number and the number of video frames cached by the current receiving end may be obtained in a variety of ways, and the present disclosure does not limit the acquisition method.

[0103] In the embodiment of the present disclosure, in response to the number of cached video frames at the receiving end satisfying the second cache number, it is detected whether the timestamp of the currently sent audio and video frame satisfies the target playback timestamp T target If the target playback timestamp T is met target If the target playback timestamp T is not met, steps S620 and S630 can be skipped. target , the sending schedule is performed in the order of audio frames first and then video frames. Specifically, the target playback timestamps T are obtained from the audio frame queue in sequence. target The previous audio frames are determined as frames to be sent to the receiving end according to the time sequence of each acquired audio frame.

[0104] In step S630, in response to the target playback timestamp T in the audio frame queue, target The previous audio frames are sent to the receiving end, and the target playback timestamps T are obtained from the video frame queue in sequence. target The previous video frame is sent to the receiving end.

[0105] In the embodiment of the present disclosure, step S620 is executed, and the target playback timestamp T in the audio frame queue is target The previous audio frame has been sent to the receiving end, and then the video frame is sent. That is, the target playback timestamp T is obtained from the video frame queue in sequence. target The previous video frames are determined as frames to be sent to the receiving end according to the time sequence of each acquired video frame.

[0106] The audio and video frame transmission scheduling method provided in the embodiment of the present disclosure provides a supplementary scheduling scheme for starting broadcast, rebroadcast and other situations where the receiving end has a minimum number of buffered video frames. Through the supplementary scheduling scheme, video frames can be sent as soon as possible for the above situations to meet the buffer number requirements of the receiving end, and switch to Figure 2 Based on this, the audio and video frame sending scheduling method can take into account the business needs of each stage in the live video broadcast.

[0107] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0108] Figure 7 FIG. 1 is a block diagram of a device for scheduling audio and video frame transmission according to an exemplary embodiment. Figure 7 The audio and video frame sending scheduling device 700 may include: a queue acquisition module 710, a scheduling module 720 and a sending module 730.

[0109] The queue acquisition module 710 is configured to acquire an audio frame queue and a video frame queue; the audio frame queue includes a number of audio frames to be sent; the video frame queue includes a number of video frames to be sent; each of the audio frame and video frame has a corresponding time stamp.

[0110] The scheduling module 720 is configured to obtain frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio; the frames to be sent are the audio frames or the video frames.

[0111] The sending module 730 is configured to send the frame to be sent to the receiving end.

[0112] In some exemplary embodiments of the present disclosure, the scheduling module 720 is further configured to determine an audio scheduling threshold R a and video scheduling threshold R v ; According to the timestamp, obtain R from the audio frame queue in sequence a audio frames as the frames to be sent; in response to obtaining R a audio frames as the frames to be sent, and obtain R from the video frame queue in sequence according to the timestamp. v video frames as the frames to be sent.

[0113] In some exemplary embodiments of the present disclosure, the scheduling module 720 is further configured to determine an audio scheduling threshold R a , video scheduling threshold R v and count value C; in response to C mod(R a +R v ) <R a , obtain the audio frame with the earliest timestamp from the audio frame queue as the frame to be sent; in response to C mod(R a +R v )≥R a , obtaining the video frame with the earliest timestamp from the video frame queue as the frame to be sent; in response to obtaining the frame to be sent, the count value C is increased by one.

[0114] In some exemplary embodiments of the present disclosure, the scheduling module 720 is also configured to obtain the to-be-sent frames from the video frame queue in response to the audio frame queue having no audio frames to be sent; and to obtain the to-be-sent frames from the audio frame queue in response to the video frame queue having no video frames to be sent.

[0115] In some exemplary embodiments of the present disclosure, the scheduling module 720 is further configured to obtain the timestamp T of the earliest audio frame in the audio frame queue. a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the audio frame satisfying T a <T v +D1, determine the audio frame as the frame to be sent; wherein D1 is the first leading threshold.

[0116] In some exemplary embodiments of the present disclosure, the scheduling module 720 is further configured to obtain the timestamp T of the earliest audio frame in the audio frame queue. a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the video frame satisfying T v <T a +D2, determine the video frame as the frame to be sent; wherein D2 is the second leading threshold.

[0117] In some exemplary embodiments of the present disclosure, the scheduling module 720 is also configured to obtain the frames to be sent from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames in response to the number of video frames cached at the receiving end being less than the first cache number; and send the frames to be sent to the receiving end until the number of video frames cached at the receiving end meets the first cache number.

[0118] In some exemplary embodiments of the present disclosure, the scheduling module 720 is further configured to, in response to being in the target playback phase, obtain the to-be-sent frames from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frame and the video frame, and send them to the receiving end; in response to the number of video frames cached by the receiving end satisfying the second cache number, sequentially obtain the target playback timestamps T from the audio frame queue. target The previous audio frame is sent to the receiving end; in response to the target playback timestamp T in the audio frame queue target The previous audio frames are sent to the receiving end, and the target playback timestamps T are obtained from the video frame queue in sequence. target The previous video frame is sent to the receiving end.

[0119] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0120] Refer to the following Figure 8 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0121] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0122] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 The steps shown.

[0123] For example, electronic devices can achieve Figure 2 The steps shown.

[0124] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 821 and / or a cache memory unit 822 , and may further include a read-only memory unit (ROM) 823 .

[0125] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, such program modules 825 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0126] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0127] The electronic device 800 may also communicate with one or more external devices 870 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0128] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0129] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0130] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the method in the above embodiment is implemented.

[0131] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0132] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for scheduling audio and video frame transmission, characterized in that: include: Acquire an audio frame queue and a video frame queue; the audio frame queue includes a number of audio frames to be sent; the video frame queue includes a number of video frames to be sent; each of the audio frame and the video frame has a corresponding time stamp; According to a preset scheduling ratio, frames to be sent are obtained from the audio frame queue and the video frame queue in sequence; the frames to be sent are the audio frames or the video frames; The frame to be sent is sent to a receiving end.

2. The method according to claim 1, characterized in that: The obtaining the frames to be sent from the audio frame queue and the video frame queue in sequence according to the preset scheduling ratio includes: Determine the audio scheduling threshold R a and video scheduling threshold R v ; According to the timestamp, R is obtained from the audio frame queue in sequence. a audio frames as the frames to be sent; In response to obtaining R a audio frames as the frames to be sent, and obtain R from the video frame queue in sequence according to the timestamp. v video frames as the frames to be sent.

3. The method according to claim 1, characterized in that The obtaining the frames to be sent from the audio frame queue and the video frame queue in sequence according to the preset scheduling ratio includes: Determine the audio scheduling threshold R a , video scheduling threshold R v and count value C; In response to C mod(R a +R v ) <R a , obtaining the audio frame with the earliest timestamp from the audio frame queue as the frame to be sent; In response to C mod(R a +R v )≥R a , obtaining the video frame with the earliest timestamp from the video frame queue as the frame to be sent; In response to obtaining the frame to be sent, the count value C is increased by one.

4. The method according to claim 1, 2 or 3, characterized in that: The method of sequentially acquiring frames to be sent from the audio frame queue and the video frame queue according to a preset scheduling ratio also includes: In response to the audio frame queue having no audio frame to be sent, acquiring the frame to be sent from the video frame queue; In response to the video frame queue having no video frame to be sent, obtaining the frame to be sent from the audio frame queue.

5. The method according to claim 1, characterized in that The method of sequentially acquiring frames to be sent from the audio frame queue and the video frame queue according to a preset scheduling ratio also includes: Get the timestamp T of the earliest audio frame in the audio frame queue a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the timestamp of the audio frame satisfying T a <T v +D1, determine the audio frame as the frame to be sent; wherein D1 is the first leading threshold.

6. The method according to claim 1, characterized in that The method of sequentially acquiring frames to be sent from the audio frame queue and the video frame queue according to a preset scheduling ratio also includes: Get the timestamp T of the earliest audio frame in the audio frame queue a ; Get the timestamp T of the earliest video frame in the video frame queue v ; In response to the time stamp of the video frame satisfying T v <T a +D2, determine the video frame as the frame to be sent; wherein D2 is the second leading threshold.

7. The method according to claim 1, characterized in that The method further comprises: In response to the number of video frames cached by the receiving end being less than the first cache number, obtaining the frames to be sent from the audio frame queue and the video frame queue in the order of the timestamps corresponding to the audio frames and the video frames; The frames to be sent are sent to the receiving end until the number of video frames cached at the receiving end meets the first cache number.

8. The method according to claim 1, characterized in that: The method further comprises: In response to being in the target playback phase, obtaining the to-be-sent frames from the audio frame queue and the video frame queue in the order of the corresponding timestamps of the audio frames and the video frames, and sending them to the receiving end; In response to the number of video frames cached by the receiving end satisfying the second cache number, the target playback timestamps T are sequentially obtained from the audio frame queue. target The previous audio frame is sent to the receiving end; In response to the target playback timestamp T in the audio frame queue target The previous audio frames are sent to the receiving end, and the target playback timestamps T are obtained from the video frame queue in sequence. target The previous video frame is sent to the receiving end.

9. An audio and video frame transmission scheduling device, characterized in that: include: A queue acquisition module is configured to acquire an audio frame queue and a video frame queue; the audio frame queue includes a number of audio frames to be sent; the video frame queue includes a number of video frames to be sent; each of the audio frame and the video frame has a corresponding time stamp; A scheduling module is configured to obtain frames to be sent from the audio frame queue and the video frame queue in sequence according to a preset scheduling ratio; the frames to be sent are the audio frames or the video frames; The sending module is configured to send the frame to be sent to the receiving end.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement the audio and video frame sending scheduling method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the audio and video frame sending scheduling method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the audio and video frame sending scheduling method as described in any one of claims 1 to 8 is implemented.

Images