Audio and video data transmission method and related device

By detecting the cache status of the audio and video data receiving end and adjusting the data transmission strategy, the problem that the data sending end is difficult to perceive the client's cache situation, and improving the audio and video data transmission efficiency and playback effect.

CN119071275APending Publication Date: 2024-12-03TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
CN202310632510.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the process of audio and video data transmission, the data sending end is difficult to sense the client's player cache in a timely manner, resulting in the inability to effectively adjust the data transmission strategy, affecting the data transmission efficiency and audio and video playback effect.

Method used

By sending a collection of traffic messages to the data receiver, and determining the integrity and playback order of the audio and video frames received by the receiving end based on the received response message, the playable frames are detected, thereby predicting the cache status of the receiver and adjusting the data transmission strategy .

Benefits of technology

It improves data transmission efficiency and audio and video playback quality, adapts to the business experience needs in audio and video live broadcast scenarios, and improves the audio and video service quality of service providers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071275A_ABST
    Figure CN119071275A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and provides an audio and video data transmission method and a related device, which are used for improving the data transmission efficiency, and the method comprises the following steps: a cloud server sends the respective flow message set of each audio and video frame to be transmitted to a client, receives each response message returned by a data receiving end, and then transmits the response message to the client; and when it is determined that the data receiving end receives at least one complete frame in the audio and video frames based on the response messages, detecting playable frames supporting complete playing of the data receiving end from the at least one complete frame based on the playing sequence of the audio and video frames, and obtaining a predicted cache state of the data receiving end based on a detection result, and carrying out subsequent data transmission by adopting a data transmission strategy preset for the predicted cache state.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] With the continuous development of audio and video technology, various forms of audio and video services, such as live broadcast services, audio and video on demand services, and audio and video calls, have brought diversified user experiences to users. However, audio and video services mainly rely on network transmission services to achieve, so the transmission process of audio and video data is crucial.

[0003] In related technologies, when the data sender transmits audio and video data to the client, it adjusts the sending strategy of the audio and video data according to the real-time network status (such as packet loss rate, latency, etc.) through protocol improvement, congestion control, etc., to ensure that the client receives complete audio and video data and ensure the quality of audio and video playback.

[0004] However, the audio and video playback effect of the client does not entirely depend on the network transmission performance, but may also be related to the client's player cache status. If the data sender only adjusts the data sending strategy based on the network transmission performance, it cannot adjust the data sending strategy in a timely and effective manner, which seriously affects the data transmission efficiency and audio playback effect. In addition, since the player cache is highly dynamic, the data sender cannot perceive the client's player cache status in a timely manner, which makes it difficult for the data sender to adjust the data sending strategy according to the player cache status, affecting the flow control effect.

[0005] For example, if audio or video data is lost during transmission, and there is less cached data in the client's player cache, even if the message is retransmitted according to the network transmission performance, when the lost message is retransmitted to the client, playback will still be stuck because the client has no cached data. Summary of the invention

[0006] The embodiments of the present application provide an audio and video data transmission method and related devices to improve data transmission efficiency and audio and video playback quality.

[0007] In a first aspect, an embodiment of the present application provides an audio and video data transmission method, including:

[0008] Acquire each audio and video frame to be transmitted, and send the flow message set of each audio and video frame to the data receiving end respectively, and receive each response message returned by the data receiving end respectively;

[0009] When it is determined based on the response messages that the data receiving end has received at least one complete frame of the audio and video frames, based on the playback order of the audio and video frames, detecting a playable frame that supports complete playback by the data receiving end from the at least one complete frame;

[0010] Based on the detection result, the predicted cache state of the data receiving end is obtained, and a data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the data receiving end.

[0011] In a second aspect, an embodiment of the present application provides an audio and video data transmission device, including:

[0012] A traffic transceiver unit, used to obtain each audio and video frame to be transmitted, and send the traffic message set of each audio and video frame to the data receiving end respectively, and receive each response message returned by the data receiving end respectively;

[0013] a frame integrity sensing unit, configured to detect, when it is determined based on the response messages that the data receiving end has received at least one complete frame of the audio and video frames, a playable frame that supports complete playback by the data receiving end from the at least one complete frame based on the playback order of the audio and video frames;

[0014] The player cache prediction unit is used to obtain the predicted cache state of the data receiving end based on the detection result, and adopt a data transmission strategy preset for the predicted cache state to send new audio and video frames to be transmitted to the data receiving end.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the above method.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of the above method.

[0018] In an embodiment of the present application, the data sending end sends a set of traffic messages for each audio and video frame to be transmitted to the data receiving end, and receives each response message returned by the data receiving end. Then, when it is determined based on each response message that the data receiving end has received at least one complete frame of each audio and video frame, based on the playback order of each audio and video frame, a playable frame that supports complete playback by the data receiving end is detected from at least one complete frame. Then, based on the detection result, the predicted cache state of the data receiving end is obtained, and the data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the data receiving end.

[0019] In this way, the data sender determines the audio and video frames that the client has completely received in each audio and video frame through the response message, and predicts the audio and video frames that support complete playback by the client, thereby giving the data sender the ability to predict the client player cache information. Therefore, based on the predicted client player cache information capability, the data sending strategy can be adjusted in a timely and effective manner, thereby improving data transmission efficiency and improving audio and video playback effects. In addition, based on the use of a data transmission strategy preset for the predicted cache status (such as sufficient data status or insufficient data status) for subsequent data transmission, it can better adapt to the client's business experience needs in the dubbing and video live broadcast scenario, thereby improving the service provider's audio and video service quality, and then improving the object's usage experience.

[0020] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A schematic diagram of the first application scenario provided in the embodiments of the present application;

[0023] Figure 2 A schematic diagram of a second application scenario provided in an embodiment of the present application;

[0024] Figure 3 A flowchart of an audio and video data transmission method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a traffic message provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a response message provided in an embodiment of the present application;

[0027] Fig. 6A A logical schematic diagram of detecting a complete frame provided in an embodiment of the present application;

[0028] Figure 6B Another logic diagram for detecting a complete frame provided in an embodiment of the present application;

[0029] Figure 7 A logical schematic diagram of a state information table update process provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a playback sequence provided in an embodiment of the present application;

[0031] Fig. 9 A logical diagram of a cache duration determination process provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of two data transmission strategies for predicting cache states provided in an embodiment of the present application;

[0033] Fig.11 A schematic diagram of another application scenario provided in an embodiment of the present application;

[0034] Fig.12 A schematic diagram of the structure of an audio and video data transmission device provided in an embodiment of the present application;

[0035] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0037] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0038] The relevant data collection and processing in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in the examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and personal information subjects. Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0039] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0040] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, allowing various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" are infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0041] As a cloud computing basic capability provider, a cloud computing resource pool (referred to as a cloud platform, generally referred to as Infrastructure as a Service (IaaS)) platform will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0042] According to the logical function division, the Platform as a Service (PaaS) layer can be deployed on the IaaS layer, and the Software as a Service (SaaS) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0043] With the continuous development of audio and video technology, various forms of audio and video services, such as live broadcast services, audio and video on demand services, and audio and video calls, have brought diversified user experiences to users. However, audio and video services mainly rely on network transmission services to achieve, so the transmission process of audio and video data is crucial.

[0044] In related technologies, when the data sender transmits audio and video data to the client, it adjusts the sending strategy of the audio and video data according to the real-time network status (such as packet loss rate, latency, etc.) through protocol improvement, congestion control, etc., to ensure that the client receives complete audio and video data and ensure the quality of audio and video playback.

[0045] However, the audio and video playback effect of the client does not entirely depend on the network transmission performance, but may also be related to the client's player cache status. If the data sender only adjusts the data sending strategy based on the network transmission performance, it cannot adjust the data sending strategy in a timely and effective manner, which seriously affects the data transmission efficiency and audio playback effect. In addition, since the player cache is highly dynamic, the data sender cannot perceive the client's player cache status in a timely manner, which makes it difficult for the data sender to adjust the data sending strategy according to the player cache status, affecting the flow control effect.

[0046] For example, if audio or video data is lost during transmission, and there is less cached data in the client's player cache, even if the message is retransmitted according to the network transmission performance, when the lost message is retransmitted to the client, playback will still be stuck because the client has no cached data.

[0047] In an embodiment of the present application, the data sending end sends a set of traffic messages for each audio and video frame to be transmitted to the data receiving end, and receives each response message returned by the data receiving end. Then, when it is determined based on each response message that the data receiving end has received at least one complete frame of each audio and video frame, based on the playback order of each audio and video frame, a playable frame that supports complete playback by the data receiving end is detected from at least one complete frame. Then, based on the detection result, the predicted cache state of the data receiving end is obtained, and the data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the data receiving end.

[0048] In this way, the data sender determines the audio and video frames that the client has completely received in each audio and video frame through the response message, and predicts the audio and video frames that support complete playback by the client, thereby giving the data sender the ability to predict the client player cache information. Therefore, based on the predicted client player cache information capability, the data sending strategy can be adjusted in a timely and effective manner, thereby improving data transmission efficiency and improving audio and video playback effects. In addition, based on the use of a data transmission strategy preset for the predicted cache status (such as sufficient data status or insufficient data status) for subsequent data transmission, it can better adapt to the client's business experience needs in the dubbing and video live broadcast scenario, thereby improving the service provider's audio and video service quality, and then improving the object's usage experience.

[0049] See also Figure 1 As shown, it is a schematic diagram of a possible application scenario provided in an embodiment of the present application, and the scenario includes: a data sending end 110 and a data receiving end 120.

[0050] The data sending end 110 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0051] The data receiving end 120 may be a terminal with audio and video playback requirements, and the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0052] The data transmitting end 110 and the data receiving end 120 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0053] In the implementation of the present application, the data sending end 110 obtains each audio and video frame to be transmitted, and sends the respective traffic message sets of each audio and video frame to the client, and receives each response message returned by the client. When it is determined that the data receiving end 120 has received at least one complete frame of each audio and video frame based on the received message identifiers, based on the playback order of each audio and video frame, from at least one complete frame, a playable frame that supports complete playback by the client is detected; based on the detection result, the predicted cache state of the client is obtained, and the data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the data receiving end 120, so that the data receiving end 120 plays the transmitted audio and video frames.

[0054] In the embodiments of the present application, the audio and video data transmission method can be applied to various audio and video data transmission scenarios, such as live broadcast scenarios, instant messaging scenarios, online conference scenarios, etc., but is not limited thereto.

[0055] See also Figure 2 As shown, taking the live broadcast scene as an example, the live broadcast scene includes an anchor client, a cloud data transmitter and an audience client. The anchor client obtains each audio and video data through a device with an audio and video recording function, and after encoding each audio and video data, uploads the encoded audio and video data to the cloud server. After receiving the encoded audio and video data, the cloud server obtains each audio and video data by decoding, and can generate audio and video frames of one or more bit rates (i.e., clarity) in the form of one or more encodings. When the cloud server receives a viewing request from a certain audience client, it transmits the flow message of each audio and video frame of the specified bit rate to the audience client, and then predicts the audio and video frames completely received by the audience client according to each response message returned by the audience client, and predicts the audio and video frames that support the complete playback of the audience client, and then, according to the predicted audio and video frames completely received by the audience client, and the audio and video frames that support the complete playback of the audience client, the predicted cache state of the client is obtained, and finally, the data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the audience client. First, take the live broadcast scene as an example to explain the audio and video data transmission process.

[0056] See also Figure 3 As shown, it is a flow chart of an audio and video data transmission method provided in an embodiment of the present application. The process is applied to the data sending end, and the data sending end is a server and the data receiving end is a viewing client (also directly referred to as a client) as an example for explanation. The specific process is as follows:

[0057] S301. The server obtains each audio and video frame to be transmitted, and sends a flow message set of each audio and video frame to the client, and receives each response message returned by the client.

[0058] In the embodiment of the present application, the audio and video frame may refer to an audio frame, or a video frame, or a combination of the two, without limitation.

[0059] In the live broadcast scenario, the audio and video frames to be transmitted obtained by the server can be transmitted to the server by the anchor client. Figure 2 As shown, when the server receives a viewing request from a viewer client, the server sends a set of flow messages for each audio and video frame to the client, and receives each response message returned by the client. It should be noted that in the embodiment of the present application, the data transmission process between the host client and the server is not limited and will not be described in detail here.

[0060] After the server obtains each audio and video frame to be transmitted, it can also identify one or more of the play time and frame type (frame_type) of each audio and video frame.

[0061] The playback time of each audio and video frame can be expressed as a presentation timestamp (pts). Pts is a relative timestamp, usually expressed in milliseconds (ms). Pts is used to represent the playback time of the audio and video frame in the client relative to the playback time of the first audio and video frame in the client. For example, the pts of the first audio and video frame is 0, so when the pts of an audio and video frame is 1, it represents that the audio and video frame is played 1ms after the first audio and video frame is played.

[0062] The pts of each audio and video frame can be obtained based on the adopted audio and video protocol, which includes but is not limited to the streaming media format (Flash Video, FLV) protocol. Taking the FLV protocol as an example, in the FLV file of each audio and video frame, the pts of the audio and video frame is recorded in the "Timestamps" field in the tag header.

[0063] The frame_type of each audio and video frame is used to characterize the frame type corresponding to the audio and video frame. In the embodiment of the present application, the frame type includes but is not limited to audio frames and video frames. For video frames, the video frame type can be further subdivided into key frames and non-key frames, where key frames refer to video I frames, and non-key frames include video P frames and video B frames. Video I frames do not depend on information of other frames, while video P frames depend on previous I frames or P frames, and video B frames depend on adjacent I frames or P frames.

[0064] The frame_type of each audio and video frame can be obtained based on the adopted audio and video protocol. Taking the FLV protocol as an example, in the FLV file of each audio and video frame, the frame type of the audio and video frame is recorded in the Type field in the Tag header. For example, when the value of the Type field is 0x08, it means that the audio and video frame is an audio frame, and when the value of the Type field is 0x09, it means that the audio and video frame is a video frame. For video frames, when the value of the Frametype field in the tag data (Tag Data) is 2, it means that the video frame is a non-key frame, and when the value of the Frame type field value is 1, it means that the video frame is a key frame.

[0065] In an embodiment of the present application, referring to Table 1, the server may also maintain a status information table for each audio and video frame that has not been fully confirmed. In addition to pts and frame_type, the status information table may also include one or more of: message number (Pkt_num), complete reception (Rcv_bool), and complete playback (Play_bool).

[0066] Table 1 Status information table

[0067]

[0068] In Table 1, the Pkt_num field of each audio and video frame contains the Pkt_num of each flow message in the flow message set of the audio and video frame. In the embodiment of the present application, the server can transmit an audio and video frame through one or more flow messages. The Pkt_num field in Table 1 is used to record which flow messages an audio and video frame is sent through. In the Pkt_num field, the initial value of each Pkt_num can be empty, and the content of the Pkt_num field can be updated after some or all of the flow messages in the flow message set of the audio and video frame are sent.

[0069] The Rcv_bool field of each audio and video frame is used to indicate whether the audio and video frame can be completely received by the client. The initial value of the Rcv_bool field can be a set value (such as 0) indicating that the audio and video frame cannot be completely received. For the update of the content of the Rcv_bool field, see S302.

[0070] The Play_bool field of each audio and video frame is used to indicate whether the audio and video frame can be played completely by the client. The initial value of the Play_bool field can be a set value (such as 0) indicating that the audio and video frame cannot be played completely. For the update of the content of the Rcv_bool field, see S302.

[0071] After the server obtains the audio and video frames to be transmitted, it can also detect whether there is any predicted cache evaluation information of the player (i.e., historical cache evaluation information) before sending the respective traffic message sets of each audio and video frame to the client. If there is no historical cache evaluation information, or the existing historical cache evaluation information does not meet the cache sufficiency condition and the cache insufficiency condition, then the respective traffic messages of each audio and video are sent to the network according to the transmission parameters calculated by the conventional congestion control method.

[0072] In the conventional congestion control method, on the one hand, the server can use optimized protocols for data transmission, and the optimized protocols include but are not limited to Quick User Datagram Protocol (UDP) Internet Connection (QUIC), multi-path transmission protocol, etc. The multi-path transmission protocol includes but is not limited to Multi Path TCP (MPTCP) based on Transmission Control Protocol (TCP) and Multipath QUIC (MPQUIC) based on QUIC. Among them, the QUIC protocol can largely solve the problems faced by the existing widely deployed TCP protocol. While improving the traffic transmission performance, the QUIC protocol adopts the multiplexing of Hypertext Transfer Protocol 2.0 (HTTP2), the security of HTTPS and the reliability of TCP protocol. Multi-path transmission (such as MPTCP and MPQUIC) solves the problem of low transmission performance caused by limited transmission resources faced by the existing TCP protocol and QUIC protocol on a single path. By adding a new transmission path, the problem that audio and video traffic cannot reach the terminal in time during traffic transmission and packet loss repair is solved, which greatly improves the efficiency of traffic transmission.

[0073] On the other hand, the server can sense the degree of congestion in the intermediate network and the capacity of transmission resources, and adjust data transmission strategies such as traffic sending and packet loss retransmission based on the sensed information. For example, the server can use the TCP congestion control algorithm (Reno algorithm) to determine whether the intermediate network is congested based on packet loss, use the Timely algorithm to determine whether the intermediate network is congested based on traffic transmission delay, and use the bottleneck bandwidth and round-trip time (Rround-Trip Time, RTT)-based congestion control algorithm (bottleneck bandwidth and RTT, BBR) algorithm to calculate the available transmission resources (i.e., available network bandwidth) of the intermediate network through network modeling. These algorithms determine the degree of congestion in the intermediate network based on different signals, and then adjust the sending window or sending rate of the sender to "make the best possible" use of the intermediate network resources, thereby improving network transmission efficiency, in the hope of improving the terminal's service experience.

[0074] If the existing historical cache evaluation information meets the control condition of sufficient cache condition or insufficient cache condition, the target transmission parameter is calculated according to the congestion control method in S303, and the target transmission parameter is used to send the flow message of each audio and video to the client. See S303 for details.

[0075] As a possible implementation method, after the server sends the traffic message sets of each audio and video frame to the client respectively, for each audio and video frame in each audio and video frame, the pkt_num of each traffic message in the traffic message set of the audio and video frame is filled in the pkt_num column in Table 1.

[0076] For example, see Figure 4 As shown, it is assumed that the video frames to be transmitted include: frame 3, frame 4, etc. In Table 1, the frame type of frame 3 is a video frame, specifically a video P frame, Pts=30, and pkt_num includes: 50, 51, ..., 60, that is, frame 3 is a video frame, specifically a video P frame, which is played 30ms after the client plays the first video frame, and the video frame is transmitted through traffic messages with message numbers 50, 51, ..., 60. In Table 1, frame 4 is a video B frame, Pts=31, and pkt_num includes: 61, 63, 64, ..., 72, that is, frame 4 is a video frame, specifically a video B frame, which is played 31ms after the client plays the first video frame, and the video frame is transmitted through traffic messages with message numbers 61, 63, 64, ..., 720.

[0077] It should be noted that, in the embodiment of the present application, in the pkt_num column of Table 1, the first bit of each pkt_num is used to characterize whether the traffic message is a retransmission message. For example, if it is a retransmission message, the value of the first bit is a binary number 1, otherwise, the value of the first bit is a binary number 0. When the value of the first bit of pkt_num is a binary number 1, the remaining m bits of pkt_num represent the message number of the retransmission message and the message number of the retransmitted traffic message. Among them, the message number of the retransmitted traffic message can be represented by the relative number of the traffic message in each traffic message corresponding to the audio and video frame, or it can be directly represented by the message number of the traffic message, and there is no limitation on this.

[0078] For example, assume that in Table 1, the pkt_num column of frame 2 contains the following pkt_num: {0, 40}, {0, 41}, ..., {0, 49}, {1, 1, 62}, where {0, 40} represents a traffic message with message number 40, {0, 41} represents a traffic message with message number 41, {0, 49} represents a traffic message with message number 49, and {1, 1, 62} represents a traffic message with message number 62, and the traffic message with message number 62 is a retransmission message of the first traffic message (i.e., the traffic message with message number 40), and the traffic message with message number 62 is used to repair the loss of the first traffic message of frame 2.

[0079] In an embodiment of the present application, after receiving a certain flow message sent by the server, the client sends a response message to the server. The response message may carry the message number of one or more flow messages. The response message may be used to declare that the client has received the corresponding flow message, or may be used to declare that the corresponding flow message has not been received, and there is no limitation on this. The following only takes the example that the response message can be used to declare that the client has received the corresponding flow message as an example for explanation.

[0080] For example, see Figure 5 As shown, it is assumed that the response message is used to declare that the client has received the corresponding traffic message. Each response message carries the message number of a traffic message. After receiving the traffic messages with message numbers 50, 51, 52, ..., 74, etc. sent by the server, the client returns response messages carrying message numbers 50, 52, ..., 74, etc. to the server to declare to the server that the client has received the traffic messages with message numbers 50, 51, 52, ..., 74, etc.

[0081] S302: When the server determines, based on each response message, that the client has received at least one complete frame of each audio and video frame, based on the playback order of each audio and video frame, the server detects a playable frame that supports complete playback by the client from at least one complete frame.

[0082] After the server receives each response message, as a possible implementation method, for each response message in each response message, based on the message identifier (i.e., message number) carried by each response message, it detects whether the client has received each audio and video frame, and regards the audio and video frames received by the client as complete frames.

[0083] When the server detects whether the client has received each audio and video frame based on each received response message, it can use but is not limited to the following methods:

[0084] The server obtains a message identification set of each audio and video frame recorded locally, each message identification set including a message identification of each flow message in a flow message set corresponding to the audio and video frame;

[0085] The server determines, based on the message identifier carried in each response message, that when the data receiving end receives at least one audio or video frame, it regards the at least one audio or video frame as at least one complete frame.

[0086] Taking audio and video frame x as an example, audio and video frame x can be each audio and video frame in each audio and video frame. In the embodiment of the present application, the message identifier set of audio and video frame x is recorded in the Pkt_num column in Table 1 above. After the server obtains each response message through the Pkt_num column in Table 1, it is assumed that the response message is used to declare that the client has received the corresponding traffic message. The server determines that the data receiving end receives each traffic message of audio and video frame x based on the message identifier carried by each response message, and treats the audio and video frame x as a complete frame.

[0087] For example, see Fig. 6A As shown, frame 3 is sent via traffic messages with message numbers 50, 51, ..., 60. Among the response messages (i.e., ACK messages) received by the server, there are ACK messages stating that the client has received traffic messages with message numbers 50, 52, ..., 60, but there is no ACK message stating that the client has received traffic messages with message number 51. At this time, it is determined that the client has not received frame 3 and frame 3 is not a complete frame.

[0088] For example, see Figure 6B As shown, frame 4 is sent via traffic messages with message numbers 63, 64, ..., 72. Assuming that, among the response messages received by the server, there is a response message stating that the client has received the traffic messages with message numbers 63, 64, ..., 72, it is determined that the client has received frame 4 and regards frame 4 as a complete frame.

[0089] It should be noted that in an embodiment of the present application, if the server determines that all traffic messages of a certain audio and video frame have been received by the client, the Rcv_bool value of the frame in the status information table can be set to 1, that is, if it is determined that the audio and video frame x is a complete frame, the value of the Rcv_bool field in Table 1 is set to 1.

[0090] For example, see Figure 7 As shown, it is assumed that if the server determines that frame 3 is a complete frame, the value of the Rcv_bool field (i.e., the complete reception field) in Table 1 is set to 1. Further, if the server determines that frame 3 is a playable frame that supports complete playback by the client, the value of the Play_bool field (i.e., complete playback) in Table 1 is set to 1.

[0091] When the server detects that a certain audio or video frame has been completely received by the client, it further detects whether the audio or video frame can be completely played by the client. Specifically, the server performs the following operations for each complete frame in at least one complete frame:

[0092] When it is determined based on the playback order of each audio and video frame that there is at least one other frame before a complete frame in each audio and video frame, based on the detection result corresponding to each of the at least one other frame, it is detected whether a complete frame is a playable frame that supports complete playback by the client.

[0093] Assuming that the audio and video frame x has been completely received by the client, still taking the audio and video frame x as an example, the process of how to detect whether a certain audio and video frame can be completely played by the client is explained.

[0094] As a possible situation, when it is determined based on the playback order of each audio and video frame that there are no other frames before the audio and video frame x in each audio and video frame, that is, when the audio and video frame x is the first frame, the audio and video frame x can be directly regarded as a playable frame that supports complete playback by the client.

[0095] As another possible situation, when it is determined based on the playback order of each audio and video frame that there is at least one other frame before the audio and video frame x in each audio and video frame, based on the detection result corresponding to at least one other frame, it is detected whether the audio and video frame x is a playable frame that supports complete playback by the client.

[0096] For example, see Figure 8As shown, it is assumed that the playback order of each video frame is: frame 1, frame 2, frame 3, frame 4, frame 5, etc., among which frame 1, frame 2, frame 4, and frame 5 are complete frames, and frame 3 is not a complete frame. For frame 1, there are no other frames before frame 1 in each audio and video frame, and frame 1 is directly regarded as a playable frame that supports complete playback by the client. For frame 2, there is frame 1 before frame 2 in each audio and video frame. Therefore, based on the detection result of frame 1, it is detected whether frame 2 is a playable frame that supports complete playback by the client. Similarly, for frame 4, based on the respective detection results of frame 1, frame 2, and frame 3, it is detected whether frame 4 is a playable frame that supports complete playback by the client. For frame 5, based on the respective detection results of frame 1, frame 2, frame 3, and frame 4, it is detected whether frame 1, frame 2, frame 3, and frame 4 are playable frames that support complete playback by the client.

[0097] As a possible implementation method, when detecting whether the audio and video frame x is a playable frame that supports complete playback by the client based on the detection results corresponding to at least one other frame, if it is determined that at least one other frame is a complete frame based on the detection results corresponding to at least one other frame, then the audio and video frame x is determined to be a playable frame that supports complete playback by the client; if it is determined that there is an audio and video frame in at least one other frame that has not been received by the client based on the detection results corresponding to at least one other frame, then it is determined that the audio and video frame x is not a playable frame that supports complete playback by the client.

[0098] In an embodiment of the present application, the detection result corresponding to at least one other frame can be determined by the Play_bool field of at least one other frame. For example, when the value of Play_bool is 1, it indicates that the corresponding frame is a playable frame that supports complete playback by the client. When the value of Play_bool is 0, it indicates that the corresponding frame is not a playable frame that supports complete playback by the client.

[0099] Still reading Figure 8 As shown, only frames 2 and 5 are used as examples for explanation. The server determines that frame 2 is a playable frame that supports complete playback by the client based on the detection result of frame 1 (the value of Play_bool of frame 1 is 1). The server determines that frame 2 is a playable frame that supports complete playback by the client based on the detection results of frame 1, frame 2, frame 3, and frame 4. In the detection results, the values ​​of Play_bool of frames 1 and 2 are both 1, and the values ​​of Play_bool of frames 3 and 4 are both 0, that is, frames 1 and 2 are both playable frames that support complete playback by the client, but frames 3 and 4 are not playable frames. Therefore, frame 5 is not a playable frame that supports complete playback by the client.

[0100] The following describes whether audio frames and video frames support complete playback on the client.

[0101] If the audio / video frame x is an audio frame, then the server determines whether there are other frames before the audio / video frame x in each audio / video frame based on the playback order of the audio / video frames. If not (i.e., the audio / video frame x is the first frame), the audio / video frame x is a playable frame that supports complete playback by the client. If so (i.e., the audio / video frame x is not the first frame), the server further detects whether the audio / video frame x is a playable frame that supports complete playback by the client based on the detection result corresponding to at least one other frame. In the further detection process, if the Play_bool corresponding to at least one other frame is all 1, then it is determined that the audio / video frame x is a playable frame that supports complete playback by the client. If the Play_bool corresponding to at least one other frame is not all 1, then it is determined that the audio / video frame x is not a playable frame that supports complete playback by the client.

[0102] If the audio / video frame x is a video frame I frame, when the audio / video frame x meets any of the following conditions, it is determined that the audio / video frame x is a playable frame that supports complete playback by the client:

[0103] Condition 1: There is no I frame, P frame or B frame before the video I frame. In other words, there is no other frame before the video I frame in each audio and video frame.

[0104] Condition 2: Play_bool of all video frames before the video I frame is 1. That is, Play_bool corresponding to at least one other frame before the video I frame is 1.

[0105] If the audio / video frame x is a video P frame, it is determined that the audio / video frame x is a playable frame that supports complete playback by the client when the audio / video frame x meets any of the following conditions: Play_bool of all I frames, P frames, and B frames preceding the video P frame are all 1.

[0106] If the audio / video frame x is a video B frame, it is determined that the audio / video frame x is a playable frame that supports complete playback by the client when the audio / video frame x meets any of the following conditions: Play_bool of all I frames, P frames, and B frames preceding the video P frame are all 1.

[0107] Further, when it is determined that the audio and video frame x is a playable frame that supports complete playback by the client, the Play_bool of the audio and video frame x is set to 1, otherwise the Play_bool of the audio and video frame x is maintained to 0.

[0108] In a possible implementation, the playback order of the audio and video frames can be determined by:

[0109] Obtain a message identification set for each audio and video frame recorded locally, each message identification set containing a message identification of each flow message in a flow message set corresponding to the audio and video frame, each message identification being obtained by numbering the corresponding flow message in the order in which it is sent;

[0110] The playback order of each audio and video frame is determined based on the values ​​of each message identifier contained in each acquired message identifier set.

[0111] Exemplarily, in the playback order, if the minimum message identifier of one audio and video frame is less than the minimum message identifier of another audio and video frame, then one audio and video frame is located before another audio and video frame, that is, the basis for judging the "front and back" order of audio and video frames can be the minimum pkt_num corresponding to the audio and video frames: if the minimum pkt_num of one frame (frame a) is less than the minimum pkt_num of another frame (frame b), it means that frame a is in front of frame b. For example, the pkt_num of frame a includes: 50, 51, 52, ..., 60, and the pkt_num of frame b includes: 61, 62, 63, ..., 66, and the minimum pkt_num50 of frame a is less than the minimum pkt_num61 of another frame (frame b), therefore, it means that frame a is in front of frame b.

[0112] S303: The server obtains the predicted cache state of the client based on the detection result, and uses a data transmission strategy preset for the predicted cache state to send new audio and video frames to be transmitted to the client.

[0113] In the embodiment of the present application, the predicted cache state is used to characterize the state of the audio and video frames cached in the client player predicted by the server. The predicted cache state can be an insufficient cache state or an adequate cache state. The server obtains the predicted cache state of the client based on the detection result, and can use the data transmission strategy preset for the insufficient cache state or the adequate cache state to send new audio and video frames to be transmitted to the client, thereby dynamically adjusting subsequent data transmission, improving data transmission efficiency, and improving audio and video playback effects. Among them, the new audio and video frames to be transmitted refer to the audio and video frames that need to be transmitted subsequently.

[0114] In a possible implementation, when obtaining the predicted cache status of the client based on the detection result, the following operations may be used but are not limited to:

[0115] Based on the detection results and the audio and video frames currently played by the client, the server obtains the cache quantity and cache duration of the audio and video frames cached in the client; based on the cache quantity and cache duration, the server obtains the client's predicted cache status.

[0116] The cache duration of the audio and video frames cached in the client can be predicted by, but not limited to, the following methods:

[0117] When it is determined based on the detection results that there is at least one playable frame in at least one complete frame, the server determines a target frame whose playback time meets the set reference conditions from at least one playable frame based on the playback time of the at least one playable frame; then, the server obtains the cache duration of the audio and video frames cached in the client based on the audio and video frames currently played by the client and the target frame.

[0118] The target frame that meets the set reference condition may be a playable frame with the maximum pts among at least one playable frame, that is, play_bool=1, and an audio or video frame with the maximum pts value, but is not limited thereto.

[0119] The cache duration of the audio and video frames cached in the client can be determined according to the difference between the playback time of the audio and video frames currently played by the client and the playback time of the target frame, but is not limited thereto.

[0120] Exemplarily, the server determines the audio and video frame currently played by the client (recorded as frame_cur), and determines the pts of the audio and video frame currently played by the client (recorded as pts_cur). Then, based on the pts of each of the at least one playable frame, the playable frame with the maximum pts in the at least one playable frame is used as the target frame. Based on the pts of the target frame (recorded as pts_max) and in combination with pts_cur, the cache duration of the audio and video frame cached in the client (recorded as Frame_time_len) can be calculated using formula (1):

[0121] Frame_time_len = pts_max - pts_cur Formula (1)

[0122] It should be noted that in the embodiment of the present application, the audio and video frames currently played by the client can be fed back to the server by the client in real time, but is not limited to this.

[0123] For example, see Fig. 9 As shown, it is assumed that the playable frames include frame 1 and frame 2, and the pts of frame 1 and frame 2 are 50 and 51 respectively. The audio and video frame currently played by the client is frame 1. The server determines the audio and video frame currently played by the client, and determines that the pts value of the audio and video frame currently played by the client is 50 (that is, pts_cur=50). Then, based on the pts of frame 1 and frame 2, frame 2 with the largest pts among frame 1 and frame 2 is used as the target frame. Based on the pts of the target frame (that is, pts_max=51) and combined with pts_cur, the cache duration of the audio and video frames cached in the client is Frame_time_len=51-50=1ms.

[0124] In the embodiment of the present application, the number of cached audio and video frames cached in the client can be predicted by but not limited to the following methods:

[0125] When it is determined based on the detection result that there is at least one playable frame in at least one complete frame, the server obtains the cached number of audio and video frames cached in the client (represented as frame_buffer_amount) based on the number of at least one playable frame and the audio and video frames currently played by the client.

[0126] Specifically, as a possible implementation, the server may use the number of playable frames whose corresponding pts is not less than pts_cur in at least one playable frame as the number of cached audio and video frames cached in the client. Fig. 9 As shown, pts_cur=50, the playable frames include frame 1 and frame 2, the pts of frame 1 and frame 2 are 50 and 51 respectively, the corresponding playable frames whose pts are not less than pts_cur include frame 1 and frame 2, and the number of cached audio and video frames in the client is 2.

[0127] As another possible implementation method, in order to reduce memory usage and improve the efficiency of audio and video frame detection, if the server detects that a certain video frame has been played in the client player, that is, the pts of the audio and video frame is less than pts_cur, then the relevant information of the audio and video frame is deleted from the status information table. Accordingly, when predicting the cached number of audio and video frames cached in the client, the number of playable frames contained in the status information table can be used as the cached number of audio and video frames cached in the client.

[0128] After predicting the cache quantity and cache duration of the audio and video frames cached in the client, in a possible implementation, based on the cache quantity and cache duration of the audio and video frames cached in the client, there are the following two possible predicted cache states:

[0129] The first possible state: if it is determined that the preset cache shortage condition is met based on the cache quantity and the cache duration, a predicted cache state indicating that the client is in a cache shortage state is obtained.

[0130] Among them, when at least one of the following conditions is met, the preset cache shortage condition is met:

[0131] Condition a: Cache duration (frame_time_len) < cache duration threshold (frame_time_len_thres);

[0132] Condition b: Buffer amount (frame_buffer_amount) < buffer amount threshold (frame_buffer_amount_thres).

[0133] Among them, frame_time_len_thres and frame_buffer_amount_thres can be configured by an administrator and reflected in the configuration file. For example, by default, frame_time_len_thres=100ms, frame_buffer_amount_thres=5.

[0134] The second possible state: if it is determined that the preset cache sufficiency condition is met based on the cache quantity and the cache duration, a predicted cache state representing the cache sufficiency state of the client is obtained.

[0135] Among them, when at least one of the following conditions is met, the preset cache sufficiency condition is met:

[0136] Condition c: frame_time_len>N*frame_time_len_thres;

[0137] Condition d: frame_buffer_amount>N*frame_buffer_amount_thres.

[0138] The value of N is a positive integer, which can be configured by an administrator and reflected in a configuration file. For example, by default, N=5.

[0139] In a possible implementation, a data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the client, including:

[0140] Determine initial transmission parameters based on the performance of the network between the server and the client;

[0141] Based on the data transmission strategy preset for the predicted cache state, the initial transmission parameters are adjusted to obtain the target transmission parameters, and based on the target transmission parameters, new audio and video frames to be transmitted are sent to the client.

[0142] The transmission parameters include one or more of the transmission rate and the transmission window, but are not limited thereto. The transmission rate in the initial transmission parameters is recorded as pacing_cpt, and the transmission window in the initial transmission parameters is recorded as cwnd_cpt. Pacing_cpt and cwnd_cpt can be calculated using a congestion control algorithm. The specific congestion control algorithm is not limited and will not be described here. The transmission rate in the target transmission parameters is recorded as Pacing_final, and the transmission window in the target transmission parameters is recorded as Cwnd_final.

[0143] In the embodiment of the present application, different data transmission strategies can be used for data transmission in the insufficient cache state and the sufficient cache state. The following describes data transmission in the insufficient cache state and the sufficient cache state, respectively.

[0144] See also Fig.10 As shown, if the predicted cache state indicates that the client is in a cache-insufficient state, the initial transmission parameters are adjusted according to the data transmission strategy of increasing the amount of transmitted data to obtain the target transmission parameters.

[0145] For example, Pacing_final can be calculated by formula (2), and Cwnd_final can be calculated by formula (3):

[0146] Cwnd_final = cwnd_cpt * cwnd_gain_weight_1 formula (2)

[0147] Pacing_final = pacing_cpt * pacing_gain_weight_1 Formula (3)

[0148] Among them, cwnd_gain_weight_1 is the first weight corresponding to cwnd_cpt, and pacing_gain_weight_1 is the first weight corresponding to pacing_cpt. cwnd_gain_weight_1 and pacing_gain_weight_1 can be configured by an administrator and can be reflected in the configuration manual. For example, by default, pacing_gain_weight_1=cwnd_gain_weight_1=1.5.

[0149] In the case where the predicted cache state indicates that the client is in a state of insufficient cache, the server can also use an aggressive method to perform packet loss repair operations. Specifically, the server can use redundant retransmission for packet loss, that is, retransmit M copies of the same packet loss data, where these M copies of packet loss data can be sent at the same time or at intervals of SRTT / 2. SRTT represents smooth RTT, which can be measured by the server. Among them, packet loss detection can be implemented using algorithms such as but not limited to forward acknowledgments (FACK) and time-based packet loss detection (RACK), but is not limited to this.

[0150] If the predicted cache state indicates that the client is in a state of sufficient cache, the initial transmission parameters are adjusted according to a data transmission strategy for reducing the amount of transmitted data to obtain target transmission parameters.

[0151] For example, Pacing_final can be calculated by formula (4), and Cwnd_final can be calculated by formula (5):

[0152] Cwnd_final = cwnd_cpt * cwnd_gain_weight_2 Formula (4)

[0153] Pacing_final = pacing_cpt * pacing_gain_weight_2 Formula (5)

[0154] Among them, cwnd_gain_weight_2 is the second weight corresponding to cwnd_cpt, and pacing_gain_weight_2 is the second weight corresponding to pacing_cpt. cwnd_gain_weight_2 and pacing_gain_weight_2 can be configured by an administrator and can be reflected in the configuration manual. For example, by default, pacing_gain_weight_2=cwnd_gain_weight_2=0.75.

[0155] It should be noted that in the implementation of this application, in order to achieve the situation where the client caches fewer audio and video frames, the data sender can adopt a "suitably aggressive" data transmission method (such as increasing the traffic sending window or sending window) to achieve caching more audio and video frames in the client player cache within the same time; when the client caches fewer audio and video frames, the data sender will adopt a "more aggressive" data transmission method (such as a packet loss retransmission strategy) for packet loss to ensure that the lost traffic packets are repaired in the shortest possible time. Therefore, the Cwnd_final obtained based on cwnd_gain_weight_2 should be less than the Cwnd_final obtained based on cwnd_gain_weight_1, and the Pacing_final obtained based on pacing_gain_weight_2 should be less than the Pacing_final obtained based on pacing_gain_weight_1.

[0156] When the predicted cache state indicates that the client is in a sufficient cache state, the server may also perform bandwidth detection or minimum delay detection tasks on the server. Exemplarily, in the BBR algorithm, the server will perform one or more operations of bandwidth detection (ProbeBW) or delay detection (ProbeRTT) to obtain at least one of the latest bandwidth and delay information.

[0157] The present application is described below in conjunction with a specific embodiment.

[0158] Take the online conference scenario as an example, see Fig.11 As shown, the application scenario includes client A of object A, client B of object B, client B of object C, and a server for providing voice call services. Object A is a conference speaker. Exemplarily, client A, client B, and client C are all mobile phone clients.

[0159] Client A obtains each audio frame through a device with an audio recording function, encodes each audio frame, and uploads the encoded audio frame to the server. After receiving each encoded audio frame, the server obtains each audio frame by decoding.

[0160] After receiving the conference joining request from client B of object B, the server transmits each flow message in the flow message set of each audio and video frame to client B, and receives each response message returned by client B. Then, when it is determined based on each response message that client B has received at least one complete frame of each audio and video frame, the server detects a playable frame that supports complete playback of client B from at least one complete frame based on the playback order of each audio and video frame. Furthermore, based on the detection results, the predicted cache status of client B is determined. As the first case, the predicted cache status of client B is an insufficient cache state. Then, according to the data transmission strategy preset for the predicted cache state (aggressive packet sending and redundant retransmission), new audio and video frames to be transmitted are sent to client B. As the second case, the predicted cache status of client B is an adequate cache state. Then, according to the data transmission strategy preset for the predicted cache state (mild packet sending and network detection), new audio and video frames to be transmitted are sent to client B. As the third case, the predicted cache status of client B is neither an adequate cache state nor an insufficient cache state. Then, according to the data transmission strategy preset for the predicted cache state (data transmission strategy based on a conventional congestion control algorithm), new audio and video frames to be transmitted are sent to client B.

[0161] Similarly, after receiving the conference joining request from client C of object C, the server transmits each flow message in the flow message set of each audio and video frame to client C, and receives each response message returned by client C. Then, based on each response message returned by client C, the audio and video frames completely received by client C are predicted, and the audio and video frames that support complete playback by client C are predicted. Then, based on the predicted audio and video frames completely received by client C and the audio and video frames that support complete playback by client C, the predicted cache state of the client is obtained, and then the data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the viewer client.

[0162] Based on the same inventive concept, the present application embodiment provides an audio and video data transmission device. Fig.12 As shown, it is a structural diagram of an audio and video data transmission device 1200, which may include:

[0163] The traffic transceiver unit 1201 is used to obtain each audio and video frame to be transmitted, and send the traffic message set of each audio and video frame to the data receiving end respectively, and receive each response message returned by the data receiving end respectively;

[0164] The frame integrity sensing unit 1202 is configured to detect, based on the playback order of the audio and video frames, a playable frame that supports complete playback by the data receiving end from the at least one complete frame when it is determined that the data receiving end has received at least one complete frame of the audio and video frames based on the response messages;

[0165] The player cache prediction unit 1203 is used to obtain the predicted cache state of the data receiving end based on the detection result, and adopt the data transmission strategy preset for the predicted cache state to send new audio and video frames to be transmitted to the data receiving end.

[0166] As a possible implementation manner, when detecting, based on the playback order of the audio and video frames, from the at least one complete frame, a playable frame that supports complete playback by the data receiving end, the frame integrity sensing unit 1202 is specifically configured to:

[0167] For each of the at least one complete frame, perform the following operations:

[0168] When it is determined based on the playback order of the audio and video frames that there is at least one other frame before a complete frame in the audio and video frames, based on the detection results corresponding to the at least one other frame, it is detected whether the complete frame is a playable frame that supports complete playback by the data receiving end.

[0169] As a possible implementation manner, when detecting whether the one complete frame is a playable frame supporting complete playback by the data receiving end based on the detection result corresponding to each of the at least one other frames, the frame integrity sensing unit 1202 is specifically used to:

[0170] If it is determined based on the detection results corresponding to the at least one other frame that the at least one other frame is a complete frame, determining that the one complete frame is a playable frame that supports complete playback by the data receiving end;

[0171] If, based on the detection results corresponding to the at least one other frame, it is determined that there are audio or video frames in the at least one other frame that are not received by the data receiving end, it is determined that the one complete frame is not a playable frame that supports complete playback by the data receiving end.

[0172] As a possible implementation manner, when obtaining the predicted cache state of the data receiving end based on the detection result, the player cache prediction unit 1203 is specifically used to:

[0173] Based on the detection result, in combination with the audio and video frames currently played by the data receiving end, cache evaluation information of the data receiving end is obtained;

[0174] Based on the cache evaluation information, a predicted cache state of the data receiving end is obtained.

[0175] As a possible implementation, the cache evaluation information includes: the cache quantity and cache duration of the audio and video frames cached in the data receiving end;

[0176] When the cache evaluation information of the data receiving end is obtained based on the detection result and in combination with the audio and video frames currently played by the data receiving end, the player cache prediction unit 1203 is specifically used to:

[0177] When it is determined based on the detection result that there is at least one playable frame in the at least one complete frame, based on the respective play times of the at least one playable frame, determining a target frame whose play time meets a set reference condition from the at least one playable frame;

[0178] Based on the respective playing times of the audio and video frames currently played by the data receiving end and the target frame, the cache duration is obtained, and the number of the at least one playable frame is used as the cache number.

[0179] As a possible implementation manner, when obtaining the predicted cache state of the data receiving end based on the cache evaluation information, the player cache prediction unit 1203 is specifically used to:

[0180] If it is determined based on the cache evaluation information that the preset cache shortage condition is met, a predicted cache state indicating that the data receiving end is in a cache shortage state is obtained;

[0181] If it is determined based on the cache evaluation information that the preset cache sufficiency condition is met, a predicted cache state representing a cache sufficiency state of the data receiving end is obtained.

[0182] As a possible implementation, when the data transmission strategy preset for the predicted cache state is adopted to send a new audio and video frame to be transmitted to the data receiving end, the player cache prediction unit 1203 is specifically used to:

[0183] Determining initial transmission parameters based on network transmission performance between the data sending end and the data receiving end;

[0184] Based on the data transmission strategy preset for the predicted cache state, the initial transmission parameters are adjusted to obtain target transmission parameters, and based on the target transmission parameters, new audio and video frames to be transmitted are sent to the data receiving end.

[0185] As a possible implementation manner, when the initial transmission parameters are adjusted based on the data transmission strategy preset for the predicted cache state to obtain the target transmission parameters, the player cache prediction unit 1203 is specifically used to:

[0186] If the predicted cache state indicates that the data receiving end is in a cache-scarce state, adjusting the initial transmission parameters according to a data transmission strategy of increasing the amount of transmitted data to obtain target transmission parameters;

[0187] If the predicted cache state indicates that the data receiving end is in a cache sufficient state, the initial transmission parameters are adjusted according to a data transmission strategy for reducing the amount of transmitted data to obtain target transmission parameters.

[0188] As a possible implementation manner, the frame integrity sensing unit 1202 is further configured to:

[0189] Obtaining a message identification set of each of the locally recorded audio and video frames, each message identification set containing a message identification of each flow message in a flow message set corresponding to the audio and video frame, each message identification being obtained by numbering in the order in which the corresponding flow message is sent;

[0190] The playback order of the audio and video frames is determined based on the values ​​of the message identifiers contained in each of the acquired message identifier sets.

[0191] As a possible implementation manner, when it is determined that the data receiving end has received at least one complete frame in each audio and video frame based on each response message, the frame integrity perception unit 1202 is specifically used to:

[0192] Obtaining a message identification set of each of the locally recorded audio and video frames, each message identification set including a message identification of each flow message in a flow message set corresponding to the audio and video frame;

[0193] Based on the message identifiers carried by each of the response messages, it is determined that when the data receiving end receives the at least one audio and video frame, the at least one audio and video frame is regarded as at least one complete frame.

[0194] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0195] Regarding the device in the above embodiment, the specific manner in which each unit executes the request has been described in detail in the embodiment of the method, and will not be elaborated here.

[0196] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0197] Based on the same inventive concept, the embodiment of the present application also provides an electronic device. In one embodiment, the electronic device can be a data transmitting end or a terminal device. Fig.13 As shown, it is a schematic diagram of the structure of a possible electronic device provided in an embodiment of the present application, Fig.13 In the embodiment, the electronic device 1300 includes: a processor 1310 and a memory 1320.

[0198] The memory 1320 stores a computer program that can be executed by the processor 1310 , and the processor 1310 can execute the steps of the above-mentioned audio and video data transmission method by executing the instructions stored in the memory 1320 .

[0199] The memory 1320 may be a volatile memory, such as a random-access memory (RAM); the memory 1320 may also be a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1320 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1320 may also be a combination of the above memories.

[0200] The processor 1310 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1310 is configured to implement the above-mentioned audio and video data transmission method when executing the computer program stored in the memory 1320 .

[0201] In some embodiments, the processor 1310 and the memory 1320 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0202] The specific connection medium between the processor 1310 and the memory 1320 is not limited in the embodiment of the present application. In the embodiment of the present application, the processor 1310 and the memory 1320 are connected via a bus as an example. Fig.13 The connections between the other components are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.13 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0203] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the above-mentioned audio and video data transmission method. In some possible implementations, various aspects of the audio and video data transmission method provided by the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to perform the steps in the above-mentioned audio and video data transmission method. For example, the electronic device can perform the following steps: Figure 3 Follow the steps shown in .

[0204] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (Compact Disk Read Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0205] The program product of the embodiment of the present application may adopt a CD-ROM and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a computer program, which may be used by or in combination with a command execution system, apparatus or device.

[0206] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a computer program for use by or in conjunction with a command execution system, apparatus, or device.

[0207] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0208] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for transmitting audio and video data, It is characterized in that Applied to the data sending end, including: Acquire each audio and video frame to be transmitted, and send the flow message set of each audio and video frame to the data receiving end respectively, and receive each response message returned by the data receiving end respectively; When it is determined based on the response messages that the data receiving end has received at least one complete frame of the audio and video frames, based on the playback order of the audio and video frames, detecting a playable frame that supports complete playback by the data receiving end from the at least one complete frame; Based on the detection result, the predicted cache state of the data receiving end is obtained, and a data transmission strategy preset for the predicted cache state is adopted to send new audio and video frames to be transmitted to the data receiving end.

2. The method according to claim 1, It is characterized in that The detecting, based on the playback order of the audio and video frames, from the at least one complete frame, a playable frame that supports complete playback by the data receiving end comprises: For each of the at least one complete frame, perform the following operations: When it is determined based on the playback order of the audio and video frames that there is at least one other frame before a complete frame in the audio and video frames, based on the detection results corresponding to the at least one other frame, it is detected whether the complete frame is a playable frame that supports complete playback by the data receiving end.

3. The method according to claim 2, It is characterized in that The detecting, based on the detection result corresponding to each of the at least one other frame, whether the one complete frame is a playable frame supporting complete playback by the data receiving end comprises: If it is determined based on the detection results corresponding to the at least one other frame that the at least one other frame is a complete frame, determining that the one complete frame is a playable frame that supports complete playback by the data receiving end; If, based on the detection results corresponding to the at least one other frame, it is determined that there are audio or video frames in the at least one other frame that are not received by the data receiving end, it is determined that the one complete frame is not a playable frame that supports complete playback by the data receiving end.

4. The method according to claim 1, 2 or 3, It is characterized in that The obtaining, based on the detection result, a predicted cache state of the data receiving end includes: Based on the detection result, in combination with the audio and video frames currently played by the data receiving end, cache evaluation information of the data receiving end is obtained; Based on the cache evaluation information, a predicted cache state of the data receiving end is obtained.

5. The method according to claim 4, It is characterized in that The cache evaluation information includes: the cache quantity and cache duration of the audio and video frames cached in the data receiving end; Then, based on the detection result and in combination with the audio and video frames currently played by the data receiving end, obtaining the cache evaluation information of the data receiving end includes: When it is determined based on the detection result that there is at least one playable frame in the at least one complete frame, based on the respective play times of the at least one playable frame, determining a target frame whose play time meets a set reference condition from the at least one playable frame; Based on the respective playing times of the audio and video frames currently played by the data receiving end and the target frame, the cache duration is obtained, and the number of the at least one playable frame is used as the cache number.

6. The method according to claim 4, It is characterized in that The obtaining, based on the cache evaluation information, a predicted cache state of the data receiving end includes: If it is determined based on the cache evaluation information that the preset cache shortage condition is met, a predicted cache state indicating that the data receiving end is in a cache shortage state is obtained; If it is determined based on the cache evaluation information that the preset cache sufficiency condition is met, a predicted cache state representing a cache sufficiency state of the data receiving end is obtained.

7. The method according to claim 1, 2 or 3, It is characterized in that The adopting of a data transmission strategy preset for the predicted cache state to send a new audio and video frame to be transmitted to the data receiving end includes: Determining initial transmission parameters based on network transmission performance between the data sending end and the data receiving end; Based on the data transmission strategy preset for the predicted cache state, the initial transmission parameters are adjusted to obtain target transmission parameters, and based on the target transmission parameters, new audio and video frames to be transmitted are sent to the data receiving end.

8. The method according to claim 7, It is characterized in that The adjusting the initial transmission parameters based on the data transmission strategy preset for the predicted cache state to obtain target transmission parameters includes: If the predicted cache state indicates that the data receiving end is in a cache-scarce state, adjusting the initial transmission parameters according to a data transmission strategy of increasing the amount of transmitted data to obtain target transmission parameters; If the predicted cache state indicates that the data receiving end is in a cache sufficient state, the initial transmission parameters are adjusted according to a data transmission strategy for reducing the amount of transmitted data to obtain target transmission parameters.

9. The method according to claim 1, 2 or 3, It is characterized in that The playback order of the audio and video frames is determined by: Obtaining a message identification set of each of the locally recorded audio and video frames, each message identification set containing a message identification of each flow message in a flow message set corresponding to the audio and video frame, each message identification being obtained by numbering in the order in which the corresponding flow message is sent; The playback order of the audio and video frames is determined based on the values ​​of the message identifiers contained in each of the acquired message identifier sets.

10. The method of claim 1, 2 or 3, It is characterized in that The determining, based on the response messages, that the data receiving end has received at least one complete frame of the audio and video frames comprises: Obtaining a message identification set of each of the locally recorded audio and video frames, each message identification set including a message identification of each flow message in a flow message set corresponding to the audio and video frame; Based on the message identifiers carried by each of the response messages, it is determined that when the data receiving end receives the at least one audio and video frame, the at least one audio and video frame is regarded as at least one complete frame.

11. An audio and video data transmission device, It is characterized in that include: A traffic transceiver unit, used to obtain each audio and video frame to be transmitted, and send the traffic message set of each audio and video frame to the data receiving end respectively, and receive each response message returned by the data receiving end respectively; a frame integrity sensing unit, configured to detect, when it is determined based on the response messages that the data receiving end has received at least one complete frame of the audio and video frames, a playable frame that supports complete playback by the data receiving end from the at least one complete frame based on the playback order of the audio and video frames; The player cache prediction unit is used to obtain the predicted cache state of the data receiving end based on the detection result, and adopt a data transmission strategy preset for the predicted cache state to send new audio and video frames to be transmitted to the data receiving end.

12. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 10.

13. A computer-readable storage medium, It is characterized in that The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 10.

14. A computer program product, It is characterized in that It includes a computer program, which is stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the steps of any method described in claims 1 to 10.

Citation Information

Patent Citations

  • Techniques for synchronization of audio and video

    CN103621102A

  • Method and device for realizing code rate switching

    CN110022498A

  • Video code rate adjustment method and device, server and storage medium

    CN110290402A

  • Audio and video playing method, device and system, electronic equipment and storage medium

    CN116055810A

  • Congestion mitigation method and apparatus considering optimization of user satisfaction of video traffic in mobile network

    US20160295298A1

Cited By

  • Video function system of storage application chip, storage application chip and server

    CN119537303A

  • Media data transmission method and apparatus, and electronic device, storage medium and program product

    EP4611347A1