Audio and video transmission control method, system and device and storage medium

Abstract

The invention discloses an audio and video transmission control method, system and device and a storage medium, and the method comprises the steps: transmitting a real-time transmission audio and video data packet according to a preset code rate, and receiving a real-time transmission control data packet; the real-time transmission control data packet is determined according to the real-time transmission audio and video data packet; determining a delay and a packet loss rate according to the real-time transmission control data packet, and updating the preset code rate according to the delay and the packet loss rate; and returning to execute the step of sending the real-time transmission audio and video data packet according to the preset code rate until the audio and video data is sent completely. The embodiment of the invention can reduce audio and video transmission problems, improve stability and user experience, and can be widely applied to the technical field of data transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to an audio and video transmission control method, system, device and storage medium. Background Art

[0002] The relevant video door lock audio and video call module uses the PJSIP protocol. In this protocol, the audio and video bit rate is fixed after the preset initial value. However, the bandwidth of the link in network transmission is limited, and network jitter may also occur, resulting in packet loss, freeze or black screen during live streaming or video calls, resulting in a poor user experience. Summary of the invention

[0003] In view of this, an object of the embodiments of the present invention is to provide an audio and video transmission control method, system, device and storage medium, which can reduce audio and video transmission problems and improve stability and user experience.

[0004] On the one hand, an embodiment of the present invention provides an audio and video transmission control method, including:

[0005] Sending a real-time transmission audio and video data packet according to a preset bit rate, and receiving a real-time transmission control data packet; the real-time transmission control data packet is determined according to the real-time transmission audio and video data packet;

[0006] Determine a delay and a packet loss rate according to the real-time transmission control data packet, and update the preset bit rate according to the delay and the packet loss rate;

[0007] Return to execute sending real-time transmission audio and video data packets according to the preset bit rate until the audio and video data are sent.

[0008] Optionally, updating the preset bit rate according to the delay and the packet loss rate includes:

[0009] If the packet loss rate is greater than a preset second reference value, determining a new bit rate according to the delay and the packet loss rate, and updating the preset bit rate according to the new bit rate;

[0010] If the packet loss rate is less than or equal to the second reference value and the delay is greater than a preset first reference value, determine a new bit rate according to the delay and the packet loss rate, and update the preset bit rate according to the new bit rate;

[0011] If the packet loss rate is less than or equal to the second preset value and the delay is less than or equal to the first reference value, the preset bit rate is kept unchanged.

[0012] Optionally, determining a new bit rate according to the delay and the packet loss rate includes:

[0013] determining a network congestion reference value according to the first reference value, the second reference value and a preset function;

[0014] Determine a true value of network congestion according to the delay, the packet loss rate and the preset function;

[0015] A new bit rate is determined according to the network congestion reference value and the network congestion actual value.

[0016] Optionally, the preset function includes a first weight and a second weight, and the first weight and the second weight are determined by the following method:

[0017] determining a first value according to the first reference value and the first weight;

[0018] determining a second value according to the second reference value and the second weight;

[0019] The first weight and the second weight are determined according to the first value and the second value.

[0020] Optionally, determining a new bit rate according to the network congestion reference value and the network congestion actual value includes:

[0021] Determining a ratio of the network congestion reference value to the network congestion actual value;

[0022] If the ratio is less than a preset value, determining a new bit rate according to the ratio and the preset bit rate before updating;

[0023] If the ratio is greater than or equal to the preset value, the preset bit rate before updating is determined as the new bit rate.

[0024] Optionally, the delay includes an average delay, and the average delay is determined by the following method:

[0025] Obtaining several preceding delays of several real-time transmitted audio and video data packets before the current real-time transmitted audio and video data packet;

[0026] The average delay is determined according to the plurality of previous delays and a plurality of preset delay weights; the plurality of delay weights are determined according to the data sending time and the current time.

[0027] On the other hand, an embodiment of the present invention provides an audio and video transmission control system, including:

[0028] The first module is used to send a real-time transmission audio and video data packet according to a preset bit rate and receive a real-time transmission control data packet; the real-time transmission control data packet is determined according to the real-time transmission audio and video data packet;

[0029] A second module is used to determine the delay and the packet loss rate according to the real-time transmission control data packet, and update the preset bit rate according to the delay and the packet loss rate;

[0030] The third module is used to return to execute the real-time transmission of audio and video data packets according to the preset bit rate until the audio and video data are sent.

[0031] On the other hand, an embodiment of the present invention provides an audio and video transmission control device, including:

[0032] at least one processor;

[0033] at least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0035] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to perform the above method.

[0036] On the other hand, an embodiment of the present invention provides an audio and video transmission control system, including a transmitter, a server and a terminal, wherein the server is connected to the transmitter and the terminal respectively; wherein:

[0037] The sending end is used to execute the above method;

[0038] The server is used to receive the real-time transmission audio and video data packets and feed the real-time transmission audio and video data packets back to the terminal, and receive the real-time transmission control data packets and feed the real-time transmission control data packets back to the sending end;

[0039] The terminal is used to receive the real-time transmission audio and video data packets and feed back the real-time transmission control data packets to the server.

[0040] The implementation of the embodiment of the present invention includes the following beneficial effects: In this embodiment, the sending end determines the delay and packet loss rate based on the real-time transmission control data packet received from the terminal, and updates the preset bit rate for sending real-time transmission audio and video data packets based on the delay and packet loss rate, thereby dynamically determining the preset bit rate of the sending end according to the feedback from the terminal, reducing audio and video transmission problems such as packet loss, freeze or black screen during the transmission of audio and video between the sending end and the terminal, and improving stability and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a structural schematic diagram of an audio and video transmission control system provided by an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of the steps of an audio and video transmission control method provided by an embodiment of the present invention;

[0043] Figure 3 It is a schematic diagram of a process flow of updating a preset bit rate according to a delay and a packet loss rate provided by an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of a process flow for determining a new bit rate based on a delay and a packet loss rate provided by an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of a step flow of determining a preset function provided by an embodiment of the present invention;

[0046] Figure 6 It is a schematic flow chart of a step of determining a new bit rate according to a network congestion reference value and a network congestion actual value provided by an embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of a process flow for determining an average delay provided by an embodiment of the present invention;

[0048] Figure 8 It is a timing diagram of sending a real-time transmission control data packet provided by an embodiment of the present invention;

[0049] Fig. 9 It is a structural block diagram of an audio and video transmission control device provided by an embodiment of the present invention;

[0050] Fig.10 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0051] Fig.11 It is a timing diagram of an audio and video transmission process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] It should be noted that, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Some technical terms in this embodiment are explained below.

[0056] PJSIP is an open source multimedia communication library, mainly used to build applications based on SIP (Session Initiation Protocol). It is widely used in VoIP (Voice over IP) solutions, instant messaging and video communication applications. SIP is an application layer control protocol used to create, modify and terminate multimedia sessions such as voice and video calls in IP networks. PJSIP implements the SIP protocol for managing communication sessions. SDP (Session Description Protocol) is a protocol for describing multimedia sessions, used to negotiate and describe media attributes (such as encoding format, port number) in SIP sessions.

[0057] RTP (Real-time Transport Protocol) is a network protocol designed to provide end-to-end real-time data transmission services. It is commonly used for audio, video and other delay-sensitive data transmission, such as video conferencing, IP telephony, streaming media, etc.

[0058] RTCP (Real-time Transport Control Protocol) is a control protocol used in conjunction with RTP. RTCP provides a mechanism for monitoring data transmission quality and statistics, and supports control communications between session participants.

[0059] like Figure 1 As shown, an embodiment of the present invention provides an audio and video transmission control system, the system includes a transmitter, a server and a terminal. The transmitter sends real-time transmission audio and video data packets to the terminal through the server at a certain bit rate; after receiving the real-time transmission audio and video data packets, the terminal determines the real-time transmission control data packet according to the received real-time transmission audio and video data packets, and feeds the real-time transmission control data packet back to the transmitter through the server; the transmitter re-determines the transmission bit rate of the data packet according to the received real-time transmission control data packet.

[0060] See also Figure 2 The embodiment of the present invention provides an audio and video transmission control method, which is applied to a transmitting end and includes:

[0061] S100, sending real-time transmission audio and video data packets according to a preset bit rate, and receiving real-time transmission control data packets; the real-time transmission control data packets are determined according to the real-time transmission audio and video data packets.

[0062] It should be noted that the preset bit rate is determined according to actual application, and this embodiment does not impose any specific limitation. In actual application, the preset bit rate is dynamically adjusted in real time according to the transmission of audio and video.

[0063] Specifically, the sending end sends the real-time transmission audio and video data packets to the server end according to the preset bit rate. The server end forwards the real-time transmission audio and video data packets to the terminal after receiving them. The terminal determines the real-time transmission audio and video data packets based on the real-time transmission audio and video data packets and sends the real-time transmission audio and video data packets to the server end. The server end forwards the real-time transmission audio and video data packets to the sending end.

[0064] S200: Determine a delay and a packet loss rate according to the real-time transmission control data packet, and update a preset bit rate according to the delay and the packet loss rate.

[0065] After the sender receives the real-time transmission control data packet, it extracts the delay and packet loss rate from the real-time transmission control data packet. The delay and packet loss rate can feedback the congestion and network status of the audio and video transmission link between the sender and the terminal. The preset bit rate is updated according to the delay and packet loss rate. The sender's bit rate can be adjusted in real time according to the congestion of the transmission link, thereby improving the transmission quality of the link.

[0066] The sender can obtain and determine the packet loss rate through the fraction lost of the RTCP RR message, which describes the packet loss rate from the last RR message to the current RR message. The cumulative number of packetslost indicates the total number of lost packets received from the start of receiving SSRC_n data packets to the current moment; fraction lost indicates the packet loss rate since the last RR message was sent.

[0067] S300, returning to execute sending the real-time transmission audio and video data packets according to the preset bit rate until the audio and video data are sent.

[0068] The data to be transmitted by the sender can be divided into several data packets for transmission. The sender sends one data packet to the terminal through the server at a time. After the sender updates the preset bit rate according to the delay and packet loss rate corresponding to the last real-time transmission of the audio and video data packet, the sender sends the next real-time transmission of audio and video data according to the updated preset bit rate until all the audio and video data are sent.

[0069] Optionally, see Figure 3 , update the preset bitrate based on latency and packet loss rate, including:

[0070] S210, if the packet loss rate is greater than a preset second reference value, determine a new bit rate according to the delay and the packet loss rate, and update the preset bit rate according to the new bit rate;

[0071] S220: If the packet loss rate is less than or equal to the second reference value and the delay is greater than the preset first reference value, determine a new bit rate according to the delay and the packet loss rate, and update the preset bit rate according to the new bit rate;

[0072] S230: If the packet loss rate is less than or equal to the second preset value and the delay is less than or equal to the first reference value, keep the preset bit rate unchanged.

[0073] It should be noted that the second reference value and the first reference value are determined according to actual applications, and this embodiment does not impose specific restrictions; for example, the first reference value is 40ms, and the second reference value is 5%. When the packet loss rate is greater than the preset second reference value, it means that network congestion occurs and the sending bit rate needs to be adjusted. The new bit rate is determined according to the delay and packet loss rate, and the preset bit rate is updated according to the new bit rate; when the packet loss rate is less than or equal to the preset second reference value, it means that the network condition is good and the delay condition needs to be further judged. When the delay is greater than the preset first reference value, the current sending bit rate needs to be further reduced to reduce network congestion; when the packet loss rate is less than or equal to the second preset value and the delay is less than or equal to the first reference value, it means that the network transmission is good and the preset bit rate can be kept unchanged.

[0074] Optionally, see Figure 4, determine the new bit rate based on the delay and packet loss rate, including:

[0075] S211, determining a network congestion reference value according to the first reference value, the second reference value and a preset function;

[0076] S212, determining the real value of network congestion according to the delay, packet loss rate and a preset function;

[0077] S213: Determine a new bit rate according to the network congestion reference value and the network congestion actual value.

[0078] It should be noted that the preset function is determined according to actual applications and is not specifically limited in this embodiment. The preset function is a function including a delay parameter and a packet loss rate parameter.

[0079] Specifically, first, the first reference value and the second reference value are substituted into a preset function to determine a network congestion reference value, then, the delay and the packet loss rate are substituted into the preset function to determine a network congestion actual value, and finally, a new bit rate is determined based on the relationship between the network congestion reference value and the network congestion actual value, such as determining the new bit rate based on a size relationship or proportional relationship between the network congestion reference value and the network congestion actual value.

[0080] Optionally, see Figure 5 , the preset function includes a first weight and a second weight, and the first weight and the second weight are determined by the following method:

[0081] S2121, determining a first value according to a first reference value and a first weight;

[0082] S2122, determining a second value according to a second reference value and a second weight;

[0083] S2123. Determine a first weight and a second weight according to the first value and the second value.

[0084] The preset function includes weights corresponding to the delay parameter and the packet loss rate parameter respectively. The first weight represents the weight of the delay, and the second weight represents the weight of the packet loss rate. The first weight and the second weight are determined according to the importance of the delay and the packet loss rate, and the importance can be reflected in the form of proportion, etc.

[0085] In a specific embodiment, the preset bit rate is 1Mbps, the first reference value of the delay is set to 40ms, the second reference value of the packet loss rate is set to 5%, and the expression of the preset function is: Q=delay*J+packet loss rate*K, J represents the first weight, K represents the second weight, delay and packet loss rate are equally important, 40*J=5*K, J+K=1, J=11.11%, K=88.89%, therefore, the preset function is: Q=delay*11.11%+packet loss rate*88.89%.

[0086] Optionally, see Figure 6 , determine the new bit rate according to the network congestion reference value and the network congestion actual value, including:

[0087] S2131, determining a ratio between a network congestion reference value and a network congestion true value;

[0088] S2132, if the ratio is less than the preset value, determine a new bit rate according to the ratio and the preset bit rate before updating;

[0089] S2133: If the ratio is greater than or equal to the preset value, the preset bit rate before updating is determined as the new bit rate.

[0090] It should be noted that the preset value is determined according to actual application and is not specifically limited in this embodiment. For example, the preset value is set to 1.

[0091] In a specific embodiment, the network congestion reference value is recorded as W, and the network congestion real value is recorded as H. If W / H is greater than or equal to 1, the preset bit rate remains unchanged; if W / H is less than 1, a new bit rate needs to be calculated and the data is sent using the new bit rate. The new bit rate calculation formula is as follows: new bit rate = preset bit rate * W / H. The sender sends RTP packets according to the new bit rate (i.e., the updated preset bit rate) to obtain a new W / H ratio. If it is greater than or equal to 1, the bit rate is maintained. Otherwise, the new bit rate is calculated according to the formula and the data is sent using the new bit rate until the value of W / H is greater than or equal to 1.

[0092] Optionally, see Figure 7 , the delay includes the average delay, which is determined by the following method:

[0093] S010, obtaining several preceding delays of several real-time transmitted audio and video data packets before the current real-time transmitted audio and video data packet;

[0094] S020. Determine an average delay according to a number of previous delays and a number of preset delay weights; the number of delay weights are determined according to the data sending time and the current time.

[0095] It should be noted that the farther the data is sent from the current time, the smaller the delay weight corresponding to the data; the closer the data is sent from the current time, the greater the delay weight corresponding to the data.

[0096] In a specific embodiment, see Figure 8, assuming that a group of packets is 100 RTP packets, each RTP contains a frame of video stream data; two RTP data packets are sent at time Ti-1 and Ti, and the receiving end receives these two data packets at ti-1 and ti respectively, then the delay durations of these two RTP packets are ti-1-Ti-1 and ti-Ti respectively, and the delay durations of 100 groups of RTP packets are calculated by analogy, and the delay duration of each RTP packet is recorded as t(n), where n is 1 to 100 and is fed back to the sender through the RTCP packet; after the sender receives the delay duration of 100 groups of RTP packets, the average delay of this group of RTP packets is calculated according to the convolution formula and recorded as t(w), and the specific calculation is as follows.

[0097] According to the convolution formula:

[0098]

[0099] In combination with the embodiment of the present invention, a weighted summation formula for calculating the average delay can be obtained:

[0100]

[0101] Wherein, t represents delay, g represents weight, t(n) represents the delay of the nth RTP packet, g(100-n) represents the weight before the (100-n)th frame, for example, when n=1, t(1) represents the delay duration of the 1st RTP packet, and g(100-1) represents the weight before the 99th frame, i.e., the weight of the first frame; in the embodiment of the present invention, the transmitting end obtains a group of delay data of RTP packets, and it is considered that the delay with the closest time distance should have the highest weight, so it can start from 1 and increase by 1 each time to generate an arithmetic progression, and then these The values ​​are normalized (i.e. divided by their sum) to obtain a set of weight arrays, denoted as g(n), where n is 1 to 100; the average delay can be calculated based on the weighted sum to obtain the average delay duration, which is denoted as t(w); PJSIP is set to send 20 frames of video stream data within 1 second, so the time required to send each frame of video stream is 50ms, and the delay needs to be less than 50ms. Based on 40ms, if t(w)>40ms, the delay is considered to be too long; if t(w)<40ms, it is considered to be within the acceptable delay range.

[0102] The implementation of the embodiment of the present invention includes the following beneficial effects: In this embodiment, the sending end determines the delay and packet loss rate based on the real-time transmission control data packet received from the terminal, and updates the preset bit rate for sending real-time transmission audio and video data packets based on the delay and packet loss rate, thereby dynamically determining the preset bit rate of the sending end according to the feedback from the terminal, reducing audio and video transmission problems such as packet loss, freeze or black screen during the transmission of audio and video between the sending end and the terminal, and improving stability and user experience.

[0103] See also Fig. 9 , an embodiment of the present invention provides an audio and video transmission control system, comprising:

[0104] The first module is used to send real-time transmission audio and video data packets according to a preset bit rate and receive real-time transmission control data packets; the real-time transmission control data packets are determined according to the real-time transmission audio and video data packets;

[0105] The second module is used to determine the delay and packet loss rate according to the real-time transmission control data packet, and update the preset bit rate according to the delay and packet loss rate;

[0106] The third module is used to return to execute the real-time transmission of audio and video data packets according to the preset bit rate until the audio and video data are sent.

[0107] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0108] The embodiment of the present invention further provides an audio and video transmission control device, comprising:

[0109] at least one processor;

[0110] at least one memory for storing at least one program;

[0111] When the at least one program is executed by the at least one processor, the at least one processor implements the steps of the audio and video transmission control method described in the above method embodiment.

[0112] Among them, the memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a remote memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0113] It can be seen that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0114] In addition, the embodiment of the present application also discloses a computer program product or a computer program, and the computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above method. Similarly, the contents in the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those in the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0115] An embodiment of the present invention further provides a computer-readable storage medium, which stores a program executable by a processor. The program executable by the processor is used to implement the above method when executed by the processor.

[0116] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0117] Specifically, see Fig.10The computer device 1000 may include an RF (Radio Frequency) circuit 1010, a memory 1020 including one or more computer-readable storage media, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a short-range wireless transmission module 1070, a processor 1080 including one or more processing cores, and a power supply 1090. Those skilled in the art will appreciate that Fig.10 The device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0118] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 1080 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a user identity module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0119] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device 1000 (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1020 may also include a memory controller to provide the processor 1080 and the input unit 1030 with access to the memory 1020. Although Fig.10 The RF circuit 1010 is shown, but it is understandable that it is not a necessary component of the device 1000 and can be omitted as needed without changing the essence of the invention.

[0120] The input unit 1030 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 1030 may include a touch-sensitive surface 1031 and other input devices 1032. The touch-sensitive surface 1031, also known as a touch display screen or a touch pad, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch-sensitive surface 1031), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch-sensitive surface 1031 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0121] The display unit 1040 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the control 1000, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 1031 may be covered on the display panel 1041. When the touch-sensitive surface 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.10 In the embodiment, the touch-sensitive surface 1031 and the display panel 1041 are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface 1031 and the display panel 1041 can be integrated to implement input and output functions.

[0122] The computer device 1000 may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the device 1000 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that recognize the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can also be configured in the device 1000, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0123] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the device 1000. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output by the speaker 1061; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is processed by the output processor 1080 and sent to another control device through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing. The audio circuit 1060 may also include an earphone jack to provide communication between an external headset and the device 1000.

[0124] The short-distance wireless transmission module 1070 may be a WIFI (wireless fidelity) module, a Bluetooth module, an infrared module, etc. The device 1000 may transmit information with a wireless transmission module provided on a competing device through the short-distance wireless transmission module 1070 .

[0125] The processor 1080 is the control center of the device 1000. It uses various interfaces and lines to connect various parts of the entire control device. By running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, it executes various functions of the device 1000 and processes data, thereby monitoring the control device as a whole. Optionally, the processor 1080 may include one or more processing cores; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1050.

[0126] The device 1000 also includes a power supply 1090 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 1090 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0127] Although not shown, the device 1000 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0128] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0129] The embodiment of the present invention provides an audio and video transmission control system, including a sending end, a service end and a terminal, wherein the service end is connected to the sending end and the terminal respectively; wherein:

[0130] A sending end, used to execute the above method;

[0131] The server is used to receive the real-time transmission audio and video data packets and feed the real-time transmission audio and video data packets back to the terminal, and receive the real-time transmission control data packets and feed the real-time transmission control data packets back to the sending end;

[0132] The terminal is used to receive real-time transmission audio and video data packets and feed back real-time transmission control data packets to the server.

[0133] Specifically, for the sending end, it may include but is not limited to video or audio acquisition components, data processing components and data storage components; the service end includes but is not limited to servers and other devices with data transmission functions; and for the terminal, it can be different types of electronic devices, including but not limited to speakers, wearable devices, video playback devices, etc.

[0134] See also Fig.11In a specific embodiment, a video lock is set at the sending end, the video lock controls, the terminal is an audio device, and the video lock controls the data transmission of the sending end. The video lock at the sending end sends RTP packets to the server at a preset bit rate, and the server forwards the received RTP packets to the terminal device. The terminal device determines the reply RTCP packet based on the received RTP packet, and sends the RTCP packet to the server; the server forwards the RTCP packet to the video lock at the sending end. The video lock at the sending end obtains the delay and packet loss rate based on the RTCP packet, calculates the average delay based on the delays of multiple RTCP packets, and calculates the new video encoding bit rate based on the average delay and packet loss rate, and sends the next RTP packet to the terminal device through the server at the new video encoding bit rate.

[0135] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0137] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for controlling audio and video transmission, characterized in that: include: Send real-time transmission audio and video data packets according to the preset bit rate and receive real-time transmission control data packets; The real-time transmission control data packet is determined according to the real-time transmission audio and video data packet; Determine a delay and a packet loss rate according to the real-time transmission control data packet, and update the preset bit rate according to the delay and the packet loss rate; Return to execute sending real-time transmission audio and video data packets according to the preset bit rate until the audio and video data are sent.

2. The method according to claim 1, characterized in that: The updating of the preset bit rate according to the delay and the packet loss rate includes: If the packet loss rate is greater than a preset second reference value, determining a new bit rate according to the delay and the packet loss rate, and updating the preset bit rate according to the new bit rate; If the packet loss rate is less than or equal to the second reference value and the delay is greater than a preset first reference value, determine a new bit rate according to the delay and the packet loss rate, and update the preset bit rate according to the new bit rate; If the packet loss rate is less than or equal to the second preset value and the delay is less than or equal to the first reference value, the preset bit rate is kept unchanged.

3. The method according to claim 2, characterized in that The determining a new bit rate according to the delay and the packet loss rate includes: determining a network congestion reference value according to the first reference value, the second reference value and a preset function; Determine a true value of network congestion according to the delay, the packet loss rate and the preset function; A new bit rate is determined according to the network congestion reference value and the network congestion actual value.

4. The method according to claim 3, characterized in that The preset function includes a first weight and a second weight, and the first weight and the second weight are determined by the following method: determining a first value according to the first reference value and the first weight; determining a second value according to the second reference value and the second weight; The first weight and the second weight are determined according to the first value and the second value.

5. The method according to claim 3, characterized in that: The determining a new bit rate according to the network congestion reference value and the network congestion actual value comprises: Determining a ratio of the network congestion reference value to the network congestion actual value; If the ratio is less than a preset value, determining a new bit rate according to the ratio and the preset bit rate before updating; If the ratio is greater than or equal to the preset value, the preset bit rate before updating is determined as the new bit rate.

6. The method according to any one of claims 1 to 5, characterized in that: The delay includes an average delay, and the average delay is determined by the following method: Obtaining several preceding delays of several real-time transmitted audio and video data packets before the current real-time transmitted audio and video data packet; The average delay is determined according to the plurality of previous delays and a plurality of preset delay weights; the plurality of delay weights are determined according to the data sending time and the current time.

7. An audio and video transmission control system, characterized in that: include: The first module is used to send real-time transmission audio and video data packets according to a preset bit rate and receive real-time transmission control data packets; The real-time transmission control data packet is determined according to the real-time transmission audio and video data packet; A second module is used to determine the delay and the packet loss rate according to the real-time transmission control data packet, and update the preset bit rate according to the delay and the packet loss rate; The third module is used to return to execute the real-time transmission of audio and video data packets according to the preset bit rate until the audio and video data are sent.

8. An audio and video transmission control device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

10. An audio and video transmission control system, characterized in that: It includes a sending end, a service end and a terminal, wherein the service end is connected to the sending end and the terminal respectively; wherein, The sending end is used to execute the method according to any one of claims 1 to 6; The server is used to receive the real-time transmission audio and video data packets and feed the real-time transmission audio and video data packets back to the terminal, and receive the real-time transmission control data packets and feed the real-time transmission control data packets back to the sending end; The terminal is used to receive the real-time transmission audio and video data packets and feed back the real-time transmission control data packets to the server.

Images