Multimedia resource transmission method and device, equipment and medium

Abstract

The invention discloses a multimedia resource transmission method and device, equipment and a medium, and the method comprises the steps: detecting network quality information with a client, and receiving first experience quality information fed back by the client at a current time node; according to the network quality information, a first resource set can be determined from a plurality of resource backups with different code rates corresponding to the multimedia resources, and then the first resource set is determined according to the change condition of the first experience quality information relative to the experience quality information fed back by the client before the current time node. According to the embodiment of the invention, the target resource backup can be determined from the first resource set, and then the target multimedia resource data sent to the client is generated. According to the invention, the network quality and the service experience of the client can be integrated, the transmission strategy of the multimedia resources can be cooperatively adjusted, and the playing effect of the client can be optimized. The technical scheme of the invention can be widely applied to various scenes such as cloud technology and content distribution.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method, apparatus, device, and medium for transmitting multimedia resources. Background Art

[0002] With the rapid development of information technology, related applications have gradually become integrated into people's lives, providing a wide variety of services. For example, multimedia applications exist in some scenarios. Multimedia applications use related media technologies (such as audio, video, and images) to create, transmit, and present multimedia content. Multimedia applications typically include a client and a server. The server is responsible for storing, encoding, and transmitting multimedia resources, while the client is responsible for receiving and playing multimedia resources.

[0003] In related technologies, when transmitting multimedia resources between servers and clients, certain resource specifications, such as video resolution, are typically set to suit the client's playback requirements. However, in practice, due to the complexity and volatility of network and client hardware environments, servers often struggle to determine the appropriate bitrate for transmission. This results in poor utilization of network bandwidth and compromised client playback quality.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, and medium for transmitting multimedia resources, which can collaboratively adjust the transmission strategy of multimedia resources and optimize the playback effect of the client.

[0006] An aspect of an embodiment of the present application provides a method for transmitting multimedia resources, the method comprising:

[0007] Detect network quality information between the client and the client;

[0008] receiving first experience quality information fed back by the client at a current time point;

[0009] Determining, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each of the resource backups has a different bit rate, and the first resource set includes a predetermined number of resource backups;

[0010] Determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generating target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node;

[0011] The target multimedia resource data is sent to the client.

[0012] On the other hand, an embodiment of the present application provides a multimedia resource transmission device, the device comprising:

[0013] A detection unit, used to detect network quality information between the client and the network;

[0014] A receiving unit, configured to receive first experience quality information fed back by the client at a current time node;

[0015] a processing unit, configured to determine, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each of the resource backups has a different bit rate, and the first resource set includes a predetermined number of resource backups;

[0016] a generating unit, configured to determine a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generate target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node;

[0017] A sending unit is used to send the target multimedia resource data to the client.

[0018] Optionally, the network quality information includes available network bandwidth; and the processing unit is specifically configured to:

[0019] sorting the resource backups according to their bit rates;

[0020] Compare the available network bandwidth and the sizes of each of the bit rates, and determine a predetermined number of consecutively ordered resource backups as a first resource set based on the comparison result; wherein, the first resource set includes resource backups whose corresponding bit rates are less than the available network bandwidth, and the first resource set includes resource backups whose corresponding bit rates are greater than the available network bandwidth.

[0021] Optionally, the processing unit is specifically configured to:

[0022] Record the sorting numbers corresponding to each resource backup in sorted order;

[0023] Comparing the available network bandwidth with the respective bit rates, and determining a first bit rate from the bit rates of the resource backups; wherein the first bit rate is a bit rate that is closest to the available network bandwidth among the bit rates that are less than or equal to the available network bandwidth;

[0024] Querying a first sorting number corresponding to a first resource backup; wherein the first resource backup is a resource backup corresponding to the first bit rate;

[0025] Dividing the difference obtained by subtracting 1 from the predetermined number by 2, and rounding the quotient to obtain a first value;

[0026] Determining a second sort number according to a difference between the first sort number and the first value;

[0027] Starting from the second sorting number, in the sorting direction of increasing bit rate, the predetermined number of resource backups are selected in sequence to obtain a first resource set.

[0028] Optionally, the first experience quality information includes first jamming indicator data; and the generating unit is specifically configured to:

[0029] Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information;

[0030] If the jamming situation of the client worsens, a target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

[0031] Optionally, the generating unit is specifically configured to:

[0032] Determining second experience quality information from the historical experience quality information; the second experience quality information is experience quality information fed back by the client at a time node previous to the current time node, and the second experience quality information includes second jamming indicator data;

[0033] Calculating a first difference value between the first jamming indicator data and the second jamming indicator data;

[0034] comparing the first difference value with a first threshold;

[0035] If the first difference value is less than or equal to the first threshold, it is determined that the client's jamming situation has not worsened; or, if the first difference value is greater than the first threshold, it is determined that the client's jamming situation has worsened.

[0036] Optionally, the generating unit is specifically configured to:

[0037] Determine, based on the historical quality of experience information, a first cumulative value corresponding to the jamming indicator data before the current time node;

[0038] Performing a weighted summation on the first jamming indicator data and the first cumulative value to obtain a second cumulative value;

[0039] calculating a second difference value of the second cumulative value relative to the first cumulative value;

[0040] Comparing the second difference value with the second threshold;

[0041] If the second difference value is less than or equal to the second threshold, it is determined that the client's jamming situation has not worsened; or, if the second difference value is greater than the second threshold, it is determined that the client's jamming situation has worsened.

[0042] Optionally, the generating unit is specifically configured to:

[0043] If the jamming condition of the client worsens, determining a jamming deterioration level corresponding to the client according to the first difference value and the first threshold;

[0044] According to the jamming deterioration level, a target resource backup is determined from resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

[0045] Optionally, the first experience quality information includes first jamming indicator data and first cache indicator data; and the generating unit is specifically configured to:

[0046] Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information;

[0047] determining, based on the first cache indicator data and the historical quality of experience information, whether a cache situation of the client has deteriorated;

[0048] If the client's lag situation does not worsen and the client's cache situation worsens, determine that the second resource backup in the first resource set is the target resource backup; wherein, the second resource backup is a resource backup corresponding to a second bit rate, and the second bit rate is greater than or equal to the bit rate used by the client at the previous time node of the current time node.

[0049] Optionally, the generating unit is specifically configured to:

[0050] If the jamming situation of the client does not worsen, and the cache situation of the client worsens, frame loss processing is performed on the target resource backup to generate target multimedia resource data.

[0051] Optionally, the generating unit is specifically configured to:

[0052] Detecting the frame type of each resource frame in the target resource backup;

[0053] Determining, according to the frame type, a target frame to be discarded in the target resource backup;

[0054] Detecting a playback time interval between two resource frames adjacent to the target frame;

[0055] If the playback time interval is less than or equal to a third threshold, the target frame is discarded; or if the playback time interval is greater than the third threshold, the target frame is retained.

[0056] Optionally, the generating unit is specifically configured to:

[0057] Detecting the frame type of each resource frame in the target resource backup;

[0058] Determining, according to the frame type, a target frame to be discarded in the target resource backup;

[0059] Determine the first consecutive target frames as a frame set;

[0060] A target frame in the frame set is discarded, and other target frames in the frame set except the discarded target frame are retained.

[0061] Optionally, the first experience quality information includes first jamming indicator data and first cache indicator data; and the generating unit is specifically configured to:

[0062] Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information;

[0063] determining, based on the first cache indicator data and the historical quality of experience information, whether a cache situation of the client has deteriorated;

[0064] If the client's freezing condition does not worsen and the client's cache condition does not worsen, a target resource backup is determined from resource backups in the first resource set whose corresponding bit rates are greater than the available network bandwidth.

[0065] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0066] The memory is used to store computer programs;

[0067] The processor executes the computer program to implement the aforementioned method for transmitting multimedia resources.

[0068] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the aforementioned method for transmitting multimedia resources.

[0069] The embodiments of the present application include at least the following beneficial effects: The present application provides a method, apparatus, device and medium for transmitting multimedia resources, which detects network quality information between the server and the client, and receives first experience quality information fed back by the client at the current time node. The network quality information can reflect the network situation between the server and the client, and the first experience quality information can reflect the actual service situation of the client playing the multimedia resources; based on the network quality information, a first resource set can be determined from a plurality of resource backups with different bit rates corresponding to the multimedia resources, and then, based on the change of the first experience quality information relative to the experience quality information fed back by the client before the current time node, a target resource backup can be determined from the first resource set, and then target multimedia resource data sent to the client is generated. The present application can comprehensively consider the network quality and the client's service experience, and coordinately adjust the transmission strategy of the multimedia resources, which is beneficial to improving the response efficiency of the multimedia application server and optimizing the playback effect of the client. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0071] Figure 1 A schematic diagram of a transmission of multimedia resources provided in an embodiment of the present application;

[0072] Figure 2 A schematic diagram of an implementation environment of a multimedia resource transmission method provided in an embodiment of the present application;

[0073] Figure 3 A flowchart of a method for transmitting multimedia resources provided in an embodiment of the present application is provided;

[0074] Figure 4 A schematic diagram of another multimedia resource transmission method provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of a specific process of multimedia resource transmission provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of obtaining and transmitting quality of experience information provided in an embodiment of the present application;

[0077] Figure 7A schematic diagram of selecting an alternative set provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of determining a playback time interval provided in an embodiment of the present application;

[0079] Figure 9 A schematic diagram of frame loss processing provided in an embodiment of the present application;

[0080] Figure 10 A schematic structural diagram of a multimedia resource transmission device provided in an embodiment of the present application;

[0081] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0083] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0084] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0086] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0087] 1) TCP (Transport Control Protocol), a connection-oriented, reliable transport layer protocol used to transmit data in computer networks.

[0088] 2) QUIC (Quick UDP Internet Connection), a transport protocol based on UDP (User Datagram Protocol), aims to provide faster network connection speeds and lower latency.

[0089] 3) Blockchain is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks generated using cryptographic methods. Each block contains information about a batch of network transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product and service layer, and the application service layer. Blockchains can include public chains, consortium chains, and private chains. Public chains are blockchains where anyone can access the blockchain network at any time to read data, send data, or compete for recordkeeping. Consortium chains are blockchains jointly managed by several organizations or institutions. Private chains are blockchains with a certain degree of centralized control. Write access to the private chain's ledger is controlled by a specific organization or institution, and data access and use are subject to strict permission management.

[0090] With the rapid development of information technology, related applications have gradually become integrated into people's lives, providing a wide variety of services. For example, multimedia applications exist in some scenarios. Multimedia applications use related media technologies (such as audio, video, and images) to create, transmit, and present multimedia content. Multimedia applications typically include a client and a server. The server is responsible for storing, encoding, and transmitting multimedia resources, while the client is responsible for receiving and playing multimedia resources.

[0091] In related technologies, when multimedia resources are transmitted between a server and a client, some resource specifications are generally fixed, such as the definition specifications of video resources, to facilitate adaptation to the playback conditions of the client. Figure 1 , Figure 1 A transmission diagram of a multimedia resource provided in an embodiment of the present application is shown. Figure 1In this example, the multimedia resource is audio and video data. The server, as the sender of the audio and video data, may include an application layer and a transport layer. The application layer may include an encoder responsible for encoding the audio and video data into a specified data format for easy transmission to the client. The client, as the receiver of the audio and video data, may include a player for playing the received audio and video data.

[0092] Since the network environment between the sender and different receivers, as well as the hardware environment of different receivers, may be different, in order to maximize the transmission performance of audio and video data, so that the audio and video data can be transmitted to the receiver reliably and stably, and reduce the occurrence of freezes, etc., for the same audio and video data, the sender often sets different resource specifications, such as multiple definition specifications. Each definition specification corresponds to a different transmission bit rate and frame rate, thus forming multiple resource files. For details, please refer to Figure 1 For example, for audio and video data, four definition specifications can be set, namely 1080p, 720p, 360p and 144p, and their corresponding bit rates are 5Mbps, 2.5Mbps, 1Mbps and 0.5Mbps respectively. When the sender transmits audio and video data to the receiver, it can detect the network conditions between it and the receiver, so as to adapt the appropriate resource files for the receiver. For example, if the sender detects that the available network bandwidth between it and the receiver is reduced from 5Mbps to 3Mbps, it can adaptively replace the definition specification of the transmitted audio and video data from 1080p to 720p, thereby reducing the probability of audio and video data transmission being affected and improving the stability of multimedia resource playback.

[0093] However, in actual applications, it is found that the environment of the network and client hardware is highly complex and prone to fluctuations. If the transmission bit rate is regulated solely by the network conditions between the server and the client, the available network bandwidth may not be fully utilized, which may easily affect the actual playback service experience of the client. In order to improve the above problems, the server needs to set as many different resource specifications as possible to adapt to changes in network conditions. However, setting more resource specifications will increase the encoding and storage overhead of the server and increase the implementation cost of multimedia resource transmission. Therefore, the current implementation method is difficult to adapt to the environmental complexity of the network and client hardware in the transmission of multimedia resources, which can easily affect the playback effect of the client.

[0094] In view of this, an embodiment of the present application provides a method, apparatus, device and medium for transmitting multimedia resources, which detects network quality information between the server and the client, and receives first experience quality information fed back by the client at the current time node. The network quality information can reflect the network situation between the server and the client, and the first experience quality information can reflect the actual service situation of the client playing the multimedia resources. According to the network quality information, a first resource set can be determined from a plurality of resource backups with different bit rates corresponding to the multimedia resources, and then, according to the change of the first experience quality information relative to the experience quality information fed back by the client before the current time node, a target resource backup can be determined from the first resource set, and then the target multimedia resource data sent to the client is generated. The present application can comprehensively consider the network quality and the client's service experience, and coordinately adjust the transmission strategy of the multimedia resources, which is conducive to improving the response efficiency of the multimedia application server and optimizing the playback effect of the client.

[0095] The method for transmitting multimedia resources provided in the embodiments of the present application mainly relates to the field of information technology, and may include, but is not limited to, various application scenarios such as cloud technology and content distribution. It will be understood by those skilled in the art that the method for transmitting multimedia resources provided in the embodiments of the present application can be implemented in various application scenarios. Specifically, an exemplary description is given below:

[0096] For example, in some embodiments, the multimedia resource transmission method of the embodiments of this application can be applied in the context of live webcasting. Live webcasting is the transmission of real-time video and audio content to the audience's terminal devices via the network. In live webcasting, multimedia resources are transmitted in real time, allowing the audience to watch and listen to the content instantly. Common applications of live webcasting include online live broadcast platforms, live sports events, live music and concerts, etc., which are not limited by this application.

[0097] For example, in some embodiments, the multimedia resource transmission method of the embodiments of the present application can be applied in the video playback scenario. Video playback involves transmitting recorded or stored video content to the viewer's device for viewing. In the video playback scenario, the multimedia resource can be accessed through streaming services or download transmission, and the viewer can watch the video content through an online video platform or a local video player.

[0098] For example, in some embodiments, the multimedia resource transmission method of the embodiments of the present application can be applied in the context of online conferencing. For example, online conferencing connects multiple participants over a network for real-time audio and video communication and collaboration. In online conferencing, multimedia resources are transmitted in real time, including video, audio, and shared screens between participants.

[0099] Of course, it is understood that the above application scenarios are merely exemplary and are not intended to limit the practical application of the multimedia resource transmission method provided in the embodiments of the present application. Those skilled in the art will understand that the multimedia resource transmission method provided in the embodiments of the present application can be used to perform specified tasks in different application scenarios.

[0100] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on the object's information, the object's behavioral data, the object's historical data, and the object's location information, etc., which are related to the object's identity or characteristics, the object's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain sensitive information of the object, the object's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the object's separate permission or consent, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0101] The following describes the specific implementation of the embodiment of the present application in detail with reference to the accompanying drawings. First, a method for transmitting multimedia resources provided in the embodiment of the present application is described with reference to the accompanying drawings.

[0102] Please refer to Figure 2 , Figure 2 The following is a schematic diagram showing an implementation environment of a multimedia resource transmission method provided in an embodiment of the present application. In this implementation environment, the main hardware and software entities involved include a terminal device 210 and a server 220. The terminal device 210 and the server 220 are in communication connection with each other.

[0103] Specifically, the multimedia resource transmission method provided in the embodiments of the present application can be executed on the server 220 side.

[0104] For example, in some embodiments, a software client for playing multimedia resources may be installed on terminal device 210, and server 220 may be the backend server of the software client. Terminal device 210 may send a request to server 220 to obtain multimedia resources, and server 220 may transmit the multimedia resources to terminal device 210. Specifically, in embodiments of the present application, server 220 may detect network quality information between terminal device 210 and receive experience quality information fed back by terminal device 210, which includes first experience quality information corresponding to a current time node and historical experience quality information fed back before the current time node. Next, server 220, based on the network quality information, determines a first resource set from multiple resource backups corresponding to the multimedia resource. The first resource set includes a predetermined number of resource backups, each having a different bitrate. Furthermore, server 220, based on changes in the first experience quality information relative to the historical experience quality information, determines a target resource backup from the first resource set, generates target multimedia resource data based on the target resource backup, and transmits it to terminal device 210. After receiving the target multimedia resource data, the terminal device 210 plays it, and then can further collect and feedback experience quality information to the server 220, and execute this cycle, thereby improving the stability of multimedia resource transmission and improving the multimedia resource playback effect of the client.

[0105] Among them, the terminal device 210 in the above embodiment may include a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto.

[0106] Server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0107] In addition, the server 220 may also be a node server in the blockchain network.

[0108] The terminal device 210 and the server 220 can establish a communication connection via a wireless network or a wired network. The wireless network or wired network uses standard communication technologies and / or protocols. The network can be set to the Internet or any other network, such as but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. Furthermore, the above-mentioned software and hardware entities can use the same communication connection method or different communication connection methods, and this application does not impose specific restrictions on this.

[0109] Of course, it is understandable that Figure 2 The implementation environment in the embodiment of the present application is only some optional application scenarios of the multimedia resource transmission method provided in the embodiment of the present application. The actual application is not fixed to Figure 2 The method provided in the embodiment of the present application can be applied to various technical fields, such as cloud technology, content distribution and other fields, and the present application does not impose any specific limitation on this.

[0110] Below, in combination with the above description of the implementation environment, a method for transmitting multimedia resources provided in an embodiment of the present application is introduced and explained in detail.

[0111] like Figure 3 As shown, in the embodiment of the present application, a method for transmitting multimedia resources is provided, which can be applied to Figure 2 In the server 220 shown. Figure 3 The multimedia resource transmission method provided in the embodiment of the present application specifically includes but is not limited to steps 310 to 350:

[0112] Step 310: Detect network quality information between the client and the client;

[0113] In an embodiment of the present application, the provided method for transmitting multimedia resources can be executed by a server, which can be connected to a client for communication, receive relevant requests or feedback information from the client, and transmit multimedia resources to the client. Specifically, in an embodiment of the present application, there is no restriction on the data type of multimedia resources transmitted between the server and the client. For example, in some embodiments, the multimedia resources transmitted between the server and the client may include audio and video data, which includes a series of continuous images and their corresponding audio data, which may be real-time (such as live broadcast) or recorded (such as movies and TV programs); in some embodiments, the multimedia resources transmitted between the server and the client may include audio data; in other embodiments, the multimedia resources transmitted between the server and the client may include image data, which may be static or dynamic, and the present application does not impose any restrictions on this.

[0114] It should be noted that in the embodiments of the present application, the server can communicate with multiple different clients simultaneously, and the present application does not impose any restrictions on the specific number of client connections. Furthermore, the multimedia resources transmitted between the server and different clients can be of the same data type or of different data types. For example, the server can transmit audio and video data to some clients while transmitting audio data to other clients. The multimedia resource transmission method provided in the embodiments of the present application can be used in both processes.

[0115] In this step, when transmitting multimedia resources to the client, the server may detect network quality information between the server and the client. Specifically, in embodiments of the present application, the network quality information detected by the server may include one or more data indicators that can measure the network quality between the server and the client. For example, network quality information may include, but is not limited to, data indicators such as latency (delay), network available bandwidth, network packet loss rate, and network jitter. Latency refers to the time elapsed between sending and receiving data. Lower latency means data can reach the client faster, resulting in better network communication performance, which can reduce the waiting time for multimedia resource playback and improve real-time performance. Network available bandwidth refers to the rate at which network data is transmitted, indicating the amount of data that can be transmitted per unit time. Higher network available bandwidth means that larger amounts of data can be transmitted faster, facilitating the provision of higher-quality multimedia resources. Network packet loss rate refers to the proportion of lost data packets to the original data packets sent during data transmission. A lower packet loss rate indicates a more stable network and good data transmission guarantee. Conversely, a higher packet loss rate indicates lower network stability and insufficient data transmission reliability. Network jitter refers to the delay variation of data packets during transmission. Similarly, lower network jitter means more stable intervals between data packets and better data transmission reliability, which facilitates smoother multimedia resource playback.

[0116] In this step, there is no restriction on the specific implementation method of the server detecting network quality information, which can be implemented according to actual needs and in combination with technologies in related fields. For example, in some embodiments, the server can send a small test data packet to the client and measure the round-trip delay of the test data packet. After multiple tests, relatively accurate indicator data such as delay and network jitter can be obtained; in some embodiments, the server can send a test data packet of a certain size to the client and then detect the time required for its transmission, thereby calculating the indicator data of the network available bandwidth; in some embodiments, the server side can also be installed with relevant network diagnostic tools, which can track and detect the test data packets sent to the client through the network diagnostic tools to determine indicator data such as delay, packet loss rate, and network jitter.

[0117] Step 320: Receive first experience quality information fed back by the client at the current time node;

[0118] In an embodiment of the present application, while playing multimedia resources, the client can periodically monitor the playback status of the multimedia resources and summarize the quality of experience information. The quality of experience information can be used to characterize the playback performance of the multimedia resources on the client side and can also include at least one indicator data. In the embodiment of the present application, there is no limitation on the type of indicator data included in the quality of experience information. For example, in some embodiments, the quality of experience information can include, but is not limited to, indicator data related to stuttering, caching, and playback frame rate. Stuttering refers to the phenomenon of pauses or interruptions during the playback of multimedia resources. Stuttering indicator data can further include the number of stutters and the duration of stuttering. Cache indicator data can include the duration of cached resource frames, the number of frames, and the specific size of the cached resource. The playback frame rate can be used to record the average number of frames rendered per second during the playback of video-type multimedia resources, which can indicate the smoothness of the video playback. Of course, it is understood that the quality of experience information in the embodiment of the present application is not limited to the indicator data types described above. The specific content included in the present application can be set according to actual needs.

[0119] After the client obtains the experience quality information through aggregation, it can send it to the server. In this step, the server can receive the experience quality information fed back by the client at the current time node and record it as the first experience quality information.

[0120] It is understood that in the embodiments of the present application, the client can periodically feedback experience quality information to the server. Therefore, if the client has been playing multimedia resources for a period of time, the server will also receive experience quality information feedback from the client once or multiple times before the current time point. In the embodiments of the present application, the experience quality information feedback from the client to the server before the current time point is recorded as historical experience quality information. The historical experience quality information can specifically include at least one experience quality information feedback from the client.

[0121] In the embodiments of the present application, there is no restriction on the way in which the client detects the playback status of multimedia resources to obtain experience quality information. For example, in some embodiments, the client side can be configured with a relevant plug-in, which can be a performance monitoring tool. By calling the relevant interface, it can obtain the number of frames rendered per second when the client plays the multimedia resource, the timestamps of the start and end of each rendering, the stored data in the buffer, etc., so as to calculate the indicator data related to the jamming, playback frame rate, etc.; or it can monitor the pause or interruption events during the playback of the multimedia resource and analyze the indicator data related to the jamming. In addition, the present application does not limit the length of the period for the client to feedback the experience quality information to the server, which can be flexibly set according to actual needs.

[0122] Step 330: Determine a first resource set from multiple resource backups corresponding to the multimedia resource based on the network quality information; wherein the bit rates of the resource backups are different, and the first resource set includes a predetermined number of resource backups.

[0123] In the embodiments of the present application, the server can configure resource backups of different specifications for each multimedia resource. Specifically, for the application of the present application, different bit rates are configured for each resource backup, thereby conveniently adapting to various network transmission environments. Of course, in the embodiments of the present application, for each multimedia resource, the number of corresponding resource backups can be two or more, and the present application does not impose any restrictions on the specific number.

[0124] Furthermore, it should be noted that, in the embodiments of the present application, the number of resource backups corresponding to different multimedia resources may be the same or different. For example, for multimedia resources with greater transmission requirements, the corresponding number of resource backups may be set to be greater; and for multimedia resources with less transmission requirements, the corresponding number of resource backups may be set to be less.

[0125] In this step, after the server determines the network quality information between the server and the client, a resource set can be determined from the multiple resource backups corresponding to the multimedia resource required by the client based on the network quality information. The resource set can include some or all of the multiple resource backups corresponding to the multimedia resource. In this embodiment of the present application, this resource set is recorded as a first resource set. The number of resource backups included in the first resource set can be determined in advance. In this embodiment of the present application, this number is recorded as a predetermined number.

[0126] Specifically, in this step, based on the network quality information, the first resource set is determined from the multiple resource backups corresponding to the multimedia resources. This can be done by combining the current network quality information to determine some resource backups that are more suitable for transmission. For example, assuming that the current network available bandwidth is 2.7Mbps, the bit rates corresponding to the multiple resource backups include 0.5Mbps, 1Mbps, 2.5Mbps, 3Mbps and 5Mbps, and the predetermined number is 2, then the resource backups with the two bit rates closest to the current network available bandwidth can be selected as the first resource set, that is, the resource backups with the corresponding bit rates of 2.5Mbps and 3Mbps can be determined to constitute the first resource set. Of course, in other embodiments, the first resource set can also be determined by combining other data indicators reflecting the network quality in the network quality information, and this application does not impose any restrictions on this.

[0127] It should be added that in the embodiment of the present application, before determining the first resource set, it is necessary to first determine the multimedia resources required by the client. This can be determined by parsing the client's request message. The relevant technology has been widely used and will not be elaborated in this application.

[0128] It is understandable that, in the embodiment of the present application, in order to successfully determine the first resource set from the multiple resource backups corresponding to the multimedia resource, it should be ensured that the total number of resource backups corresponding to the multimedia resource is greater than or equal to the set predetermined number. Therefore, in the embodiment of the present application, when setting the predetermined number, one can first query all multimedia resources transmitted using the method in the embodiment of the present application, the number of resource backups corresponding to each of them, determine the minimum value therefrom, and then set the predetermined number to be less than or equal to the minimum value, so as to facilitate the subsequent determination of the first resource set for each multimedia resource.

[0129] Step 340: Determine a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generate target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time point;

[0130] In this step, for the obtained first resource set, a target resource backup can be determined based on the change of the first experience quality information relative to the historical experience quality information, and then target multimedia resource data is generated based on the target resource backup.

[0131] Specifically, in an embodiment of the present application, the first resource set includes a predetermined number of resource backups, which may include resource backups with corresponding bit rates greater than the available network bandwidth and resource backups with corresponding bit rates less than the available network bandwidth. In an embodiment of the present application, based on the first experience quality information and the historical experience quality information, it can be determined whether the playback effect of the current client is in a good situation or deteriorated. It can be understood that if the playback effect of the current client remains stable, then a resource backup with a larger bit rate can be appropriately selected for transmission to further improve the playback effect of the client as much as possible; if the playback effect of the current client deteriorates, then a resource backup with a smaller bit rate can be appropriately selected for transmission to improve the stability of resource transmission and improve the playback effect of the client.

[0132] In this step, after determining the target resource backup from the first resource set, target multimedia resource data can be generated based on the target resource backup. Here, in some embodiments, the target resource backup can be directly determined as the target multimedia resource data; in some embodiments, the target resource backup can also be optimized to obtain more suitable target multimedia resource data. This application does not limit the specific optimization method.

[0133] Step 350: Send the target multimedia resource data to the client.

[0134] In this step, after obtaining the target multimedia resource data, it can be sent to the client. After receiving the target multimedia resource data, the client can use the relevant playback component to play it, thereby continuously feeding back the experience quality information to the server, realizing cyclic resource transmission control.

[0135] It can be understood that the multimedia resource transmission method provided in the embodiment of the present application detects the network quality information between the server and the client, and receives the first experience quality information fed back by the client at the current time node. The network quality information can reflect the network situation between the server and the client, and the first experience quality information can reflect the actual service situation of the client playing the multimedia resource; according to the network quality information, a first resource set can be determined from a plurality of resource backups with different bit rates corresponding to the multimedia resource, and then according to the change of the first experience quality information relative to the experience quality information fed back by the client before the current time node, the target resource backup can be determined from the first resource set, and then the target multimedia resource data sent to the client is generated. The method in the embodiment of the present application can comprehensively consider the network quality and the client's service experience, and coordinately adjust the transmission strategy of the multimedia resource, which is conducive to improving the response efficiency of the multimedia application server and optimizing the playback effect of the client.

[0136] Specifically, in a possible implementation, the network quality information includes available network bandwidth; and determining, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource includes:

[0137] Sort each resource backup according to its bit rate;

[0138] Compare the available network bandwidth and the sizes of various bit rates, and determine a predetermined number of consecutively ordered resource backups as a first resource set based on the comparison result; wherein the first resource set includes resource backups whose corresponding bit rates are less than the available network bandwidth, and the first resource set includes resource backups whose corresponding bit rates are greater than the available network bandwidth.

[0139] In the embodiments of the present application, as described above, the network quality information may include indicator data of the network's available bandwidth, which can effectively reflect the bitrate of multimedia resources suitable for transmission on the current network. Therefore, when determining a first resource set from multiple resource backups corresponding to multimedia resources based on the network quality information, the resource backups included in the first resource set may be determined based on the correspondence between the network's available bandwidth and the bitrates of the resource backups.

[0140] Specifically, in an embodiment of the present application, the resource backups corresponding to the multimedia resources can be sorted according to their bitrates. For example, the resource backups can be sorted in ascending order of their corresponding bitrates, or in descending order of their corresponding bitrates. Then, the available network bandwidth and the bitrates can be compared, and based on the comparison result, a predetermined number of consecutively sorted resource backups can be determined as a first resource set.

[0141] For example, let's say there are currently 6 resource backups, with corresponding bit rates of 1.5Mbps, 5Mbps, 2.4Mbps, 3.6Mbps, 8Mbps, and 1Mbps, respectively. The network available bandwidth is 2.7Mbps, and the predetermined number is 3. Then, these resource backups are sorted according to the relationship between the bit rates. Here, the corresponding bit rates are used to characterize each resource backup. Taking sorting from small to large as an example, the sorting results are 1Mbps, 1.5Mbps, 2.4Mbps, 3.6Mbps, 5Mbps, and 8Mbps. Then, the available network bandwidth of 2.7Mbps can be compared with the size of each bit rate, and it can be found that it is between 2.4Mbps and 3.6Mbps. It can be understood that since 2.4Mbps and 3.6Mbps are relatively close to the available network bandwidth of 2.7Mbps, using resource backups corresponding to these two bit rates makes it easier to achieve stable and reliable transmission, and has a higher network transmission resource utilization rate. Therefore, the two can be added to the first resource set. Compared to 1Mbps and 8Mbps, which are not sequentially ranked with 2.4Mbps or 3.6Mbps, 1.5Mbps, which is sequentially ranked with 2.4Mbps, and 5Mbps, which is sequentially ranked with 3.6Mbps, are closer to the available network bandwidth of 2.7Mbps. Therefore, it is more reasonable to add 1.5Mbps or 5Mbps to the first resource set. The resulting first resource set can be a sequentially ranked resource backup corresponding to 1.5Mbps, 2.4Mbps, and 3.6Mbps, or a sequentially ranked resource backup corresponding to 2.4Mbps, 3.6Mbps, and 5Mbps.

[0142] It is understandable that in the embodiment of the present application, when determining the first resource set, a predetermined number of resource backups that are sequentially ordered can be selected, and these resource backups are constrained to include resource backups with a bit rate less than the available network bandwidth, as well as resource backups with a bit rate greater than the available network bandwidth. In this way, it can be ensured that the first resource set includes resource backups that are closest to the available network bandwidth, which is beneficial to improving the transmission stability of multimedia resources. Furthermore, in the embodiment of the present application, since the first resource set includes both resource backups corresponding to bit rates less than the available network bandwidth and resource backups corresponding to bit rates greater than the available network bandwidth, it can facilitate the flexibility of subsequent determination of target resource backups, thereby adapting to complex client hardware environments.

[0143] In the embodiments of the present application, it is understood that, when determining the first resource set, in order to achieve a more balanced bitrate selection, the bitrate corresponding to the resource backups in the first resource set with the available network bandwidth can be positioned as close to the middle as possible. The following describes a method for determining the first resource set to achieve this effect.

[0144] Specifically, in a possible implementation, the available network bandwidth and the sizes of various code rates are compared, and a predetermined number of sequentially ordered resource backups are determined as a first resource set according to the comparison result, including:

[0145] Record the sorting numbers corresponding to each resource backup in sorted order;

[0146] Comparing the available network bandwidth with the various bit rates, and determining a first bit rate from the bit rates backed up by the various resources; wherein the first bit rate is the bit rate that is closest to the available network bandwidth among the bit rates that are less than or equal to the available network bandwidth;

[0147] Querying a first sorting number corresponding to a first resource backup; wherein the first resource backup is a resource backup corresponding to a first bit rate;

[0148] Dividing the difference obtained by subtracting 1 from the predetermined number by 2, and rounding the quotient to obtain a first value;

[0149] Determine a second sort number based on a difference between the first sort number and the first value;

[0150] Starting from the second sorting number, a predetermined number of resource backups are selected in sequence in a sorting direction corresponding to increasing bit rates to obtain a first resource set.

[0151] In an embodiment of the present application, for each resource backup, after sorting according to the corresponding bit rate, the sorting number of each resource backup can be recorded in the sorting order. The sorting number can be increased in ascending order. For example, it can start from 1 (or any number such as 0), and the resource backup with the first bit rate sorting is numbered 1, the next resource backup is numbered 2, and so on, until the number corresponding to the resource backup with the last bit rate sorting is determined. Then, the available network bandwidth and the size of each bit rate can be compared, and the bit rate that is less than or equal to the available network bandwidth and closest to the available network bandwidth can be determined from the bit rates of the resource backups. In an embodiment of the present application, it is recorded as the first bit rate. For example, for the bit rates corresponding to the resource backups introduced above are 1Mbps, 1.5Mbps, 2.4Mbps, 3.6Mbps, 5Mbps, and 8Mbps, if the available network bandwidth is 2.7Mbps, then the first bit rate can be determined to be 2.4Mbps. Then, the sorting number corresponding to the first resource backup can be queried and recorded as the first sorting number. Assuming that the bit rates of 1 Mbps, 1.5 Mbps, 2.4 Mbps, 3.6 Mbps, 5 Mbps, and 8 Mbps are numbered 1 to 6, the determined first sorting number is 3.

[0152] In an embodiment of the present application, the code rate corresponding to the first sorting number is less than or equal to the network available bandwidth, and is the code rate closest to the network available bandwidth. If the number of code rates less than the network available bandwidth and the number of code rates greater than the network available bandwidth in the determined first resource set are to be as equal as possible, the difference obtained after subtracting 1 can be divided by 2 according to the set predetermined number, and then the quotient is rounded to an integer, and the resulting value is recorded as the first value. From the position where the first sorting number is located, the resource backup of the first value is taken in the sorting direction of the corresponding code rate decrease, and then from the position where the first sorting number is located, the resource backup of the predetermined number minus the first value is taken in the sorting direction of the corresponding code rate increase, and the first resource set can be obtained. Specifically, in an embodiment of the present application, the difference between the first sorting number and the first value can be calculated, and the difference can be determined as the second sorting number. Then, starting from the second sorting number, a predetermined number of resource backups can be selected in the sorting direction of the corresponding code rate increase to obtain the first resource set.

[0153] For example, assume that there are 20 resource backups sorted from smallest to largest according to their corresponding bitrates, with the corresponding numbers being 1 to 20, and the predetermined number being 6. The available network bandwidth is greater than the bitrate corresponding to the resource backup numbered 7, but less than the bitrate corresponding to the resource backup numbered 8. Ideally, it is desirable to select resource backups numbered 5 to 10 as the first resource set. According to the method provided in the embodiments of the present application, the first sorting number is determined to be 7, and the first value is 2, so the second sorting number obtained is 5. Starting from the resource backup numbered 5, 5 resource backups are selected in sequence in the sorting direction of increasing corresponding bitrates, and resource backups numbered 5 to 10 can be obtained as the first resource set.

[0154] It can be understood that in the embodiment of the present application, determining the first resource set in the above-mentioned manner can make the bit rate distribution of the resource backup within the first resource set as balanced as possible, and subsequently can more efficiently determine the applicable target resource backup, which is beneficial to improving the response efficiency of the multimedia application server and the stability of multimedia resource transmission.

[0155] Specifically, in a possible implementation, the first experience quality information includes first jamming indicator data; and determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information includes:

[0156] Determine whether the client's lag situation has worsened based on the first lag indicator data and historical experience quality information;

[0157] If the client's freezing condition worsens, a target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

[0158] In the embodiment of the present application, as described above, the experience quality information may include jamming indicator data. In the embodiment of the present application, the jamming indicator data in the first experience quality information is recorded as the first jamming indicator data. When determining the change in the first experience quality information relative to the historical experience quality information, it can be determined based on the first jamming indicator data and the historical experience quality information whether the client's jamming situation has deteriorated.

[0159] Specifically, in the embodiments of the present application, the worsening of the client's jamming situation can be divided into two situations. For example, in some embodiments, there may not have been jamming before, and the first jamming indicator data received at the current time node reflects that the client has jammed. In this case, it can be considered that the client's jamming situation has worsened; in some embodiments, there may have been jamming before, and the first jamming indicator data received at the current time node reflects that the client still has jamming, and the relevant indicator data indicates that the jamming situation is more serious, such as an increase in the number of jams or a longer duration of jams, etc., in which case it can also be considered that the client's jamming situation has worsened.

[0160] In contrast, if the first lag indicator data received at the current time node indicates that the client is not experiencing lag, such as the number of lags is 0 or the duration of the lag is 0, then it indicates that the client is not experiencing lag, and it can be considered that the client's lag situation has not worsened. Alternatively, in some embodiments, if the first lag indicator data indicates that the client's lag situation has improved, such as the number of lags has decreased or the duration of the lag has shortened, it can also be considered that the client's lag situation has not worsened.

[0161] In an embodiment of the present application, if the client's lag situation worsens, it indicates that the current communication environment between the server and the client is poor, or the client's hardware environment is poor. In order to minimize the probability of lag and optimize the client's playback effect, a resource backup with a corresponding bit rate less than the available network bandwidth can be selected from the first resource set as the target resource backup. In this way, it can better adapt to the current multimedia resource transmission and playback environment, increase the probability of stable transmission of target multimedia resource data to the client, and reduce the probability of subsequent lag on the client.

[0162] Specifically, in one possible implementation, determining whether the client's jamming condition has deteriorated based on the first jamming indicator data and historical experience quality information includes:

[0163] Determining second experience quality information from historical experience quality information; the second experience quality information is experience quality information fed back by the client at a time node previous to the current time node, and the second experience quality information includes second jamming indicator data;

[0164] Calculating a first difference value between the first jamming indicator data and the second jamming indicator data;

[0165] comparing the first difference value with a first threshold;

[0166] If the first difference value is less than or equal to the first threshold, it is determined that the client's jamming situation has not worsened; or, if the first difference value is greater than the first threshold, it is determined that the client's jamming situation has worsened.

[0167] In an embodiment of the present application, when determining whether the jamming situation of the client has deteriorated based on the first jamming indicator data and the historical experience quality information, in some embodiments, the experience quality information of the previous time node can be used to make a separate judgment. Specifically, in an embodiment of the present application, the experience quality information corresponding to the previous time node of the current time node is recorded as the second experience quality information, and the jamming indicator data included in the second experience quality information is recorded as the second jamming indicator data. It can be understood that the second jamming indicator data can characterize the jamming situation reported by the client at the previous time node. If the relevant value of the first jamming indicator data exceeds the second jamming indicator data by a large amount, it can be considered that the jamming situation of the client has deteriorated; if the relevant value of the first jamming indicator data and the second jamming indicator data are basically close, or the relevant value of the first jamming indicator data is less than the second jamming indicator data, it can be considered that the jamming situation of the client has not deteriorated.

[0168] In an embodiment of the present application, the difference value of the first jamming indicator data relative to the second jamming indicator data can be calculated and recorded as the first difference value. The specific calculation method of the difference value here can be flexibly set as needed. For example, in some embodiments, the difference between the first jamming indicator data and the second jamming indicator data can be calculated, and then the difference is determined as the first difference value; in some embodiments, the ratio of the difference to the first jamming indicator data (or the second jamming indicator data) can also be calculated, and the ratio is determined as the first difference value.

[0169] It can be understood that in the embodiment of the present application, if the first jamming indicator data is less than or equal to the second jamming indicator data, it can be considered that the jamming situation of the client has not worsened. If the first jamming indicator data is greater than the second jamming indicator data, and the difference between the two is large or the difference accounts for a large proportion of the first jamming indicator data (or the second jamming indicator data), it can be considered that the jamming situation of the client has worsened. Therefore, in the embodiment of the present application, a threshold can be set, recorded as the first threshold, the first threshold is a positive number, and the present application does not limit its size. If the calculated first difference value is a negative number, it can be determined that the jamming situation of the client has not worsened; if the calculated first difference value is a positive number and the value is small, and is within the first threshold, it can be determined that the relevant value of the first jamming indicator data and the second jamming indicator data are basically close, and the jamming situation of the client has not worsened; if the calculated first difference value is large and greater than the first threshold, it can be considered that the jamming situation of the client has worsened.

[0170] It can be understood that in the embodiment of the present application, based on the comparison between the second jamming indicator data and the first jamming indicator data, it is possible to quickly determine whether the jamming situation of the client has worsened, which is beneficial to improving the transmission efficiency of multimedia resources.

[0171] Specifically, in one possible implementation, determining whether the client's jamming condition has deteriorated based on the first jamming indicator data and historical experience quality information includes:

[0172] Determine, based on historical quality of experience information, a first cumulative value corresponding to the jamming indicator data before the current time node;

[0173] Performing a weighted summation on the first jamming indicator data and the first cumulative value to obtain a second cumulative value;

[0174] calculating a second difference value of the second cumulative value relative to the first cumulative value;

[0175] Comparing the second difference value with the second threshold;

[0176] If the second difference value is less than or equal to the second threshold, it is determined that the client's jamming situation has not worsened; or, if the second difference value is greater than the second threshold, it is determined that the client's jamming situation has worsened.

[0177] In an embodiment of the present application, when determining whether the jamming situation of the client has deteriorated based on the first jamming indicator data and historical experience quality information, in some embodiments, the experience quality information of the historical time period can be used to make a comprehensive judgment. Specifically, in an embodiment of the present application, the cumulative value corresponding to the jamming indicator data before the current time node can be determined based on the historical experience quality information, and recorded as the first cumulative value. Then, the first jamming indicator data and the first cumulative value can be weighted and summed to obtain a new cumulative value, which is recorded as the second cumulative value. In an embodiment of the present application, there is no restriction on the weights corresponding to the first jamming indicator data and the first cumulative value when performing the weighted summation, and they can be flexibly set according to actual needs.

[0178] When the second cumulative value is obtained, the difference between the second cumulative value and the first cumulative value can be calculated. In the embodiment of the present application, it is recorded as the second difference value. The specific meaning and calculation method of the second difference value can be implemented with reference to the aforementioned first difference value, and the present application will not elaborate on it here. It can be understood that if the relevant value of the first jamming indicator data is large, that is, when the jamming situation of the client worsens, the second cumulative value obtained will have a large increase relative to the first cumulative value, so that the value of the second difference value will also be larger; relatively, if the relevant value of the first jamming indicator data is small, that is, when the jamming situation of the client does not worsen, the second cumulative value obtained will not increase or only increase slightly relative to the first cumulative value, and the value of the second difference value will also be smaller. Therefore, in the embodiment of the present application, a threshold value can be set, recorded as the second threshold value, and the second threshold value is a positive number. Similarly, the present application does not limit its size. If the calculated second difference value is a negative number, it can be determined that the client's lag situation has not worsened; if the calculated second difference value is a positive number and the value is small, and is within the second threshold, it can be considered that the client's lag situation has not worsened; if the calculated first difference value is large and greater than the second threshold, it can be considered that the client's lag situation has worsened.

[0179] It should be noted that in the embodiments of the present application, when determining the first cumulative value, it can also be gradually updated in a weighted manner. For example, the first cumulative value can be obtained by weighting the second jamming indicator data and the cumulative value obtained at the previous time node of the current time node, and this application does not limit this.

[0180] It can be understood that in the embodiment of the present application, making a comprehensive judgment based on the experience quality information of the historical time period can reduce the interference caused by temporary performance fluctuations, can more accurately reflect the client's lag situation, and facilitate the adaptive transmission of multimedia resources.

[0181] Specifically, in one possible implementation, if the client's lag worsens, determining a target resource backup from resource backups in the first resource set whose corresponding bitrate is less than the available network bandwidth includes:

[0182] If the jamming condition of the client worsens, determining the jamming deterioration level corresponding to the client according to the first difference value and the first threshold;

[0183] According to the jamming deterioration level, the target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

[0184] In an embodiment of the present application, if it is determined that the jamming situation of the client has deteriorated, when determining the target resource backup from the resource backup in the first resource set, the target resource backup can be determined in combination with the severity of the deterioration. Specifically, in an embodiment of the present application, the jamming deterioration level corresponding to the client can be determined based on the first difference value and the first threshold. For example, when the first difference value exceeds the first threshold, after determining that the jamming situation of the client has deteriorated, the multiple between the first difference value and the first threshold can be further calculated. If the multiple between the first difference value and the first threshold is larger, it means that the jamming situation of the client has deteriorated more seriously, that is, the jamming deterioration level corresponding to the client is higher; relatively, if the multiple between the first difference value and the first threshold is smaller, it means that the jamming situation of the client has deteriorated less seriously, that is, the jamming deterioration level corresponding to the client is lower.

[0185] In an embodiment of the present application, after determining the jam deterioration level, the target resource backup can be determined according to the jam deterioration level. Specifically, it can be understood that if the jam deterioration level of the client is high, it means that the transmission and playback of the multimedia resources are relatively unstable. In order to improve the jam situation of the client as much as possible, the transmission bit rate can be reduced, that is, the resource backup with a bit rate less than the available network bandwidth and a relatively small size in the first resource set can be selected as the target resource backup. On the contrary, if the jam deterioration level of the client is low, it means that the transmission and playback of the multimedia resources are relatively stable. In order to improve the playback effect of the client as much as possible, a higher bit rate transmission can be used, that is, the resource backup with a bit rate less than the available network bandwidth but a relatively large size in the first resource set can be selected as the target resource backup.

[0186] Of course, it should be noted that in the embodiment of the present application, the corresponding lag deterioration level of the client can also be determined based on the second difference value and the second threshold, so as to adaptively determine the target resource backup from the first resource set. This application will not go into details here.

[0187] Specifically, in one possible implementation, the first experience quality information includes first jamming indicator data and first cache indicator data; and determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information includes:

[0188] Determine whether the client's lag situation has worsened based on the first lag indicator data and historical experience quality information;

[0189] Determining whether a cache condition of the client has deteriorated based on the first cache indicator data and historical experience quality information;

[0190] If the client's lag situation does not worsen and the client's cache situation worsens, the second resource backup in the first resource set is determined to be the target resource backup; wherein, the second resource backup is a resource backup corresponding to the second bit rate, and the second bit rate is greater than or equal to the bit rate used by the client at the previous time node of the current time node.

[0191] In the embodiment of the present application, as described above, the experience quality information may also include cache indicator data. Specifically, the cache indicator data in the first experience quality information is recorded as the first cache indicator data. In the embodiment of the present application, it is also possible to determine whether the client's cache situation has deteriorated based on the first cache indicator data and historical experience quality information. The specific determination method can be implemented with reference to the jam indicator data, and this application will not elaborate on this.

[0192] In an embodiment of the present application, if it is determined that the client's jamming situation has not deteriorated, but the client's cache situation has deteriorated, it means that the communication environment between the server and the client was good in the previous stage, and the client's hardware performance has affected the playback effect to a certain extent. In an embodiment of the present application, you can first query the bit rate used at the previous time node of the current time node, and record it as the third bit rate. If the current communication environment between the server and the client remains good, you can appropriately increase the transmission bit rate, that is, you can determine a bit rate greater than or equal to the third bit rate in the first resource set, record it as the second bit rate, and record the resource backup corresponding to the second bit rate as the second resource backup, and then you can determine the second resource backup as the target resource backup.

[0193] It should be noted that in an embodiment of the present application, if the current communication environment between the server and the client deteriorates and there is no second bit rate greater than or equal to the third bit rate in the first resource set, the server can first maintain the use of the third bit rate resource backup for transmission to avoid the situation where the playback effect deteriorates due to changing the bit rate.

[0194] Specifically, in a possible implementation, generating target multimedia resource data based on the target resource backup includes:

[0195] If the client's freezing situation does not worsen and the client's cache situation worsens, the target resource backup is subjected to frame loss processing to generate target multimedia resource data.

[0196] In the embodiment of the present application, as described above, if the client's lag situation does not deteriorate but the cache situation deteriorates, the server can appropriately increase the transmission bit rate of the multimedia resource when the communication environment is good, and maintain the previously used transmission bit rate when the communication environment deteriorates. In order to alleviate the cache deterioration problem on the client side, in the embodiment of the present application, frame loss processing can be performed on the target resource backup to obtain the target multimedia resource data, thereby enabling the client to achieve normal and stable playback of the multimedia resource as much as possible.

[0197] Specifically, in a possible implementation, performing frame loss processing on a target resource backup to generate target multimedia resource data includes:

[0198] Detecting the frame type of each resource frame in the target resource backup;

[0199] Determine the target frame to be discarded in the target resource backup according to the frame type;

[0200] Detecting the playback time interval between two resource frames adjacent to the target frame;

[0201] If the playback time interval is less than or equal to the third threshold, the target frame is discarded; or if the playback time interval is greater than the third threshold, the target frame is retained.

[0202] In an embodiment of the present application, when performing frame loss processing on a target resource backup, in some embodiments, the frame type of each resource frame in the target resource backup can be first detected. For example, taking video frames as an example, video frames include frames that are required for encoding and decoding, and also include frames that are not relied upon by other frames during encoding and decoding. The second type of video frames can be considered as a type that can be discarded. Therefore, the frames to be discarded in the target resource backup can be determined based on the frame type of the resource frame. In an embodiment of the present application, these frames are recorded as target frames.

[0203] After determining the target frame, in an embodiment of the present application, in order to reduce the performance impact on the client playback, these target frames are not discarded directly, but the playback time interval of the two resource frames adjacent to the discarded target frame is constrained to be within a certain range, so as to make the picture smooth as possible. Specifically, in an embodiment of the present application, the playback time interval of two resource frames adjacent to each target frame can be detected. It should be noted that, in an embodiment of the present application, the two resource frames adjacent to the target frame refer to a resource frame before the target frame and a resource frame after the target frame. If the playback time interval of the two resource frames is short, it means that the smoothness of the playback is less affected by discarding the target frame between them, and the target frame can be discarded; relatively, if the playback time interval of the two resource frames is long, it means that the smoothness of the playback is greatly affected by discarding the target frame between them, and the target frame can be retained at this time.

[0204] In the embodiment of the present application, a time threshold may be set, denoted as a third threshold. If the playback time interval between two adjacent resource frames of a target frame is less than or equal to the third threshold, the target frame may be discarded; if the playback time interval between two adjacent resource frames of the target frame is greater than the third threshold, the target frame may be retained.

[0205] Specifically, in a possible implementation, performing frame loss processing on a target resource backup to generate target multimedia resource data includes:

[0206] Detecting the frame type of each resource frame in the target resource backup;

[0207] Determine the target frame to be discarded in the target resource backup according to the frame type;

[0208] Determine the first consecutive target frames as a frame set;

[0209] A target frame in the frame set is discarded, and other target frames in the frame set except the discarded target frame are retained.

[0210] In an embodiment of the present application, when performing frame loss processing on the target resource backup, in some embodiments, after determining the target frame to be discarded, there may be multiple target frames that are continuous. If all these target frames are discarded, it is easy to cause screen fragmentation or freeze problems. Therefore, in an embodiment of the present application, the continuity of the target frames can be detected, a numerical value is pre-set, recorded as the first number, and the target frames of the first continuous number are determined as a frame set, where continuity refers to the proximity of each target frame. For the entire frame set, one target frame is discarded, and the other target frames except the discarded target frame are retained. In this way, the probability of screen fragmentation or freeze caused by frame loss processing can be reduced, which is beneficial to improving the playback effect of the client.

[0211] In the embodiment of the present application, when dividing frame sets, target frames already in a frame set are no longer included in the division of other frame sets. For example, if the first number is set to 5 and a certain segment of data in the target resource backup contains 20 target frames, then these 20 target frames can be divided into 4 frame sets, with 1 target frame discarded from each frame set, leaving 4 target frames. After the 20 target frames are processed, 16 target frames remain.

[0212] It should be noted that in the embodiment of the present application, the two aforementioned strategies for frame loss processing can be combined for use. For example, the target frames that need to be discarded are first determined based on the playback time interval between two adjacent resource frames of each target frame. Then, based on whether these target frames that need to be discarded are continuous, they are screened, and some of the target frames that need to be discarded are updated to target frames that need to be retained, and finally the discard processing operation is performed.

[0213] Specifically, in one possible implementation, the first experience quality information includes first jamming indicator data and first cache indicator data; and determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information includes:

[0214] Determine whether the client's lag situation has worsened based on the first lag indicator data and historical experience quality information;

[0215] Determining whether a cache condition of the client has deteriorated based on the first cache indicator data and historical experience quality information;

[0216] If the client's freezing situation has not worsened and the client's cache situation has not worsened, the target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are greater than the available network bandwidth.

[0217] In this embodiment of the present application, if it is currently determined that the client's lag and cache conditions have not deteriorated, a higher bitrate can be selected for multimedia resource transmission. That is, within the first resource set, a resource backup with a corresponding bitrate greater than the available network bandwidth can be determined as the target resource backup. This improves the client's playback performance and better utilizes network communication resources and client hardware resources.

[0218] Below, in conjunction with the specific application implementation process, the transmission method of multimedia resources provided in this application is introduced and explained in detail.

[0219] Reference Figure 4 , Figure 4 FIG. 1 shows another transmission diagram of multimedia resources provided in an embodiment of the present application. Figure 4 In this paper, taking the multimedia resource being audio and video data as an example, a server structure using the transmission method of multimedia resources provided in the embodiments of this application is introduced. Specifically, in the embodiments of this application, the server is enabled to detect network quality information and perceive client experience quality information. Through the collaboration of the transport layer and the application layer, the corresponding encoding and decoding strategy can be quickly adjusted, overcoming the problems of limited encoding strategies and inability to respond quickly in traditional transmission methods, and can effectively optimize the client's playback performance and service experience.

[0220] Reference Figure 4 In the embodiment of the present application, the server includes an application layer and a transport layer. The application layer can store multiple resource backups corresponding to multimedia resources, each with a different bit rate. The transport layer can include a control module, a sending module, a receiving module, and a monitoring module. The sending module and the receiving module can be connected to the receiving end (i.e., the client) via the Internet.

[0221] In the embodiment of the present application, during the transmission of multimedia resources, the following steps are mainly performed:

[0222] ① After receiving the audio and video traffic, the client replies with a message confirmation message to the server, and periodically feeds back to the server the cache indicator data of its own player (such as cache duration, number of cached audio and video frames, cached data volume, etc.) or jam indicator data (such as the number of jams and jam duration in the recent period, etc.); ② After receiving the above information, the server's receiving module passes the information to the monitoring module. The monitoring module evaluates the network quality information based on the above information and extracts the client's cache indicator data or jam indicator data; ③ The monitoring module passes the above-evaluated network quality information and the client's experience quality information (including cache indicator data or jam indicator data) to the server's control module. The control module Based on the above information, the control instructions sent to the application layer are decided; ④ the control module sends the above control instructions to the application layer, and the application layer selects appropriate audio and video data with a certain clarity, bit rate and frame rate according to the control instructions; ⑤ the application layer passes the selected audio and video data (i.e., target resource backup) to the control module of the transport layer; ⑥ the control module performs relevant processing on the received audio and video data, and passes the processed audio and video data (i.e., target multimedia resource data) to the sending module; ⑦ the sending module sends the above audio and video data in the form of a message; ⑧ the client plays the above audio and video data after receiving it, and then periodically obtains the experience quality information of the current player, and feeds back the experience quality information to the server.

[0223] Specifically, the above process is introduced and explained in detail below.

[0224] Reference Figure 5 , Figure 5 A schematic diagram of a specific process of multimedia resource transmission provided in an embodiment of the present application is shown.

[0225] In the embodiment of the present application, after receiving the audio and video data from the server, the client replies to the server with a message confirmation message (Pkt_ack message) and a message containing the client's quality of experience information QoE (Pkt_QoE_ack message):

[0226] In the embodiment of the present application, the client's quality of experience information QoE includes but is not limited to the following indicators: the freeze indicator data (Player_freezing) encountered by the client player in the past period of time, specifically including the number of freezes (freezing_times) and the freeze duration (freezing_duration); the cache indicator data (Player_buffering) of the current client player, specifically including the cached audio and video frame length (buffer_time_len), the number of frames (buffer_frame_amount), the cached data size (buffer_frame_size), etc.

[0227] In the embodiment of the present application, in order to obtain the quality of experience information QoE of the client, a QoE acquisition module and a QoE feedback module can be configured on the client. Figure 6 , Figure 6 A schematic diagram of the acquisition and transmission of quality of experience information provided in an embodiment of the present application is shown. In the embodiment of the present application, after receiving the audio and video data from the server, the client passes it to the player cache of the application layer through the transport layer; the player obtains the audio and video frames from its own cache, and displays these audio and video frames in the player screen and sound. In the embodiment of the present application, the QoE acquisition module of the client periodically obtains the indicator data of the quality of experience information QoE from the player cache and the player. Specifically, the time interval for the QoE acquisition module of the client to obtain the quality of experience information QoE can be recorded as T_QoE. The size of T_QoE can be configured by the administrator. For example, T_QoE = 100ms.

[0228] In the embodiment of the present application, after the QoE acquisition module of the client periodically obtains the jamming indicator data and the cache indicator data, it can pass this information to its own QoE feedback module; after obtaining the jamming indicator data and the cache indicator data passed from the QoE acquisition module, the QoE feedback module constructs a QoE feedback message Pkt_QoE_ack containing the above information and returns the message to the server; Here, the content carried by the message Pkt_QoE_ack can be expressed by formula (1):

[0229] Pkt_QoE_ack={Player_freezing, Player_bufferin} (1)

[0230] In the embodiment of the present application, after receiving the reply message confirmation message (Pkt_ack message) and the Pkt_QoE_ack message containing the quality of experience information QoE from the client, the receiving module of the server transmits these messages to its own monitoring module; after receiving the ACK message or Pkt_QoE_ack message sent by the receiving module, the monitoring module detects the real-time network quality information QoS and extracts the quality of experience information QoE of the client, as follows:

[0231] In this embodiment of the present application, the server's monitoring module can detect (or measure) the following network quality information: network available bandwidth BtlBW; round-trip delay RTT; network packet loss rate Loss_rate. The monitoring module periodically transmits a Pkt_QoS message to the control module, wherein the message contains the above-detected network quality information, as shown in formula (2):

[0232] Pkt_QoS={BtlBW, RTT, Loss_rate} (2)

[0233] In the embodiment of the present application, after receiving the Pkt_QoE_ack message transmitted from the receiving module, the monitoring module of the server extracts the freeze indicator data Player_freezing and the buffer indicator data Player_buffering of the client player carried in the message from the message, and transmits the above information to the control module of the server via the message Pkt_QoE; the format of the Pkt_QoE message transmitted to the control module is shown in formula (3):

[0234] Pkt_QoE={Play_freezing, Player_buffering} (3)

[0235] In the embodiment of the present application, the control module of the server maintains a table for each connected client's network quality information QoS and experience quality information QoS. Upon receiving the Pkt_QoS message and Pkt_QoE message from the monitoring module, the server updates the above table and decides the control instructions to be sent to the application layer based on the parameter information in the table, as follows:

[0236] Refer to Table 1, which shows the maintenance table Table_QoS corresponding to the network quality information QoS provided in an embodiment of the present application.

[0237] Table 1

[0238] index Previous accumulated value Latest value Latest cumulative value BtW Cum_bw_last BW_new Cum_bw_new RTT Cum_rtt_last RTT_new Cum_rtt_new Loss_rate Cum_loss_last Loss_rate_new Cum_loss_new

[0239] In the embodiment of the present application, when the control module receives the Pkt_QoS message from the monitoring module, it extracts the latest network available bandwidth BtlBW, round-trip delay RTT, network packet loss rate Loss_rate and other information carried therein, and updates the "latest value" (denoted as value_new) of these three indicators in the above Table 1; the corresponding latest cumulative value in Table_QoS (denoted as Cum_new) is calculated by formula (4):

[0240] Cum_new=alpha*Cum_last+(1-alpha)*value_new (4)

[0241] Cum_last refers to the cumulative value of the metric before the latest value (i.e., BW_new, RTT_new, and Loss_rate_new) is obtained. The parameter alpha is the weight used to calculate the latest cumulative value, 0 ≤ alpha ≤ 1. This parameter is configurable by the administrator; by default, alpha = 0.8. In Table 1, the previous cumulative values of the three metrics, network available bandwidth, round-trip delay, and network packet loss rate, are represented by Cum_bw_last, Cum_rtt_last, and Cum_loss_last, respectively; the latest values of these three metrics are represented by BW_new, RTT_new, and Loss_rate_new, respectively; and the latest cumulative values of these three metrics are represented by Cum_bw_new, Cum_rtt_new, and Cum_loss_new, respectively.

[0242] Refer to Table 2, which shows the maintenance table Table_QoE corresponding to the quality of experience information QoS provided in an embodiment of the present application.

[0243] Table 2

[0244] index Previous accumulated value Latest value Latest cumulative value freezing_times Cum_ft_last Ft_new Cum_ft_new freezing_duration Cum_fd_last Fd_new Cum_fd_new buffer_time_len Cum_btl_last Bt_new Cum_bt_new buffer_frame_amount Cum_bfa_last Bfa_new Cum_bfa_new buffer_frame_size Cum_bfs_last Bfs_new Cum_bfs_new

[0245] In the embodiment of the present application, when the control module receives the Pkt_QoE message from the monitoring module, it extracts the latest jamming indicator data (i.e., the first jamming indicator data) and cache indicator data (i.e., the first cache indicator data) carried by the message, and updates the "latest value" of these indicators in the above Table 2 (recorded as value_qoe_new); the latest cumulative value Cum_qoe_new of the maintenance table Table_QoE is obtained by calculating the value of formula (5):

[0246] Cum_qoe_new=beta*Cum_qoe_last+(1-beta)*value_qoe_new (5)

[0247] Among them, Cum_qoe_last refers to the cumulative value before the latest indicator value is obtained (i.e., the first cumulative value); the parameter beta is the weight value for calculating the latest cumulative value, 0≤beta≤1, and this parameter can be configured by the administrator. By default, beta=0.8; in Table 2, the "previous cumulative values" of the number of freezes, freeze duration, audio and video frame duration, number of frames, and cached data size are represented by Cum_ft_last, Cum_fd_last, Cum_btl_last, Cum_bfa_last, and Cum_bfs_last, respectively; the latest values of these five indicators are represented by Ft_new, Fd_new, Bt_new, Bfa_new, and Bfs_new, respectively; and the latest cumulative values of these five indicators (i.e., the second cumulative values) are represented by Cum_ft_new, Cum_fd_new, Cum_bt_new, Cum_bfa_new, and Cum_bfs_new, respectively.

[0248] In the embodiment of the present application, the control module of the server decides the control instructions to be sent to the application layer based on the latest Table_QoS table and Table_QoE table, as follows:

[0249] The control instruction ins_ctl sent by the control module to the application layer is mainly to select the resource backup of audio and video data with a certain bit rate (frame rate, clarity); in the embodiment of the present application, the resource backup of the same audio and video data resource (uniquely identified by URI) is represented by its identifier BKP_ID; for example, when there are n resource backups, BKP_0 is the audio and video data resource backup with the lowest clarity, the lowest bit rate, and the lowest frame rate of the audio and video resource; BKP_n is the audio and video data resource backup with the highest clarity, the highest bit rate, and the highest frame rate of the audio and video resource.

[0250] At this time, the ins_ctl message (denoted as Pkt_ins_ctl message) sent by the control module can be expressed by the following formula (6):

[0251] Pkt_ins_ctl={conn_id, URI, BKP_ID} (6)

[0252] Among them, conn_id represents the identifier of the connection: in the TCP protocol, it can be uniquely identified by a five-tuple (i.e., source address, source port number, destination address, destination port number, and protocol number); in the QUIC protocol, it can be represented by the connection identifier that comes with the protocol.

[0253] In this embodiment of the present application, the resource backup selected in the previous cycle is represented by BKP_s; the BKP_ID in the control instruction ins_ctl sent by the control module to the application layer is determined by the detected network quality information QoS and experience quality information QoS. Specifically, the control module first determines the candidate set BKP_Set (i.e., the first resource set) for the BKP_ID to be selected based on the network available bandwidth BtlBW in the Pkt_QoS message. The specific method is as follows:

[0254] The control module selects resource backup from the candidate set BKP_Set near the network available bandwidth BtlBW. For example, please refer to Figure 7 , Figure 7 FIG. 1 shows a schematic diagram of selecting an alternative set provided by an embodiment of the present application. Figure 7 As shown, the control module selects the BKP_ID closest to the detected network available bandwidth BtlBW based on the bit rate of the audio and video data. In the embodiment of the present application, the number m (i.e., the predetermined number) of resource backup BKPs in the BKP_Set can be configured in advance by the administrator. For example, m can be 4. Therefore, the candidate set BKP_Set can be represented by the following formula (7):

[0255] BKP_Set={BKP_j, BKP_j+1, BKP_j+2, …, BKP_j+m-1} (7)

[0256] Wherein, the code rate of BKP_{(2*j+m-1) / 2}≤BtlBW≤BKP_{(2*j+m-1) / 2+1}; In the embodiment of the present application,

[0257] The integer part of the result of (2*j+m-1) / 2 is taken.

[0258] In the embodiment of the present application, for ease of description, the BKP_IDs adjacent to the network available bandwidth BtlBW in the bitrate corresponding to the resource backup BKP are respectively recorded as BKP_p and BKP_q.

[0259] The control module makes a decision based on the jamming indicator data and cache indicator data in the message Pkt_QoE, and selects the target resource backup from the candidate set BKP_Set. The details are as follows:

[0260] If the jamming indicator data fed back by the client deteriorates, a resource backup BKP with a bitrate less than the available network bandwidth BtlBW is selected from the candidate set BKP_Set. Specifically, in the embodiment of the present application, the judgment basis for the deterioration of Play_freezing is any one of the following conditions:

[0261] i. The ratio (denoted as ratio_freezing_new) of the client's latest freeze count Ft_new (or freeze duration Fd_new) compared to the previous monitoring period exceeds the set threshold ratio_freezing_threshold. The threshold ratio_freezing_threshold is configured by the administrator. For example, ratio_freezing_threshold = 10%;

[0262] ii. The ratio (denoted as ratio_cum_freezing) of the latest accumulated number of freezes Cum_ft_new (or duration Cum_fd_new) of the client compared to the previous accumulated number Cum_ft_last (or accumulated duration Cum_fd_last) exceeds the threshold ratio_cum_freezing_threshold; wherein the threshold ratio_cum_freezing_threshold is configured by the administrator, illustratively, atio_cum_freezing_threshold = 10%;

[0263] In the embodiment of the present application, the above ratio_cum_freezing can be calculated by formula (8) (taking the cumulative number of freezes as an example):

[0264] Ratio_cum_freezing=(cum_fd_new–cum_fd_last) / cum_fd_last (8)

[0265] It should be noted that, in an embodiment of the present application, if ratio_freezing_new (or ratio_cum_freezing) exceeds 1 times (but less than 2 times) of ratio_freezing_threshold (or ratio_cum_freezing), the resource backup BKP (i.e., BKP_p) with the smallest difference between the bit rate and the network available bandwidth BtlBW is selected from the resource backup BKP whose bit rate is less than the network available bandwidth BtlBW; if ratio_freezing_new (or ratio_cum_freezing) exceeds 2 times (but less than 3 times) of ratio_freezing_threshold (or ratio_cum_freezing), the resource backup BKP (i.e., BKP_p-1) with the second smallest difference between the bit rate and the network available bandwidth BtlBW is selected from the resource backup BKP whose bit rate is less than the network available bandwidth BtlBW; other situations can be deduced by analogy and will not be elaborated here.

[0266] In an embodiment of the present application, if the cache indicator data fed back by the client deteriorates but the jamming indicator data does not deteriorate, a resource backup with a bit rate greater than the network available bandwidth BtlBW can be selected from the alternative set, or the previous BKP_s can be continued to be selected, and the corresponding differentiated frame loss operation can be performed.

[0267] In the embodiment of the present application, the basis for determining whether the cache indicator data is deteriorating is any one of the following conditions:

[0268] i. The ratio (denoted as ratio_buffering_new) by which the client's latest buffering duration bt_new (or the number of buffered frames bfa_new or the buffered frame size bfs_new) decreases compared to the previous monitoring period exceeds a preset threshold ratio_buffering_threshold; this threshold is configurable by the administrator, for example, ratio_buffering_threshold = 10%;

[0269] ii. The ratio (denoted as ratio_buffering_cum) by which the client's latest cumulative buffering duration cum_bt_new (or the cumulative number of buffered frames cum_bfa_new, or the cumulative buffered frame size cum_bfs_new) decreases compared to the previous monitoring period exceeds a preset threshold ratio_cum_buffering_threshold; this threshold is configurable by the administrator, for example, ratio_cum_buffering_threshold = 10%;

[0270] In an embodiment of the present application, if the cache indicator data fed back by the client has not deteriorated and the jamming indicator data has not deteriorated, a resource backup with a bit rate greater than the network available bandwidth BtlBW is selected from the BKP_Set set; the specific range of resource backups with a bit rate greater than the bit rate can be set according to actual needs; for example, the current server determines that the cache indicator data has not deteriorated and the jamming indicator data has not deteriorated, and the network available bandwidth BtlBW = 1Mbps, and in the alternative set BKP_Set, the bit rates of the resource backup BKP that exceeds the network available bandwidth BtlBW are 1.5Mbps, 2Mbps and 5Mbps respectively; if the constraint is that the bit rate should be selected to exceed the network available bandwidth BtlBW in the range of 0.5Mbps to 2Mbps, then the resource backup with the above-mentioned 2Mbps bit rate can be selected.

[0271] In the embodiment of the present application, the control module of the server can send the selected BKP_ID to the application layer in the form of a Pkt_ins_ctl message. The latter pulls the corresponding audio and video data according to the above BKP_ID and transmits it to the control module. After receiving the audio and video data from the application layer, the control module decides the original data information to be transmitted to the sending module based on whether the frame drop operation needs to be performed, as follows:

[0272] If the control module detects that the current connection does not need to perform a frame drop operation, the audio and video data received from the application layer will be passed to the sending module; if the control module detects that the current connection needs to perform a frame drop operation, it is necessary to perform a drop operation on non-critical video frames in the audio and video stream: In the embodiment of the present application, the control module has the ability to obtain the frame type of audio and video data, and can obtain the frame type, size and other information of the data to be sent through parsing. Specifically, the audio and video frame types include audio frames, video I frames, video P frames, discardable video B frames, and non-discardable video B frames; wherein, a discardable B frame means that other frames do not rely on this frame for encoding and decoding, and a non-discardable B frame means that other frames will rely on this frame for encoding and decoding; in the embodiment of the present application, when the control module performs a drop operation on the B frame of the current connection, it complies with the following frame drop rules:

[0273] i audio frames, video I frames, P frames and non-discardable B frames, etc. The above four types of audio and video frames cannot be discarded; the control module performs a frame drop operation on the discardable B frames;

[0274] ii. The playback time interval Delta_pts between two adjacent video frames after the frame loss cannot exceed its threshold Delta_threshold (ie, the third threshold). Figure 8 , Figure 8 A schematic diagram of determining a playback time interval provided in an embodiment of the present application is shown. Figure 8 In the example, there are multiple consecutive video frames, among which the fourth video frame 4 is a discardable video B frame. When deciding whether to discard the fourth video frame 4, the playback time interval between its two adjacent video frames (the third video frame 3 and the fifth video frame 5) is calculated. Specifically, it can be calculated using formula (9):

[0275] Delta_pts=Pts_5–Pts_3 (9)

[0276] Where Pts_3 and Pts_5 represent the playback time points of the third video frame 3 and the fifth video frame 5, respectively, on the client player. This is then compared with Delta_threshold. If the playback time interval Delta_pts is greater than the threshold Delta_threshold, the fourth video frame 4 is retained; otherwise, the fourth video frame 4 is discarded. In this embodiment of the present application, the threshold Delta_threshold is configured by the administrator and declared in the configuration file; illustratively, Delta_threshold = 200ms.

[0277] iii. In an embodiment of the present application, if multiple consecutive video frames are all discardable video B frames, the control module no longer discards all of the above B frames, but instead performs frame drop according to the parameter Amount_can_drop_b. Specifically, at most one of the consecutive Amount_can_drop_b discardable B frames can be discarded.

[0278] For example, please refer to Figure 9 , Figure 9 A schematic diagram of frame loss processing provided in an embodiment of the present application is shown. Figure 9 In the example, a video frame includes six frames, all of which can be discarded. If all six frames are discarded, the number of freezes or the duration of freezes in the client player may increase. In the embodiment of the present application, the parameter Amount_can_drop_b can be set to 3, that is, if three discardable frames appear consecutively, the control module can only discard one of them. For example, Figure 9 In the embodiment, the first video frame 1, the second video frame 2 and the third video frame 3 can be determined as a frame set 910, and then one video frame (for example, the first video frame 1) can be discarded from the frame set 910, and the remaining video frames can be retained, thereby completing the frame loss processing.

[0279] In an embodiment of the present application, after the control module performs the frame drop operation according to the above rules, the audio and video data after the frame drop is passed to the sending module; after receiving the above audio and video data from the control module, the sending module sends these data to the client in the form of a message, so that the audio and video data can be played on the client.

[0280] Reference Figure 10 , the embodiment of the present application further provides a multimedia resource transmission device, the device comprising:

[0281] The detection unit 1010 is used to detect network quality information between the client and the network;

[0282] The receiving unit 1020 is configured to receive first experience quality information fed back by the client at a current time node;

[0283] The processing unit 1030 is configured to determine, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each resource backup has a different bit rate, and the first resource set includes a predetermined number of resource backups;

[0284] A generating unit 1040 is configured to determine a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generate target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node;

[0285] The sending unit 1050 is configured to send the target multimedia resource data to the client.

[0286] Optionally, the network quality information includes available network bandwidth; and the processing unit is specifically configured to:

[0287] Sort each resource backup according to its bit rate;

[0288] Compare the available network bandwidth and the sizes of various bit rates, and determine a predetermined number of consecutively ordered resource backups as a first resource set based on the comparison result; wherein the first resource set includes resource backups whose corresponding bit rates are less than the available network bandwidth, and the first resource set includes resource backups whose corresponding bit rates are greater than the available network bandwidth.

[0289] Optionally, the processing unit is specifically configured to:

[0290] Record the sorting numbers corresponding to each resource backup in sorted order;

[0291] Comparing the available network bandwidth with the various bit rates, and determining a first bit rate from the bit rates backed up by the various resources; wherein the first bit rate is the bit rate that is closest to the available network bandwidth among the bit rates that are less than or equal to the available network bandwidth;

[0292] Querying a first sorting number corresponding to a first resource backup; wherein the first resource backup is a resource backup corresponding to a first bit rate;

[0293] Dividing the difference obtained by subtracting 1 from the predetermined number by 2, and rounding the quotient to obtain a first value;

[0294] Determine a second sort number based on a difference between the first sort number and the first value;

[0295] Starting from the second sorting number, a predetermined number of resource backups are selected in sequence in a sorting direction corresponding to increasing bit rates to obtain a first resource set.

[0296] Optionally, the first experience quality information includes first jamming indicator data; and the generating unit is specifically configured to:

[0297] Determine whether the client's lag situation has worsened based on the first lag indicator data and historical experience quality information;

[0298] If the client's freezing condition worsens, a target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

[0299] Optionally, the generating unit is specifically configured to:

[0300] Determining second experience quality information from historical experience quality information; the second experience quality information is experience quality information fed back by the client at a time node previous to the current time node, and the second experience quality information includes second jamming indicator data;

[0301] Calculating a first difference value between the first jamming indicator data and the second jamming indicator data;

[0302] comparing the first difference value with a first threshold;

[0303] If the first difference value is less than or equal to the first threshold, it is determined that the client's jamming situation has not worsened; or, if the first difference value is greater than the first threshold, it is determined that the client's jamming situation has worsened.

[0304] Optionally, the generating unit is specifically configured to:

[0305] Determine, based on historical quality of experience information, a first cumulative value corresponding to the jamming indicator data before the current time node;

[0306] Performing a weighted summation on the first jamming indicator data and the first cumulative value to obtain a second cumulative value;

[0307] calculating a second difference value of the second cumulative value relative to the first cumulative value;

[0308] Comparing the second difference value with the second threshold;

[0309] If the second difference value is less than or equal to the second threshold, it is determined that the client's jamming situation has not worsened; or, if the second difference value is greater than the second threshold, it is determined that the client's jamming situation has worsened.

[0310] Optionally, the generating unit is specifically configured to:

[0311] If the jamming condition of the client worsens, determining the jamming deterioration level corresponding to the client according to the first difference value and the first threshold;

[0312] According to the jamming deterioration level, the target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

[0313] Optionally, the first experience quality information includes first jamming indicator data and first cache indicator data; and the generating unit is specifically configured to:

[0314] Determine whether the client's lag situation has worsened based on the first lag indicator data and historical experience quality information;

[0315] Determining whether a cache condition of the client has deteriorated based on the first cache indicator data and historical experience quality information;

[0316] If the client's lag situation does not worsen and the client's cache situation worsens, the second resource backup in the first resource set is determined to be the target resource backup; wherein, the second resource backup is a resource backup corresponding to the second bit rate, and the second bit rate is greater than or equal to the bit rate used by the client at the previous time node of the current time node.

[0317] Optionally, the generating unit is specifically configured to:

[0318] If the client's freezing situation does not worsen and the client's cache situation worsens, the target resource backup is subjected to frame loss processing to generate target multimedia resource data.

[0319] Optionally, the generating unit is specifically configured to:

[0320] Detecting the frame type of each resource frame in the target resource backup;

[0321] Determine the target frame to be discarded in the target resource backup according to the frame type;

[0322] Detecting the playback time interval between two resource frames adjacent to the target frame;

[0323] If the playback time interval is less than or equal to the third threshold, the target frame is discarded; or if the playback time interval is greater than the third threshold, the target frame is retained.

[0324] Optionally, the generating unit is specifically configured to:

[0325] Detecting the frame type of each resource frame in the target resource backup;

[0326] Determine the target frame to be discarded in the target resource backup according to the frame type;

[0327] Determine the first consecutive target frames as a frame set;

[0328] A target frame in the frame set is discarded, and other target frames in the frame set except the discarded target frame are retained.

[0329] Optionally, the first experience quality information includes first jamming indicator data and first cache indicator data; and the generating unit is specifically configured to:

[0330] Determine whether the client's lag situation has worsened based on the first lag indicator data and historical experience quality information;

[0331] Determining whether a cache condition of the client has deteriorated based on the first cache indicator data and historical experience quality information;

[0332] If the client's freezing situation has not worsened and the client's cache situation has not worsened, the target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are greater than the available network bandwidth.

[0333] It is understandable that if Figure 3 The contents of the multimedia resource transmission method embodiment shown in FIG. 1 are applicable to the multimedia resource transmission device embodiment. The functions specifically implemented by the multimedia resource transmission device embodiment are similar to those in FIG. 1 . Figure 3 The multimedia resource transmission method embodiment shown is the same as that shown in FIG. Figure 3 The beneficial effects achieved by the embodiment of the multimedia resource transmission method shown are also the same.

[0334] The present application also discloses an electronic device, including:

[0335] at least one processor;

[0336] at least one memory for storing at least one program;

[0337] When at least one program is executed by at least one processor, the at least one processor implements Figure 3 An embodiment of a method for transmitting multimedia resources is shown.

[0338] It is understandable that if Figure 3 The contents of the multimedia resource transmission method embodiment shown in FIG. 1 are applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are similar to those in FIG. Figure 3 The multimedia resource transmission method embodiment shown is the same as that shown in FIG. Figure 3 The beneficial effects achieved by the embodiment of the multimedia resource transmission method shown are also the same.

[0339] The electronic device in the embodiment of the present application may be a terminal device, a computer device or a server device.

[0340] For example, referring to Figure 11 , Figure 11This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Taking the electronic device as a terminal device as an example, Figure 11 In the embodiment, the electronic device 1100 may include an RF (Radio Frequency) circuit 1110, a memory 1120 including one or more computer-readable storage media, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a short-range wireless transmission module 1170, a processor 1180 including one or more processing cores, and a power supply 1190. Those skilled in the art will appreciate that Figure 11 The device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0341] The RF circuit 1110 can be used to receive and transmit signals during information transmission or calls. Specifically, it receives downlink information from the base station and transmits it to one or more processors 1180 for processing. It also transmits uplink data to the base station. Typically, the RF circuit 1110 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a SIM card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, and the like. Furthermore, the RF circuit 1110 can communicate with the network and other devices via wireless communication. Wireless communication can utilize any communication standard or protocol, including but not limited to GSM (Global System of Mobile Communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, and SMS (Short Messaging Service).

[0342] The memory 1120 can be used to store software programs and modules (or units). The processor 1180 executes various functional applications and data processing by running the software programs and modules (or units) stored in the memory 1120. The memory 1120 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function), etc.; the data storage area may store data created according to the use of the electronic device 1100 (such as audio data, a phone book), etc. In addition, the memory 1120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1120 may also include a memory controller to provide the processor 1180 and the input unit 1130 with access to the memory 1120. Although Figure 11 The RF circuit 1110 is shown, but it is understandable that it is not an essential component of the electronic device 1100 and can be omitted as needed without changing the essence of the invention.

[0343] The input unit 1130 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to object settings and function control. Specifically, the input unit 1130 may include a touch-sensitive surface 1131 and other input devices 1132. The touch-sensitive surface 1131, also known as a touch display or touchpad, can detect touch operations performed by an object on or near it (for example, operations performed by an object using a finger, stylus, or any other suitable object or accessory on or near the touch-sensitive surface 1131) and drive corresponding connected devices according to a pre-set program. Optionally, the touch-sensitive surface 1131 may include a touch detection device and a touch controller. The touch detection device detects the touch position of the object and detects signals generated by the touch operation, transmitting the signals to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 1180. It can also receive and execute instructions from the processor 1180. In addition, the touch-sensitive surface 1131 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 1131, the input unit 1130 can also include other input devices 1132. Specifically, the other input devices 1132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, a joystick, and the like.

[0344] The display unit 1140 can be used to display information input by an object or information provided to an object and to control various graphic object interfaces of the electronic device 1100. These graphic object interfaces can be composed of graphics, text, icons, videos and any combination thereof. The display unit 1140 may include a display panel 1141. Optionally, the display panel 1141 may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like. Furthermore, the touch-sensitive surface 1131 may be covered on the display panel 1141. When the touch-sensitive surface 1131 detects a touch operation on or near it, it is transmitted to the processor 1180 to determine the type of touch event. The processor 1180 then provides corresponding visual output on the display panel 1141 according to the type of touch event. Although in Figure 11 In the embodiment, the touch-sensitive surface 1131 and the display panel 1141 are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface 1131 and the display panel 1141 can be integrated to implement input and output functions.

[0345] The electronic device 1100 may also include at least one sensor 1150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1141 or the backlight when the electronic device 1100 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the electronic device 1100 can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0346] Audio circuit 1160, speaker 1161, and microphone 1162 provide an audio interface between the subject and electronic device 1100. Audio circuit 1160 converts received audio data into electrical signals and transmits them to speaker 1161, which then converts them into sound signals for output. Microphone 1162, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1160 and converted into audio data. The audio data is then processed by processor 1180 and transmitted to another electronic device via RF circuit 1110, or the audio data is output to memory 1120 for further processing. Audio circuit 1160 may also include an earphone jack to facilitate communication between external headphones and electronic device 1100.

[0347] The short-range wireless transmission module 1170 may be a WIFI (wireless fidelity) module, a Bluetooth module, an infrared module, etc. The electronic device 1100 may transmit information to wireless transmission modules provided on other devices via the short-range wireless transmission module 1170 .

[0348] The processor 1180 is the control center of the electronic device 1100. It connects the various parts of the entire device using various interfaces and lines. By running or executing software programs or modules stored in the memory 1120 and calling data stored in the memory 1120, it performs various functions of the electronic device 1100 and processes data, thereby providing overall control over the device. Optionally, the processor 1180 may include one or more processing cores; optionally, the processor 1180 may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, object interface, and application programs, and the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 1180.

[0349] The electronic device 1100 also includes a power supply 1190 (e.g., a battery) for supplying power to various components. Optionally, the power supply 1190 can be logically connected to the processor 1180 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 1190 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0350] Although not shown, the electronic device 1100 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0351] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program is executed by the processor, it is used to implement the following Figure 3 An embodiment of a method for transmitting multimedia resources is shown.

[0352] It is understandable that Figure 3 The contents of the multimedia resource transmission method embodiment shown in the figure are applicable to the embodiment of the computer-readable storage medium. The functions specifically implemented by the computer-readable storage medium embodiment are similar to those of the embodiment of the computer-readable storage medium. Figure 3 The multimedia resource transmission method embodiment shown is the same as that of Figure 3 The beneficial effects achieved by the embodiment of the multimedia resource transmission method shown are also the same.

[0353] The embodiments of the present application further disclose a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in the above-mentioned computer-readable storage medium; Figure 11 The processor of the electronic device shown can read the computer instructions from the above-mentioned computer readable storage medium, and the processor executes the computer instructions so that the computer device executes Figure 3 An embodiment of a method for transmitting multimedia resources is shown.

[0354] It is understandable that Figure 3 The contents of the multimedia resource transmission method embodiment shown in the figure are applicable to the present computer program product or computer program embodiment, and the functions specifically implemented by the present computer program product or computer program embodiment are the same as those in the embodiment of FIG. Figure 3 The multimedia resource transmission method embodiment shown is the same as that of Figure 3 The beneficial effects achieved by the embodiment of the multimedia resource transmission method shown are also the same.

[0355] In some optional embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, the two boxes shown in succession may actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flow chart of the present application are provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logic flows presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0356] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present application as set forth in the claims using ordinary techniques without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0357] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0358] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0359] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0360] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0361] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0362] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for transmitting multimedia resources, characterized in that: The method comprises: Detect network quality information between the client and the client; receiving first experience quality information fed back by the client at a current time point; Determining, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each of the resource backups has a different bit rate, and the first resource set includes a predetermined number of resource backups; Determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generating target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node; The target multimedia resource data is sent to the client.

2. The method for transmitting multimedia resources according to claim 1, wherein: The network quality information includes available network bandwidth; and determining, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource, comprises: sorting the resource backups according to their bit rates; Compare the available network bandwidth and the sizes of each of the bit rates, and determine a predetermined number of consecutively ordered resource backups as a first resource set based on the comparison result; wherein, the first resource set includes resource backups whose corresponding bit rates are less than the available network bandwidth, and the first resource set includes resource backups whose corresponding bit rates are greater than the available network bandwidth.

3. The method for transmitting multimedia resources according to claim 2, wherein: The comparing the available network bandwidth and the respective bit rates, and determining a predetermined number of consecutively ordered resource backups as a first resource set according to the comparison result, includes: Record the sorting numbers corresponding to each resource backup in sorted order; Comparing the available network bandwidth with the respective bit rates, and determining a first bit rate from the bit rates of the resource backups; wherein the first bit rate is a bit rate that is closest to the available network bandwidth among the bit rates that are less than or equal to the available network bandwidth; Querying a first sorting number corresponding to a first resource backup; wherein the first resource backup is a resource backup corresponding to the first bit rate; Dividing the difference obtained by subtracting 1 from the predetermined number by 2, and rounding the quotient to obtain a first value; Determining a second sort number according to a difference between the first sort number and the first value; Starting from the second sorting number, in the sorting direction of increasing bit rate, the predetermined number of resource backups are selected in sequence to obtain a first resource set.

4. The method for transmitting multimedia resources according to claim 1, wherein: The first experience quality information includes first jamming indicator data; and determining target resource backup from the first resource set according to a change in the first experience quality information relative to historical experience quality information includes: Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information; If the jamming situation of the client worsens, a target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

5. The method for transmitting multimedia resources according to claim 4, characterized in that: The determining, based on the first jamming indicator data and the historical quality of experience information, whether the jamming condition of the client has deteriorated includes: Determining second experience quality information from the historical experience quality information; the second experience quality information is experience quality information fed back by the client at a time node previous to the current time node, and the second experience quality information includes second jamming indicator data; Calculating a first difference value between the first jamming indicator data and the second jamming indicator data; comparing the first difference value with a first threshold; If the first difference value is less than or equal to the first threshold, it is determined that the client's jamming situation has not worsened; or, if the first difference value is greater than the first threshold, it is determined that the client's jamming situation has worsened.

6. The method for transmitting multimedia resources according to claim 4, characterized in that: The determining, based on the first jamming indicator data and the historical quality of experience information, whether the jamming condition of the client has deteriorated includes: Determine, based on the historical quality of experience information, a first cumulative value corresponding to the jamming indicator data before the current time node; Performing a weighted summation on the first jamming indicator data and the first cumulative value to obtain a second cumulative value; calculating a second difference value of the second cumulative value relative to the first cumulative value; Comparing the second difference value with the second threshold; If the second difference value is less than or equal to the second threshold, it is determined that the client's jamming situation has not worsened; or, if the second difference value is greater than the second threshold, it is determined that the client's jamming situation has worsened.

7. The method for transmitting multimedia resources according to claim 5, characterized in that: If the jamming condition of the client worsens, determining a target resource backup from resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth includes: If the jamming condition of the client worsens, determining a jamming deterioration level corresponding to the client according to the first difference value and the first threshold; According to the jamming deterioration level, a target resource backup is determined from resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

8. The method for transmitting multimedia resources according to claim 1, wherein: The first experience quality information includes first jamming indicator data and first cache indicator data; and determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information includes: Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information; determining, based on the first cache indicator data and the historical quality of experience information, whether a cache situation of the client has deteriorated; If the client's lag situation does not worsen and the client's cache situation worsens, determine that the second resource backup in the first resource set is the target resource backup; wherein, the second resource backup is a resource backup corresponding to a second bit rate, and the second bit rate is greater than or equal to the bit rate used by the client at the previous time node of the current time node.

9. The method for transmitting multimedia resources according to claim 8, characterized in that: The generating target multimedia resource data based on the target resource backup includes: If the jamming situation of the client does not worsen, and the cache situation of the client worsens, frame loss processing is performed on the target resource backup to generate target multimedia resource data.

10. The method for transmitting multimedia resources according to claim 9, characterized in that: The performing frame loss processing on the target resource backup to generate target multimedia resource data includes: Detecting the frame type of each resource frame in the target resource backup; Determining, according to the frame type, a target frame to be discarded in the target resource backup; Detecting a playback time interval between two resource frames adjacent to the target frame; If the playback time interval is less than or equal to a third threshold, the target frame is discarded; or if the playback time interval is greater than the third threshold, the target frame is retained.

11. The method for transmitting multimedia resources according to claim 9, wherein: The performing frame loss processing on the target resource backup to generate target multimedia resource data includes: Detecting the frame type of each resource frame in the target resource backup; Determining, according to the frame type, a target frame to be discarded in the target resource backup; Determine the first consecutive target frames as a frame set; A target frame in the frame set is discarded, and other target frames in the frame set except the discarded target frame are retained.

12. The method for transmitting multimedia resources according to claim 1, wherein: The first experience quality information includes first jamming indicator data and first cache indicator data; and determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information includes: Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information; determining, based on the first cache indicator data and the historical quality of experience information, whether a cache situation of the client has deteriorated; If the client's freezing condition does not worsen and the client's cache condition does not worsen, a target resource backup is determined from resource backups in the first resource set whose corresponding bit rates are greater than the available network bandwidth.

13. A multimedia resource transmission device, characterized in that: The device comprises: A detection unit, used to detect network quality information between the client and the network; A receiving unit, configured to receive first experience quality information fed back by the client at a current time node; a processing unit, configured to determine, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each of the resource backups has a different bit rate, and the first resource set includes a predetermined number of resource backups; a generating unit, configured to determine a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generate target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node; A sending unit is used to send the target multimedia resource data to the client.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for transmitting multimedia resources according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for transmitting multimedia resources according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for transmitting multimedia resources according to any one of claims 1 to 12 is implemented.

Images