Resource allocation method, related device and storage medium

Abstract

The application discloses a resource allocation method, a first device, a second device and a storage medium. The method comprises the following steps: a first device of video communication acquires the characteristics of a video stream; based on the characteristics of the video stream and a first model, first information is determined, wherein the first information represents the complexity of the video stream; and the first information is sent to a selective forwarding unit (SFU) server, wherein the first information is used for the second device of video communication to allocate receiving resources for the video stream sent by the first device. According to the scheme disclosed by the application, resources can be evenly allocated for the video streams sent by different terminals.

Description

Technical Field

[0001] This application relates to the field of communications, and more particularly to a resource allocation method, related equipment, and storage medium. Background Technology

[0002] In video communication, to ensure good call quality, servers can introduce Quality of Service (QoS) policies to address issues such as network latency and congestion. In multi-party video communication scenarios, servers need to implement corresponding QoS policies for different terminals to allocate resources to the bitstreams sent by different terminals in order to meet the bitstream processing needs of multiple terminals in video communication.

[0003] However, how to evenly allocate resources to the bitstreams sent by different terminals is a problem that urgently needs to be solved. Summary of the Invention

[0004] To address the related technical issues, embodiments of this application provide a resource allocation method, related equipment, and storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] A resource allocation method, applied to a first device for video communication, comprising:

[0007] Obtain the features of the video stream;

[0008] Based on the features of the video stream and the first model, first information is determined, wherein the first information characterizes the complexity of the video stream;

[0009] The first information is sent to the Selective Forwarding Unit (SFU) server, and the first information is used to allocate receiving resources for the video stream sent by the first device in the video communication.

[0010] In the above scheme, obtaining the features of the video stream includes:

[0011] The features of each frame in the video stream are obtained; the features include global motion vectors, macroblock information statistics and bitrate.

[0012] In the above scheme, sending the first information to the SFU server includes:

[0013] Send control signaling to the SFU server, the control signaling containing the first information.

[0014] The method in the above scheme further includes:

[0015] The second information received from the SFU server represents the bitrate of the video stream desired by the second device.

[0016] This application also provides a resource allocation method, applied to a second device for video communication, including:

[0017] First information of at least one first device receiving video communication sent by an SFU server, the first information characterizing the complexity of the video stream sent by the first device, the first information being determined by the first device based on the features of the video stream and a first model;

[0018] Based on the first information of the at least one first device, receive resources are allocated for the video stream sent by each of the at least one first device.

[0019] In the above scheme, the first information of at least one first device receiving video communication sent by the SFU server includes:

[0020] The control signaling of at least one first device receiving video communication sent by the SFU server includes first information of the corresponding first device.

[0021] The method in the above scheme further includes:

[0022] For each of the at least one first device, second information is determined based on the receiving resources allocated to the first device; the second information characterizes the bitrate of the video stream that the second device expects the first device to send.

[0023] The second information is sent to the SFU server.

[0024] This application embodiment also provides a first device, including a first communication interface and a first processor;

[0025] The first processor is configured to acquire features of the video stream; and based on the features of the video stream and a first model, determine first information, wherein the first information characterizes the complexity of the video stream.

[0026] The first communication interface is used to send the first information to the SFU server, and the first information is used to allow the second device of video communication to allocate receiving resources for the video stream sent by the first device.

[0027] This application embodiment also provides a second device, including a second communication interface and a second processor;

[0028] The second communication interface is used to receive first information of at least one first device for video communication sent by the SFU server. The first information characterizes the complexity of the video stream sent by the first device. The first information is determined by the first device based on the characteristics of the video stream and a first model.

[0029] The second processor is configured to allocate receiving resources for the video stream sent by each of the at least one first device based on the first information of the at least one first device.

[0030] This application embodiment also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above on the first device side, or implements the steps of any of the methods described above on the second device side.

[0031] The resource allocation method, related equipment, and storage medium provided in this application embodiment involve a first device for video communication acquiring features of a video stream; determining first information based on the features of the video stream and a first model, whereby the first information characterizes the complexity of the video stream; and sending the first information to an SFU server, whereby the first information is used by a second device for video communication to allocate receiving resources for the video stream sent by the first device. The solution provided in this application embodiment reduces server resource usage by utilizing an SFU server to forward the bitstream of the video communication terminal. That is, while ensuring reduced server resource overhead, sending the first information to the second device allows the second device to know the complexity of the received bitstream, thereby enabling resource allocation for different bitstreams based on their complexity, ensuring resource balance among different bitstreams, and ultimately improving the video communication quality of multi-party calls. Attached Figure Description

[0032] Figure 1 This is a schematic flowchart of a first resource allocation method according to an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the training process of the first model in the resource allocation method of this application embodiment;

[0034] Figure 3 This is a schematic flowchart of the second resource allocation method according to an embodiment of this application;

[0035] Figure 4 This is a schematic flowchart of the third resource allocation method according to an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of the multi-party video call system architecture in the multi-party video communication bandwidth allocation method, which is an application example of this application.

[0037] Figure 6This is a schematic diagram illustrating the feature value parameter transmission process in the multi-party video communication bandwidth allocation method, which is an application example of this application.

[0038] Figure 7 This is a schematic diagram of the bandwidth allocation process in the multi-party video communication bandwidth allocation method, which is an application example of this application.

[0039] Figure 8 This is a schematic diagram of a resource allocation device architecture according to an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of another resource allocation device architecture according to an embodiment of this application;

[0041] Figure 10 This is a schematic diagram of the first device architecture according to an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of the second device architecture according to an embodiment of this application;

[0043] Figure 12 This is a schematic diagram of the resource allocation system architecture in an embodiment of this application. Detailed Implementation

[0044] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0045] With the development of the Internet of Things (IoT), the audio and video interaction capabilities of consumer-grade smart terminals (such as smartwatches, network cameras (IPC, Internet Protocol Cameras), smart robots, and smart access control systems) are becoming increasingly rich. For example, smart children's watches can now support audio and video call services up to 720P. The application scenarios of audio and video communication include not only simple one-to-one call scenarios but also multi-party call scenarios.

[0046] During audio and video communication, factors such as network jitter, network congestion, equipment aging, and backbone network traffic control can lead to poor communication quality. Servers improve this by introducing audio and video QoS policies. QoS policies mainly include network congestion control and network jitter mitigation, specifically bandwidth estimation, packet loss retransmission, and error control. In multi-party call scenarios, to enable one party (a terminal) to communicate simultaneously with several other parties (other terminals), the server (e.g., a microcontroller unit (MCU) server) needs to implement corresponding QoS policies for different terminals and estimate the bandwidth of the video streams sent by the other parties to simultaneously meet their different bitrate requirements. However, implementing separate QoS policies for different terminals significantly increases server resource overhead, and this overhead increases with the number of participating video communication terminals, leading to higher server expansion and operating costs.

[0047] To address the aforementioned issues, related technologies utilize an SFU server to forward the bitstreams sent by each terminal, thereby reducing server resource overhead. However, how the receiving device can evenly allocate resources among the multiple received bitstreams remains a pressing problem to be solved.

[0048] Based on this, in various embodiments of this application, by using the SFU server to forward the bitstream of the video communication terminal, the use of server resources is reduced. That is, while ensuring the reduction of server resource overhead, by sending the first information to the second device, the second device can know the complexity of the received bitstream, thereby allocating resources to different bitstreams according to the complexity, ensuring resource balance among different bitstreams, and thus improving the video communication quality of multi-party calls.

[0049] This application provides a resource allocation method applied to a first device for video communication, such as... Figure 1 As shown, the method includes:

[0050] Step 101: Obtain the features of the video stream;

[0051] Step 102: Based on the features of the video stream and the first model, determine the first information, which characterizes the complexity of the video stream;

[0052] Step 103: Send the first information to the SFU server. The first information is used to allocate receiving resources for the video stream sent by the first device by the second device in video communication.

[0053] In practical applications, the first device can also be referred to as a transmitting device, and this application embodiment does not limit this, as long as it can achieve its function; correspondingly, the second device can also be referred to as a receiving device, and this application embodiment does not limit this, as long as it can achieve its function. In practical applications, the first device and the second device can specifically be terminals, more specifically, mobile terminals, personal computers (PCs), smart TVs, or wearable devices, etc.

[0054] In practical applications, the video stream can also be called a bitstream. This application embodiment does not limit this, as long as its function can be achieved.

[0055] In one embodiment, acquiring the features of the video stream may include acquiring the features of each frame in the video stream.

[0056] In practical applications, the first device can obtain the features of the video stream from the encoded video stream after the encoder encodes the video stream; correspondingly, it can obtain the features of each frame in the video stream, that is, obtain the features of each frame in the encoded video stream.

[0057] In practical applications, the features include global motion vectors, macroblock information statistics, and bitrate. The global motion vectors represent the overall motion of the video; the macroblock information statistics represent macroblock parameter information, such as macroblock mode selection information (e.g., 4x4, 16x16); and the bitrate represents the number of bits per second of the video, such as 600kbps.

[0058] In practical applications, in step 102, after acquiring the features of the video stream, the first device uses the first model to calculate the complexity of the video stream. Specifically, the features of the video stream are input into the first model, and the complexity of the video stream is determined by the first model. The first model can also be called a complexity model; this application embodiment does not limit this, as long as it can achieve its function.

[0059] In practical applications, the first device can train and generate the first model using historical sample data from the video stream.

[0060] Based on this, in one embodiment, the method may further include:

[0061] The first model is trained using features from each frame in the historical video stream.

[0062] In practical applications, this involves utilizing the features of each frame in the historical video stream, that is, utilizing the features of each frame in the encoded historical video stream. Specifically, for example... Figure 2As shown, the global motion vector, macroblock information statistics, and bitrate of each frame in the encoded historical video stream are used as feature parameters and input into the first model for training.

[0063] In practical applications, the first model may include a neural network model; wherein the neural network model may include a ReLU activation function, which is used to delinearize the data input to the neural network model. Compared to other activation functions, when using the ReLU activation function for delinearization, only a threshold value needs to be set to obtain the activation value, which is not only simple to implement, but also can improve the convergence speed of the model during gradient descent.

[0064] For example, the ReLU activation function can be represented by the following formula:

[0065] q(x) = max(0, x) (1)

[0066] x = GMV*a + MB*b + FPS*c (2)

[0067] Where x represents complexity, GMV represents the global motion vector normalization value, MB represents the macroblock average complexity, FPS represents the frame rate, and a, b, and c are coefficient values, and a+b+c=1; the values ​​of a, b, and c are dynamically adjusted by the programmer based on experience.

[0068] In practical applications, after the first model has been trained, the features of the input video stream can be input into the first model to calculate the complexity of the video stream.

[0069] In practical applications, the first device can be equipped with a complexity statistics module, which includes a first model.

[0070] In video communication, the Real-time Transport Protocol (RTP) is used to transmit media traffic (e.g., video streams), and the RTP Control Protocol (RTCP) is used to transmit control traffic; therefore, RTCP can be used to transmit the first information.

[0071] Based on this, in one embodiment, sending the first information to the SFU server includes:

[0072] Send control signaling to the SFU server, the control signaling containing the first information.

[0073] In practical applications, the first device can be equipped with an RTCP sending module, which is used to send control signaling. Specifically, the RTCP sending module adds first information to a custom field of the control signaling and sends it. For example, after receiving the first information output by the first model, the RTCP sending module adds a video complexity identifier to the application data information according to the first information and sends it.

[0074] In practical applications, the first information can be sent according to a preset period, or it can be updated and sent when the application scenario changes.

[0075] Based on this, in one embodiment,

[0076] The first device periodically sends the first information to the SFU server;

[0077] And / or,

[0078] Upon determining that the complexity of the video stream has changed, the first device sends updated first information to the SFU server.

[0079] In practical applications, the first device can periodically send the first information to the SFU service using a timer. Specifically, the first device uses the RTCP sending module to receive the first information sent by the first model, and under the control of the timer, uses the RTCP sending module to add the first information to the RTCP custom field and send it to the SFU server at preset time intervals.

[0080] Of course, in practical applications, when the video stream scenario changes, such as when the number of people in the video stream changes, the complexity of the video stream will also change. At this time, the bandwidth required to process the corresponding video stream will also change. Therefore, the first device can also determine whether the complexity of the video stream has changed based on the first information received, and when the complexity of the video stream changes, add the first information to the RTCP custom field and send it to the SFU server using the RTCP sending module.

[0081] In practical applications, before the second device allocates resources to the video stream sent by the first device based on the first information, the first device needs to send the video stream to the second device.

[0082] Based on this, in one embodiment, the method may further include:

[0083] A video stream is sent to the SFU server so that the SFU server sends the video stream to the second device.

[0084] In practical applications, the first device can send a video stream to the SFU server based on RTP. This sending of the video stream by the first device to the SFU server can occur before, after, or simultaneously with the first device sending the first information to the SFU server, as long as the first device sends the video stream to the SFU server before the second device allocates bandwidth for the video stream. The specific settings are determined by the programmer based on the actual application scenario, and this application embodiment does not impose any limitations on this.

[0085] In practical applications, in order to flexibly adapt to the processing capabilities of the second device, the first device can send bitstreams with different bitrates to the SFU server.

[0086] Based on this, in one embodiment, sending the video stream to the SFU server includes:

[0087] A first video stream and a second video stream are sent to the SFU server so that the SFU server sends the first video stream or the second video stream to the second device; the bitrate of the first video stream is greater than the bitrate of the second video stream.

[0088] In practical applications, the first video stream can also be called the mainstream or the high bitrate stream. This application embodiment does not limit this, as long as its function can be achieved. The second video stream can also be called the sub-stream or the low bitrate stream. This application embodiment does not limit this, as long as its function can be achieved.

[0089] In practical applications, the first device sends both the first video stream and the second video stream to the SFU server. The SFU server can first send the first video stream to the second device. When the second device determines that the allocated bandwidth cannot meet the bandwidth requirements of the first video stream, it will send a request to the SFU server so that the SFU server can send the second video stream to the second device to adapt to the processing capacity of the second device, thereby ensuring the normal display of the video stream on the second device.

[0090] After the first device sends the first information to the SFU server, the SFU server will forward the first information to the second device, so that the second device can know the complexity of each first device and allocate bandwidth to the video stream sent by each first device according to the complexity of different first devices.

[0091] In practical applications, the second device allocates bandwidth to each video stream sent by the first device based on its total bandwidth. When the bandwidth allocated by the second device to the first device does not match the bandwidth requirements of the video stream sent by the first device, the second device can send feedback information to the first device so that the first device can adjust the bitrate of the video stream to adapt to the bandwidth resources allocated by the second device.

[0092] Based on this, in one embodiment, the method further includes:

[0093] The second information received from the SFU server represents the bitrate of the video stream desired by the second device.

[0094] In practical applications, the second information can also be referred to as the expected bitrate or the anticipated bitrate. This application does not limit this terminology, as long as the function can be achieved. The second information is used to allow the corresponding first device to adjust the bitrate of the transmitted video stream to adapt to the resources allocated by the second device for the corresponding video stream.

[0095] In practical applications, the desired bitrate can be lower than or higher than the bitrate of the video stream sent by the first device. Specifically, when the bandwidth resources allocated by the second device to the first device cannot meet the bandwidth requirements of the video stream sent by the first device, the second device can send feedback information to the first device through the SFU server to enable the first device to reduce the bitrate of the video stream. Conversely, when the bandwidth resources allocated by the second device to the first device are higher than the bandwidth requirements of the video stream sent by the first device, the second device can send feedback information to the first device through the SFU server to enable the first device to increase the bitrate of the video stream.

[0096] Accordingly, embodiments of this application also provide a resource allocation method applied to a second device for video communication, such as... Figure 3 As shown, the method includes:

[0097] Step 301: Receive first information of at least one first device for video communication sent by the SFU server, wherein the first information characterizes the complexity of the video stream sent by the first device, and the first information is determined by the first device based on the characteristics of the video stream and a first model;

[0098] Step 302: Based on the first information of the at least one first device, allocate receiving resources for the video stream sent by each of the at least one first device.

[0099] In practical applications, the first device can send the first information according to a preset cycle, or it can update and send the information when the application scenario changes.

[0100] Based on this, in one embodiment,

[0101] The second device periodically receives the first information sent by the SFU server;

[0102] And / or,

[0103] When the first device determines that the complexity of the video stream has changed, the second device receives updated first information sent by the SFU server.

[0104] In practical applications, the first device can transmit the first information based on RTCP.

[0105] Based on this, in one embodiment, the first information of at least one first device receiving video communication sent by the SFU server includes:

[0106] The second device receives control signaling from at least one first device for video communication sent by the SFU server, the control signaling containing first information of the corresponding first device.

[0107] In practical applications, before allocating resources for the video stream sent by the first device based on the first information, the second device needs to receive the video stream sent by the second device first.

[0108] Based on this, in one embodiment, the method may further include:

[0109] The second device receives at least one video stream sent by the first device from video communications sent by the SFU server.

[0110] In practical applications, in order to flexibly adapt to the processing capabilities of the second device, the first device can send bitstreams with different bitrates to the SFU server.

[0111] Based on this, in one embodiment, receiving a video stream sent by at least one first device that receives video communication sent by an SFU server includes:

[0112] In each of at least one first device for video communication, the second device receives a first video stream or a second video stream sent by the SFU server corresponding to the first device; the bit rate of the first video stream is greater than the bit rate of the second video stream.

[0113] In practical applications, the first device sends both the first video stream and the second video stream to the SFU server. The SFU server can first send the first video stream to the second device. After the second device allocates resources for each video stream according to the complexity of all received video streams, the second device determines whether the allocated resources match the first video stream, that is, whether they can meet the bandwidth required to process the first video stream. If the determination result indicates that the bandwidth requirement can be met, the first video stream is processed and displayed. If the determination result indicates that the bandwidth requirement cannot be met, in order to ensure the normal display of the video stream on the second device, the second device needs to request the SFU server to send a second video stream with a lower bitrate.

[0114] Based on this, in one embodiment, receiving the first video stream or the second video stream sent by the SFU server corresponding to the first device includes:

[0115] Receive the first video stream sent by the SFU server, corresponding to the first device;

[0116] If it is determined that the resources allocated to the first device do not meet the resource requirements of the first video stream, a first request is sent to the SFU server; the first request is used to request to obtain the second video stream corresponding to the first device.

[0117] Receive the second video stream sent by the SFU server in response to the first request.

[0118] In practical applications, when the SFU server responds to the first request and sends the second video stream to the second device, it can stop sending the corresponding first video stream to the second device.

[0119] In practical applications, after the second device receives the first information sent by at least one first device in video communication, it allocates resources for the video stream sent by each first device according to the first information of each first device.

[0120] In practical applications, the second device can allocate bandwidth to the corresponding video stream according to the proportion of the video stream's complexity to the total complexity of all video streams.

[0121] In practical applications, in order to ensure the image quality of all video streams on the second device, the second device also needs to ensure that the allocated bandwidth is not lower than the preset minimum value during the bandwidth allocation process; the preset minimum value can be set in advance based on experience value or subjective evaluation experience score.

[0122] For example, the correspondence between video stream complexity and allocated bandwidth can be expressed as:

[0123]

[0124] Where Bit(n) represents the number of display bits of the video stream sent by device n, c(n) represents the complexity of the target stream, c(total) represents the overall complexity of the received stream, totalbit represents the current downlink total bandwidth, and minbitrate represents the minimum bitrate of the video stream.

[0125] In practical applications, the second device allocates bandwidth to each video stream sent by the first device based on its total bandwidth. When the bandwidth allocated by the second device to the first device does not match the bandwidth requirements of the video stream sent by the first device, the second device can send feedback information to the first device so that the first device can adjust the bitrate of the video stream to adapt to the bandwidth resources allocated by the second device.

[0126] Based on this, in one embodiment, the method may further include:

[0127] For each of the at least one first device, second information is determined based on the receiving resources allocated to the first device; the second information characterizes the bitrate of the video stream that the second device expects the first device to send.

[0128] The second information is sent to the SFU server.

[0129] In practical applications, the desired bitrate can be lower than or higher than the bitrate of the video stream sent by the first device. Specifically, when the bandwidth resources allocated by the second device to the first device cannot meet the bandwidth requirements of the video stream sent by the first device, the second device can send feedback information to the first device through the SFU server to enable the first device to reduce the bitrate of the video stream. Conversely, when the bandwidth resources allocated by the second device to the first device are higher than the bandwidth requirements of the video stream sent by the first device, the second device can send feedback information to the first device through the SFU server to enable the first device to increase the bitrate of the video stream.

[0130] It should be noted that when the second device determines that its bandwidth cannot meet the bandwidth requirements of the video stream, the second device can send a first request to the SFU server to cause the SFU server to send a video stream with a lower bitrate; alternatively, it can send a second message to the SFU server to cause the first device to down-bitrate the sent video stream. Of course, the sending of the first request or the second message can also be determined based on preset conditions. For example, if the desired bitrate of the video stream is A, the preset first threshold is B, and the preset second threshold is C, then when A > B, the second message is sent; when B > A > C, the first request is sent. The specific sending method is determined by the programmer according to the specific application scenario, and this application embodiment does not limit this.

[0131] This application also provides a resource allocation method, such as... Figure 4 As shown, the method includes:

[0132] Step 401: The first device in video communication acquires the characteristics of the video stream;

[0133] Step 402: The first device determines first information based on the features of the video stream and the first model, wherein the first information characterizes the complexity of the video stream;

[0134] Step 403: The first device sends the first information to the SFU server. The first information is used to allocate receiving resources for the video stream sent by the first device by the second device for video communication.

[0135] Step 404: The second device receives at least one first information of the first device sent by the SFU server for video communication. The first information characterizes the complexity of the video stream sent by the first device. The first information is determined by the first device based on the characteristics of the video stream and a first model.

[0136] Step 405: The second device allocates receiving resources for the video stream sent by each of the at least one first device based on the first information of the at least one first device.

[0137] It should be noted that the specific processing procedures of the first and second devices have been detailed above and will not be repeated here.

[0138] The resource allocation method provided in this application embodiment involves a first video communication device acquiring features of a video stream; determining first information based on the features of the video stream and a first model, whereby the first information characterizes the complexity of the video stream; and sending the first information to an SFU server, whereby the first information is used by a second video communication device to allocate receiving resources for the video stream sent by the first device. The solution provided in this application embodiment reduces server resource usage by utilizing an SFU server to forward the bitstream of the video communication terminal. That is, while ensuring reduced server resource overhead, sending the first information to the second device allows the second device to know the complexity of the received bitstream, thereby enabling resource allocation for different bitstreams based on their complexity, ensuring resource balance among different bitstreams, and ultimately improving the video communication quality of multi-party calls.

[0139] The following section provides a more detailed description of this application with reference to application examples.

[0140] This application example provides a method for bandwidth allocation in multi-party video communication, applicable to, for example... Figure 5 In the multi-party video call system shown, the method includes:

[0141] Step 1: Terminal A, Terminal B, and Terminal C in the multi-party video send the main stream and sub-stream to the SFU server respectively. That is, the first device sends the video stream to the SFU server, and then Step 2 is executed.

[0142] Here, the mainstream refers to the high bitrate, i.e., the primary video stream, such as... Figure 5 As shown by the solid line, the sub-stream, also known as the low-bitrate stream, is the second video stream, as shown below. Figure 5 As shown by the dashed line;

[0143] Meanwhile, for any of terminals A, B, and C, based on the complexity of the bitstream calculated by the video complexity statistics module, a video complexity flag is added to the custom field of RTCP's app data and then sent at certain intervals.

[0144] Here, the bitstream complexity is calculated using the complexity estimation model in the video complexity estimation module, such as... Figure 2 As shown, the global motion vector, macroblock information statistics, and frame bitrate of each frame during the encoding process are used to train the neural network, and a complexity estimation model is obtained.

[0145] The neural network uses the ReLU activation function, which can be represented by formulas (1) and (2).

[0146] In practical applications, such as Figure 6As shown, the encoder sends the feature value parameters of the video stream to the complexity statistics module in real time, and the complexity estimation model calculates the complexity of the video stream; under the control of the timer, the RTCP sending module puts the complexity into the RTCP custom field App data at certain time intervals and sends it to the SFU server.

[0147] Step 2: The SFU server forwards the main streams sent by terminals A, B, and C to terminal D, and then executes step 3;

[0148] Step 3: After receiving the bitstreams sent by terminals A, B, and C, as follows: Figure 7 As shown, terminal D performs overall bandwidth estimation and intelligent allocation based on the complexity of each bitstream to ensure the best quality of experience (QOE) for the video streams of terminals A, B, and C displayed on the terminal D side.

[0149] Specifically, the relationship between the complexity of the bitstream and the allocated bandwidth can be expressed using formula (3).

[0150] Here, after terminal D calculates the input bitrate of each stream, it feeds back the expected bitrate of each end to each sending end based on the bandwidth estimation result and the complexity value, so that the sending end can downsample the stream to adapt to the bandwidth allocated by terminal D. When terminal D determines that the mainstream of terminal A, terminal B, or terminal C does not meet the corresponding bandwidth requirements, that is, when the display effect of the mainstream after downsampling is lower than the display effect of the sub-stream, terminal D sends a request to the SFU server to request the SFU server to forward the sub-stream of the corresponding terminal to terminal D.

[0151] Step 4: Repeat steps 1 to 3 to achieve adaptive bandwidth of the video stream during video communication, thereby ensuring communication quality.

[0152] The video communication bandwidth allocation method provided in this application example utilizes RTCP to transmit complexity estimates in real-time communication, enabling the receiving end to adaptively allocate bandwidth and bitrate under multi-party scenarios based on complexity. This improves video quality in multi-party call scenarios and enhances the user's viewing experience. Simultaneously, it eliminates the reliance on server bandwidth estimation, reducing server resource consumption and avoiding the problem of increased hardware resource requirements caused by server architecture modifications.

[0153] To implement the method on the first device side of this application, embodiments of this application also provide a resource allocation device, disposed on the first device for video communication, such as... Figure 8 As shown, the device includes:

[0154] Acquisition unit 801 is used to acquire features of the video stream;

[0155] The first determining unit 802 is used to determine first information based on the features of the video stream and the first model, wherein the first information characterizes the complexity of the video stream;

[0156] The first sending unit 803 is used to send the first information to the SFU server. The first information is used to allocate receiving resources for the video stream sent by the second device in video communication.

[0157] In one embodiment, the acquisition unit 801 is specifically used for:

[0158] The features of each frame in the video stream are obtained; the features include global motion vectors, macroblock information statistics and bitrate.

[0159] In one embodiment, the first transmitting unit 803 is specifically used for:

[0160] Send control signaling to the SFU server, the control signaling containing the first information.

[0161] In one embodiment, the device further includes:

[0162] The first receiving unit is used to receive second information fed back by the SFU server, the second information representing the bitrate of the video stream expected by the second device.

[0163] In practical applications, the first sending unit 803 and the first receiving unit can be implemented by the communication interface in the resource allocation device, the acquisition unit 801 can be implemented by the processor in the resource allocation device in combination with the communication interface, and the first determining unit 802 can be implemented by the processor in the resource allocation device.

[0164] To implement the method on the second device side of this application, embodiments of this application also provide a resource allocation device, disposed on the second device for video communication, such as... Figure 9 As shown, the device includes:

[0165] The second receiving unit 901 is used to receive first information of at least one first device for video communication sent by the SFU server. The first information characterizes the complexity of the video stream sent by the first device. The first information is determined by the first device based on the characteristics of the video stream and a first model.

[0166] The allocation unit 902 is configured to allocate receiving resources for the video stream sent by each of the at least one first device based on the first information of the at least one first device.

[0167] In one embodiment, the second receiving unit 901 is specifically used for:

[0168] The control signaling of at least one first device receiving video communication sent by the SFU server includes first information of the corresponding first device.

[0169] In one embodiment, the device may further include:

[0170] The second determining unit is configured to determine second information for each of the at least one first device, based on the receiving resources allocated to the first device; the second information represents the bit rate of the video stream that the second device expects the first device to send.

[0171] The second sending unit is used to send the second information to the SFU server.

[0172] In practical applications, the second sending unit and the second receiving unit 901 can be implemented by the communication interface in the resource allocation device, and the allocation unit 902 and the second determining unit can be implemented by the processor in the resource allocation device.

[0173] It should be noted that the resource allocation device provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the resource allocation device and the resource allocation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0174] Based on the hardware implementation of the above program modules, and in order to implement the method on the first device side of the embodiments of this application, the embodiments of this application also provide a first device, such as... Figure 10 As shown, the first device 1000 includes:

[0175] The first communication interface 1001 is capable of exchanging information with the SFU server; for example, sending first information to the Selective Forwarding Unit (SFU) server.

[0176] The first processor 1002 is connected to the first communication interface 1001 to enable information interaction with the SFU server and to execute the methods provided by one or more technical solutions on the first device side when running a computer program.

[0177] The computer program is stored in the first memory 1003.

[0178] Specifically, the first processor 1002 is used to acquire features of the video stream; and based on the features of the video stream and the first model, determine first information, wherein the first information characterizes the complexity of the video stream;

[0179] The first communication interface 1001 is used to send the first information to the SFU server. The first information is used to allocate receiving resources for the video stream sent by the second device in video communication.

[0180] In one embodiment, the first processor 1002 is specifically used for:

[0181] The features of each frame in the video stream are obtained; the features include global motion vectors, macroblock information statistics and bitrate.

[0182] In one embodiment, the first communication interface 1001 is specifically used for:

[0183] Send control signaling to the SFU server, the control signaling containing the first information.

[0184] In one embodiment, the first communication interface 1001 is further configured to:

[0185] The second information received from the SFU server represents the bitrate of the video stream desired by the second device.

[0186] It should be noted that the specific processing procedures of the first processor 1002 and the first communication interface 1001 can be understood by referring to the above method.

[0187] Of course, in practical applications, the various components in the first device 1000 are coupled together through the bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 10 The general labeled all buses as Bus System 1004.

[0188] The first memory 1003 in this embodiment is used to store various types of data to support the operation of the first device 1000. Examples of such data include any computer program used to operate on the first device 1000.

[0189] The methods disclosed in the embodiments of this application can be applied to the first processor 1002, or implemented by the first processor 1002. The first processor 1002 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 1002. The first processor 1002 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 1002 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 1003. The first processor 1002 reads the information in the first memory 1003 and completes the steps of the aforementioned method in combination with its hardware.

[0190] In an exemplary embodiment, the first device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0191] Based on the hardware implementation of the above program modules, and in order to implement the method on the second device side of the embodiments of this application, the embodiments of this application also provide a second device, such as... Figure 11 As shown, the second device 1100 includes:

[0192] The second communication interface 1101 is capable of exchanging information with the SFU server; for example, receiving first information of at least one first device in video communication sent by the SFU server.

[0193] The second processor 1102 is connected to the second communication interface 1101 to enable information interaction with the SFU server and to execute the methods provided by one or more technical solutions on the second device side when running computer programs.

[0194] The computer program is stored in the second memory 1103.

[0195] Specifically, the second communication interface 1101 is used to receive first information of at least one first device for video communication sent by the SFU server. The first information characterizes the complexity of the video stream sent by the first device. The first information is determined by the first device based on the characteristics of the video stream and a first model.

[0196] The second processor 1102 is configured to allocate receiving resources for the video stream sent by each of the at least one first device based on the first information of the at least one first device.

[0197] In one embodiment, the second communication interface 1101 is specifically used for:

[0198] The control signaling of at least one first device receiving video communication sent by the SFU server includes first information of the corresponding first device.

[0199] In one embodiment, the second processor 1102 is further configured to:

[0200] For each of the at least one first device, second information is determined based on the receiving resources allocated to the first device; the second information characterizes the bitrate of the video stream that the second device expects the first device to send.

[0201] The second information is sent to the SFU server using the second communication interface 1101.

[0202] It should be noted that the specific processing procedures of the second processor 1102 and the second communication interface 1101 can be understood by referring to the above method.

[0203] Of course, in practical applications, the various components in the second device 1100 are coupled together via the bus system 1104. It can be understood that the bus system 1104 is used to implement communication between these components. In addition to a data bus, the bus system 1104 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 11 The general designated all buses as Bus System 1104.

[0204] The second memory 1103 in this embodiment is used to store various types of data to support the operation of the second device 1100. Examples of such data include any computer program used to operate on the second device 1100.

[0205] The methods disclosed in the above embodiments of this application can be applied to the second processor 1102, or implemented by the second processor 1102. The second processor 1102 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the second processor 1102. The second processor 1102 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The second processor 1102 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the second memory 1103. The second processor 1102 reads the information in the second memory 1103 and completes the steps of the aforementioned method in conjunction with its hardware.

[0206] In an exemplary embodiment, the second device 1100 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0207] It is understood that the memories (first memory 1003, second memory 1103) in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0208] To implement the method of the embodiments of this application, the embodiments of this application also provide a resource allocation system, such as... Figure 12 As shown, the system includes: a first device 1201 and a second device 1202.

[0209] It should be noted that the specific processing procedures of the first device 1201 and the second device 1202 have been described in detail above and will not be repeated here.

[0210] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 1003 storing a computer program, which can be executed by the first processor 1002 of the first device 1000 to complete the steps described in the aforementioned terminal-side method. Another example is a second memory 1103 storing a computer program, which can be executed by the second processor 1102 of the second device 1100 to complete the steps described in the aforementioned network device-side method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0211] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0212] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0213] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A resource allocation method, characterized in that, The first device used in video communication includes: Obtain the features of the video stream; Based on the features of the video stream and the first model, first information is determined, wherein the first information characterizes the complexity of the video stream; The first information is sent to the Selective Forwarding Unit (SFU) server, and the first information is used to allow the second device in video communication to allocate receiving resources for the video stream sent by the first device. The first device is a transmitting device; the second device is a receiving device.

2. The method according to claim 1, characterized in that, The features of the acquired video stream include: The features of each frame in the video stream are obtained; the features include global motion vectors, macroblock information statistics, and bitrate.

3. The method according to claim 1, characterized in that, Sending the first information to the SFU server includes: Send control signaling to the SFU server, the control signaling containing the first information.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The second information received from the SFU server represents the bitrate of the video stream desired by the second device.

5. A resource allocation method, characterized in that, Second devices used in video communication include: First information of at least one first device receiving video communication sent by an SFU server, the first information characterizing the complexity of the video stream sent by the first device, the first information being determined by the first device based on the features of the video stream and a first model; Based on the first information of the at least one first device, a receiving resource is allocated for the video stream sent by each of the at least one first device; The first device is a transmitting device; the second device is a receiving device.

6. The method according to claim 5, characterized in that, The first information of at least one first device receiving video communication sent by the SFU server includes: The control signaling of at least one first device receiving video communication sent by the SFU server includes first information of the corresponding first device.

7. The method according to claim 5, characterized in that, The method further includes: For each of the at least one first device, second information is determined based on the receiving resources allocated to the first device; the second information characterizes the bitrate of the video stream that the second device expects the first device to send. The second information is sent to the SFU server.

8. A first device for video communication, characterized in that, Includes a first communication interface and a first processor; The first processor is configured to acquire features of the video stream; and based on the features of the video stream and a first model, determine first information, wherein the first information characterizes the complexity of the video stream. The first communication interface is used to send the first information to the SFU server, and the first information is used to allow the second device of video communication to allocate receiving resources for the video stream sent by the first device. The first device is a transmitting device; the second device is a receiving device.

9. A second device for video communication, characterized in that, Includes a second communication interface and a second processor; The second communication interface is used to receive first information of at least one first device for video communication sent by the SFU server. The first information characterizes the complexity of the video stream sent by the first device. The first information is determined by the first device based on the characteristics of the video stream and a first model. The second processor is configured to allocate receiving resources for the video stream sent by each of the at least one first device based on the first information of the at least one first device; The first device is a transmitting device; the second device is a receiving device.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4, or the steps of the method according to any one of claims 5 to 7.

Images