Video processing method and device, electronic equipment and storage medium

Abstract

The application provides a video processing method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring stream publishing signaling information sent by a production end; acquiring stream subscription signaling information sent by a plurality of consumption ends according to the stream publishing signaling information; determining a target substream value set of the plurality of consumption ends based on the stream publishing signaling information and the stream subscription signaling information; sending the target substream value set to the production end, and enabling the production end to encode a substream set based on the target substream value set; and sending the substream set uploaded by the production end to the plurality of consumption ends. The method of the application collects the demand of the consumption end, and then controls the production end to push the stream on demand, realizes dynamic adjustment of the video stream to be published by the production end, thereby saving the bandwidth resource of the production end, reducing the power consumption loss, and improving the processing efficiency of the video stream.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to video processing methods, apparatus, electronic devices and storage media. Background Technology

[0002] Simulcast is a video conferencing technology. Its core process involves the producer encoding the captured video frames multiple times to generate multiple video streams with different resolutions and qualities. These streams are then pushed to the server, which selects a suitable stream based on the consumer's network bandwidth and user needs before forwarding it to the consumer. The producer consistently publishes multiple video streams, regardless of whether anyone has subscribed to them. Summary of the Invention

[0003] This application provides a video processing method, apparatus, electronic device, and storage medium to improve the processing efficiency of streaming media. The technical solution of this application is as follows:

[0004] In a first aspect, embodiments of this application provide a video processing apparatus, including:

[0005] The production-side acquisition module is used to acquire the streaming signaling information sent by the production end;

[0006] The consumer acquisition module is used to acquire stream subscription signaling information sent by multiple consumers based on the stream publishing signaling information;

[0007] The demand aggregation module is used to determine the target sub-stream value set of the multiple consumer terminals based on the stream publishing signaling information and the stream subscription signaling information;

[0008] The production control module is used to send the target sub-stream value set to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set;

[0009] The stream sending module is used to send the set of sub-streams uploaded by the producer to the multiple consumer ends.

[0010] Secondly, embodiments of this application provide a video processing apparatus, including:

[0011] The production-side acquisition module is used to acquire the streaming signaling information sent by the production end;

[0012] The consumer acquisition module is used to acquire stream subscription signaling information sent by multiple consumers based on the stream publishing signaling information;

[0013] The demand aggregation module is used to determine the target sub-stream value set of the multiple consumer terminals based on the stream publishing signaling information and the stream subscription signaling information;

[0014] The production control module is used to send the target sub-stream value set to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set;

[0015] The stream sending module is used to send the set of sub-streams uploaded by the producer to the multiple consumer ends.

[0016] Thirdly, embodiments of this application provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video processing method described in the first aspect of this application.

[0017] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the video processing method described in the first aspect of this application.

[0018] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the video processing method described in the first aspect of this application.

[0019] The technical solution provided in this application has at least the following beneficial effects:

[0020] By collecting consumer demand, production can be controlled to push products on demand.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0023] Figure 1 This is an architecture diagram of the SFU architecture in the Simulcast mode.

[0024] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment.

[0025] Figure 3 This is a structural diagram of an extended RTCP message according to an exemplary embodiment.

[0026] Figure 4This is a flowchart illustrating a video processing method according to another exemplary embodiment.

[0027] Figure 5 This is an architecture diagram of a system employing a video processing method, illustrated according to an exemplary embodiment.

[0028] Figure 6 This is a flowchart illustrating a video processing method according to yet another exemplary embodiment.

[0029] Figure 7 This is a flowchart illustrating a video processing method according to yet another exemplary embodiment.

[0030] Figure 8 This is a block diagram of a video processing apparatus according to an exemplary embodiment.

[0031] Figure 9 This is a block diagram of a video processing apparatus according to another exemplary embodiment.

[0032] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] SFU (Selective Forwarding Unit) is an audio / video conferencing system architecture. The SFU server is primarily responsible for collecting and forwarding video streams, without encoding or decoding the multimedia data. An SFU acts like a video stream router, receiving audio and video streams from terminals and forwarding them to other terminals as needed. SFUs are widely used in audio / video conferencing, especially after the widespread adoption of WebRTC (Web Real-Time Communication). Media servers supporting WebRTC multi-party communication are almost all based on the SFU architecture. WebRTC clients encode the same video stream multiple times using different resolutions and bitrates, then send it to the SFU router, which determines which terminal receives which video stream.

[0036] RTCP (The RTP control protocol) is a sister protocol to RTP (Real-time Transport Protocol). RTCP works in conjunction with RTP. RTP transmits the actual data, while RTCP transmits various control signals, mainly used to provide feedback on the transmission quality of RTP.

[0037] Simulcast is a technical solution for video conferencing, such as... Figure 1 As shown, the core process is that the production end encodes the acquired video frames multiple times to generate multiple video streams with different resolutions and qualities. Then, the video streams are pushed to the SFU server, which selects a suitable video stream based on the network bandwidth and user needs of the consumer end and forwards it to the consumer end.

[0038] As you can understand, Simulcast mode means that video sharers can simultaneously send multiple video streams of different resolutions (e.g., 1080P, 720P, 360P) to the SFU server. The SFU server can then select the appropriate stream from the received streams based on the specific conditions of each terminal. For example, if the computer has a particularly good network connection, a 1080P video is sent to the computer; while if the mobile network is poor, a 360P video is sent to the mobile phone.

[0039] It should be noted that in this embodiment, the "producer" refers to the end that publishes the video stream, and the "consumer" refers to the end that receives the video stream. As an example, the SFU server connects to four terminals participating in the meeting. Each terminal can send its own audio and video stream to the SFU server, and simultaneously retrieve video streams shared by the other three participants' terminals from the SFU server. In other words, the terminals used by participants to share video streams act as producers, while their terminals used to retrieve video streams from other participants act as consumers.

[0040] In related technologies, the production side publishes multiple video streams at a fixed rate, regardless of whether anyone subscribes to them. If no one subscribes to a video stream, it results in wasted bandwidth and increased power consumption on the production side.

[0041] To address the aforementioned issues, embodiments of this application provide a video processing method, apparatus, electronic device, and storage medium. By optimizing the existing Simulcast solution, the video stream to be published by the production end is dynamically adjusted according to the needs of the consumer end, thereby saving bandwidth resources on the production end, reducing power consumption loss, and improving the processing efficiency of the video stream.

[0042] On-demand video streaming can be understood as follows: for example, if the consumer only subscribes to 360p streams, then the producer will also only send 360p streams; if the consumer does not subscribe to any streams, then the producer will not send any streams.

[0043] 360P is a video display format. The letter 'p' indicates progressive scan, and the number 360 indicates its vertical resolution, which means there are 360 ​​horizontal scan lines in the vertical direction. 360 represents a vertical resolution of 360, with an aspect ratio of 4:3 (480x360) or 15:9 (600x360).

[0044] Figure 2 This is a flowchart of a video processing method according to an embodiment of this application. It should be noted that the video processing method of this application embodiment can be applied to the video processing apparatus of this application embodiment. This video processing apparatus can be configured on an electronic device. Figure 2 As shown, the video processing method may include the following steps.

[0045] In step S201, the streaming broadcast signaling information sent by the production end is obtained.

[0046] In this embodiment, the SFU server (hereinafter referred to as the server) obtains the stream publishing signaling information sent by the production end. That is, the production end publishes the stream to the server through signaling, but does not send the actual stream data.

[0047] It's understandable that the producer needs to send relevant information from the publishing end to the server first. Only then can the consumer connected to the server specify which producer's push stream to subscribe to based on this data. Therefore, the producer needs to send information, such as its own ID, to the server first. At this point, the producer is simply notifying the server via signaling that it has joined the video conference room and is connected.

[0048] Furthermore, the streaming signaling information includes the sub-stream categories provided by the production end. The production end notifies the server of the sub-stream categories it can provide, such as 1080P, 720P, and 360P. Although the production end does not upload specific streaming data, it is already collecting audio and video data in real time. This data collection by the production end is normal and unaffected by the server. The production end may be collecting video information through a camera or screen recording.

[0049] It should be noted that the production side performs streaming based on the needs of the consumer side. For example, as long as the consumer side has a camera turned on, streaming will be performed, that is, streaming signaling information will be sent.

[0050] Optionally, the signaling format for establishing a connection can be either WebSocket or HTTP (Hypertext Transfer Protocol). WebSocket is a protocol that enables full-duplex communication over a single TCP (Transmission Control Protocol) connection.

[0051] In step S202, stream subscription signaling information sent by multiple consumer terminals is obtained based on the stream publishing signaling information.

[0052] In this embodiment, the server obtains stream subscription signaling information sent by multiple consumers based on the stream publishing signaling information. The stream subscription signaling information includes the expected substream value, which is the parameter of the substream that the consumer wants to obtain, such as 720P.

[0053] The consumer subscribes to the stream from the server via signaling, meaning the consumer notifies the server to send audio and video streams to the consumer.

[0054] In step S203, a set of target sub-stream values ​​for multiple consumer terminals is determined based on the stream publishing signaling information and the stream subscription signaling information.

[0055] In this embodiment, the server determines the target sub-stream value set for multiple consumers based on stream publishing signaling information and stream subscription signaling information.

[0056] This is understandable, because the core of the video processing method in this application is on-demand production. Therefore, the server needs to first obtain the sub-stream value required by the consumer, then send the required sub-stream value to the production end for processing, and finally send the streaming data uploaded by the production end to the consumer end.

[0057] In this embodiment, the sub-stream values ​​of all consumer-side demands are aggregated into a target sub-stream value set, and the demands are sent to the production end for processing in the form of the target sub-stream value set.

[0058] In step S204, the target sub-stream value set is sent to the production end, which then encodes the sub-stream set based on the target sub-stream value set.

[0059] In this embodiment, the server sends the target sub-stream value set to the production server, which then sets up an encoder based on the target sub-stream value set to encode and obtain the sub-stream set.

[0060] It should be noted that the sub-stream set in the embodiments of this application includes one or more sub-streams.

[0061] Optionally, the server can send the target substream value set to the producer by extending the RTCP message. An example of the extended RTCP message message structure is shown below. Figure 3 As shown, the meanings of each field in the message are explained below:

[0062] V (Version): Version, fixed at 2;

[0063] P (Padding): Whether to align at the end, fixed at 0;

[0064] RC (Report Count): Number of report blocks, for example... Figure 3 The RC value in the value is 2;

[0065] PT (Payload Type): RTCP packet type, fixed at 220;

[0066] length: Total length of the RTCP packet;

[0067] Producer ID: The publisher ID. Each device may have multiple publisher streams, for example, the IDs of the front camera and the rear camera may be different.

[0068] Simulcast preferred layer: The substream level to be pushed. As an example, it is divided into 3 levels: LOW, MIDDLE, and HIGH, where LOW=1, MIDDLE=2, and HIGH=4. If multiple levels need to be pushed, an OR operation is performed. For example, if LOW and MIDDLE are pushed simultaneously, the value is 3; if LOW and HIGH are pushed simultaneously, the value is 5. 3 levels correspond to 3 substreams. The level provided here is just an example; it can also include 4 or 5 levels, which is not limited here.

[0069] Different publishers are distinguished by their publisher IDs, and consumers specify the publisher ID when sending subscription signaling. In the overall processing flow, the producer first needs to publish the publisher ID to the server via signaling, and only then can the consumer obtain the publisher ID and specify to subscribe to the stream published by that publisher ID.

[0070] For example, after receiving the extended RTCP message from the server, the production end sets the corresponding encoder for the push stream. If a LOW stream is needed, a low-quality sub-stream is encoded; if a MIDDLE stream is needed, a medium-quality sub-stream is encoded; and if a HIGH stream is needed, a high-quality sub-stream is encoded. For example, HIGH, MIDDLE, and LOW correspond to 1080P, 720P, and 360P streams, respectively.

[0071] As a scenario example, when the production end is a WebRTC client, the same video stream is encoded multiple times using different encoders to obtain multiple different sub-streams, which are then sent to the SFU server.

[0072] In step S205, the set of sub-streams uploaded by the producer is sent to multiple consumer terminals.

[0073] The video processing method of this application embodiment, based on the stream publishing signaling information issued by the production end and the stream subscription signaling information issued by the consumer end, aggregates a set of target sub-stream values ​​corresponding to the total demand of multiple consumer ends, and then sends the set of target sub-stream values ​​to the production end, instructing the production end to produce video streams according to the demand of the consumer ends, and then sends the produced video streams to the consumer ends. By collecting the demand of the consumer ends, the production end is controlled to push streams on demand, thereby dynamically adjusting the video streams to be published by the production end, saving bandwidth resources of the production end, reducing power consumption loss, and improving the processing efficiency of video streams.

[0074] Figure 4 This is a flowchart of a video processing method according to another embodiment of this application. Figure 4 As shown, the video processing method may include the following steps, and in combination with Figure 5 This embodiment will be described.

[0075] In step S401, the streaming signaling information sent by the production end is obtained.

[0076] In step S402, stream subscription signaling information sent by multiple consumer terminals is obtained.

[0077] In step S403, the bandwidth of the current consumer among multiple consumers is estimated, and the optimal sub-flow value within the bandwidth range is determined based on the sub-flow category in the flow publishing signaling information;

[0078] It's understandable that higher video stream quality requires greater bandwidth. For example, transmitting a 1080P video stream would require several megabits of bandwidth; while transmitting a 480P video stream might only require 100Kbps. Estimating bandwidth at the consumer end primarily determines whether it can support the transmission of high-quality sub-streams. If not, a lower-quality video stream is sent to that consumer.

[0079] Taking Tencent Meeting app as an example, if the terminal opens a small window, the required video stream quality is relatively low; if it opens a large window, the required video stream quality is relatively high.

[0080] Optionally, the bandwidth of the consumer can be estimated using a congestion control algorithm. The algorithm used for estimating the bandwidth of the consumer is not limited here, and other algorithms can also be selected.

[0081] Optionally, if the bandwidth is insufficient to transmit the sub-stream with the lowest quality in the sub-stream category, then the sub-stream with the lowest quality in the sub-stream category shall be taken as the optimal sub-stream value.

[0082] In other words, when the bandwidth at the consumer end is very small and the transmission conditions are not met for any sub-stream published by the producer end, the sub-stream with the lowest quality that the producer end can provide is selected as the optimal sub-stream value.

[0083] In step S404, the smaller of the current optimal substream value and the expected substream value of the consumer is taken as the target substream value of the current consumer.

[0084] In this embodiment, the server uses the smaller of the current consumer's optimal substream value and the desired substream value as the current consumer's target substream value.

[0085] In other words, the server compares the sub-stream desired by the consumer with the sub-stream suitable for bandwidth transmission, and uses the smaller sub-stream as the target sub-stream value for the consumer.

[0086] In step S405, the target substream values ​​of multiple consumer terminals are aggregated to obtain the target substream value set.

[0087] In this embodiment, the server aggregates the target sub-stream values ​​from multiple consumers to obtain a target sub-stream value set.

[0088] It's understandable that the server needs to gather all the demands from the consumer side before it can send those demands to the production side for manufacturing.

[0089] In step S406, the target sub-stream value set is sent to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set.

[0090] In step S407, the set of sub-streams uploaded by the producer is sent to multiple consumer terminals.

[0091] It should be noted that, in this embodiment, the implementation processes of steps S201-S202 and S206-S207 can be referred to the descriptions of the implementation processes of steps S101-S102 and S104-S105, respectively, and will not be repeated here.

[0092] The video processing method of this application embodiment involves the SFU server aggregating a set of target sub-stream values ​​corresponding to the total demand of multiple consumers based on the stream publishing signaling information released by the producer and the stream subscription signaling information sent by the consumer. This set of target sub-stream values ​​is then sent to the producer, which produces a video stream according to the consumer's demand and sends the produced video stream to the consumer. By collecting the consumer's demand, the SFU server controls the producer to push the stream on demand, dynamically adjusting the video stream to be published by the producer. This saves bandwidth resources, reduces power consumption, and improves the processing efficiency of the video stream. When aggregating the target sub-stream value set of multiple consumers, the SFU server comprehensively considers the expected demand and actual bandwidth transmission capacity of each consumer, comprehensively evaluating and obtaining the target sub-stream value for each consumer, ultimately resulting in a more suitable video stream for transmission to the consumer.

[0093] Figure 6 This is a flowchart of a video processing method according to another embodiment of this application. Figure 6 As shown, the video processing method may include the following steps.

[0094] In step S601, the streaming broadcast signaling information sent by the production end is obtained.

[0095] In step S602, stream subscription signaling information sent by multiple consumer terminals is obtained based on the stream publishing signaling information.

[0096] In step S603, a set of target sub-stream values ​​for multiple consumer terminals is determined based on the stream publishing signaling information and the stream subscription signaling information.

[0097] In step S604, it is periodically checked whether the set of sub-streams currently uploaded by the production end is consistent with the set of target sub-stream values.

[0098] In this embodiment, the server periodically checks whether the set of substreams currently uploaded by the producer is consistent with the target set of substream values. As an example, the frequency of the periodic check is 1 second.

[0099] It's understandable that when the producer obtains multiple sub-streams based on the target sub-stream value set sent by the server and pushes these sub-streams to the server, network issues or insufficient bandwidth on the producer might cause some sub-streams to fail to reach the server. Alternatively, the RTCP messages sent by the server might be lost, preventing the producer from receiving them. All of these situations could lead to a discrepancy between the currently uploaded sub-stream set and the target sub-stream value set on the producer. Other possible causes for this discrepancy will not be elaborated upon here.

[0100] To avoid the above situation, the server adds a periodic detection mechanism to check whether the producer has uploaded the multiple substreams corresponding to the target substream value set to the server as required. If it is detected that the producer has not uploaded the substream set required by the consumer, the server resends the target substream value set required by the consumer to the producer, so that the producer can obtain the required multiple substreams according to the target substream value set. By using the periodic detection method, it can be ensured that the server receives the corresponding substreams from the producer according to the target substream value set sent by the server.

[0101] When changes occur on the consumer side, such as when a consumer (i.e., a small window is resized to a large window during a video conference, or when a new participant is added midway through the conference, the server will obtain a new set of sub-stream values ​​based on these changes. Another function of periodic detection is to promptly detect changes in the server's target sub-stream value set and send the new target sub-stream value set to the producer side in a timely manner, thereby obtaining the corresponding new sub-stream set. Furthermore, if there are too many changes on the consumer side, requiring the sending of new target sub-stream value sets too frequently will impact server performance. Periodic detection can control the maximum frequency of sending target sub-stream value sets, keeping server performance within a certain range.

[0102] In step S605, in response to the discrepancy between the sub-stream set currently uploaded by the production end and the target sub-stream value set, the target sub-stream value set is sent to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set.

[0103] In this embodiment, when the server detects that the sub-stream set currently uploaded by the production end is inconsistent with the target sub-stream value set, it sends the target sub-stream value set to the production end, so that the production end can set up an encoder based on the target sub-stream value set to encode and obtain the sub-stream set.

[0104] It should be noted that when the server detects through periodic checks that the set of substreams currently uploaded by the production end is consistent with the set of target substream values, it will execute the following steps according to the operation procedure.

[0105] In step S606, the set of sub-streams uploaded by the producer is sent to multiple consumer terminals.

[0106] It should be noted that, in this embodiment, the implementation processes of steps S601-S603 and S606 can be found in the descriptions of the implementation processes of steps S201-S203 and S205, respectively, and will not be repeated here.

[0107] The video processing method in this application embodiment involves the SFU server collecting a set of target sub-stream values ​​corresponding to the total demand from multiple consumers based on the stream publishing signaling information released by the producer and the stream subscription signaling information sent by the consumer. This set of target sub-stream values ​​is then sent to the producer, instructing the producer to produce video streams according to the consumer's demand. The produced video streams are then sent back to the consumer. By collecting consumer demands, the SFU server controls the producer to push streams on demand, dynamically adjusting the video streams required by the producer. This saves bandwidth resources, reduces power consumption, and improves video stream processing efficiency. The SFU server uses periodic detection to ensure that it receives the corresponding sub-streams from the producer based on the target sub-stream value set sent by the server. It also promptly detects changes in the target sub-stream value set and sends them back to the producer, thus meeting consumer demands promptly and ensuring server performance.

[0108] Figure 7 This is a flowchart of a video processing method according to another embodiment of this application. Figure 7 As shown, the video processing method may include the following steps.

[0109] In step S701, the streaming broadcast signaling information sent by the production end is obtained.

[0110] In step S702, stream subscription signaling information sent by multiple consumer terminals is obtained based on the stream publishing signaling information.

[0111] In step S703, a set of target sub-stream values ​​for multiple consumer terminals is determined based on the stream publishing signaling information and the stream subscription signaling information.

[0112] In step S704, the target sub-stream value set is sent to the production end, which then encodes the sub-stream set based on the target sub-stream value set.

[0113] In step S705, based on the set of sub-streams currently uploaded by the production end, the target sub-stream corresponding to each target sub-stream value in the target sub-stream value set is determined.

[0114] It is understandable that although the server has sent the target substream value set required by the consumer to the producer, and the producer has set up the encoder to obtain the corresponding substream, there are two possibilities for the substream set currently uploaded by the producer to the server. One possibility is that the substream set currently uploaded by the producer to the server is the same as the substream set corresponding to the target substream value set required by the consumer; the other possibility is that the substream set currently uploaded by the producer to the server is different from the substream set corresponding to the target substream value set required by the consumer.

[0115] Under normal circumstances, such as when the network is working properly, the set of sub-streams currently uploaded from the production end to the server and the set of sub-streams corresponding to the target sub-stream value set required by the consumer end are mostly consistent. The production end processes video streams very quickly, and the server can send the video streams required by the consumer end in a timely manner.

[0116] There are several reasons for the second situation mentioned above. For example, the network environment at the production end may be poor at certain times, and its uplink bandwidth may be relatively low, making it unable to encode high-quality substreams and only able to encode the low-quality substreams corresponding to the target substream value set. Alternatively, due to network reasons, the server may fail to successfully send the target substream value set to the production end. In this case, the substream set currently uploaded by the production end on the server is still the substream set uploaded based on the target substream value set sent last time.

[0117] Therefore, the set of substreams currently uploaded by the producer may not be the same as the set of substreams corresponding to the target substream value set. The server needs to determine the target substream corresponding to each target substream value in the target substream value set based on the set of substreams currently uploaded by the producer, and then send the target substream to the consumer. In other words, the server selects a suitable substream from the set of substreams currently uploaded by the consumer and sends it to the consumer.

[0118] Optionally, in response to the fact that the set of sub-streams currently uploaded by the production end includes the first sub-stream corresponding to the current target sub-stream value in the set of target sub-stream values, the first sub-stream is used as the target sub-stream.

[0119] In other words, if there is a sub-stream in the current sub-stream set uploaded by the production end that corresponds to the target sub-stream value required by the consumer end, then that corresponding sub-stream will be used as the target sub-stream.

[0120] Optionally, in response to the fact that the set of sub-streams currently uploaded by the production end does not include the first sub-stream corresponding to the current target sub-stream value in the target sub-stream value set, the highest quality sub-stream among at least one sub-stream in the set of sub-streams currently uploaded by the production end whose quality is lower than the first sub-stream is taken as the target sub-stream.

[0121] In other words, when there is no substream corresponding to the target substream value required by the consumer in the current substream set uploaded by the producer, a substream is selected from the current substream set uploaded by the producer. This substream is of lower quality than the substream corresponding to the target substream value, and is the highest quality substream among all substreams of lower quality than the target substream value.

[0122] It should be noted that after the server sends a set of substreams that are inconsistent with the target substream value set to the consumer, when the producer uploads a set of substreams that are consistent with the target substream value set to the server, the server immediately sends the new set of substreams to the consumer to meet the consumer's needs in a timely manner.

[0123] In step S706, the target sub-stream corresponding to the target sub-stream value is sent to the consumer end corresponding to the target sub-stream value.

[0124] In this embodiment, the consumer sends the target substream corresponding to the target substream value to the consumer corresponding to the target substream value.

[0125] It should be noted that, in this embodiment, the implementation process of steps S701-S704 can be referred to the description of the implementation process of steps S201-S204, and will not be repeated here.

[0126] The video processing method in this application embodiment involves the SFU server aggregating a set of target sub-stream values ​​corresponding to the total demand from multiple consumers based on the stream publishing signaling information released by the producer and the stream subscription signaling information sent by the consumer. This set of target sub-stream values ​​is then sent to the producer, instructing the producer to produce a video stream according to the consumer's demand. The produced video stream is then sent to the consumer. By collecting consumer demands, the SFU server controls the producer to push streams on demand, dynamically adjusting the video streams required by the producer. This saves bandwidth resources, reduces power consumption, and improves video stream processing efficiency. If the SFU server does not receive the required sub-stream from the producer in a timely manner, it selects a suitable sub-stream from the current sub-stream set and sends it to the consumer, promptly fulfilling the consumer's application needs.

[0127] Figure 8 This is a block diagram illustrating a video processing apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The video processing device may include: a production-end acquisition module 801, a consumer-end acquisition module 802, a demand aggregation module 803, a production control module 804, and a stream transmission module 805.

[0128] Specifically, the production end acquisition module 801 is used to acquire the streaming release signaling information sent by the production end;

[0129] The consumer acquisition module 802 is used to acquire stream subscription signaling information sent by multiple consumers based on stream publishing signaling information;

[0130] The demand aggregation module 803 is used to determine the set of target sub-stream values ​​for multiple consumers based on stream publishing signaling information and stream subscription signaling information.

[0131] The production control module 804 is used to send the target sub-stream value set to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set.

[0132] The stream sending module 805 is used to send the set of sub-streams uploaded by the producer to multiple consumers.

[0133] The video processing apparatus of this application embodiment, based on the stream publishing signaling information released by the production end and the stream subscription signaling information sent by the consumer end, aggregates a set of target sub-stream values ​​corresponding to the total demand of multiple consumer ends, and then sends the set of target sub-stream values ​​to the production end, instructing the production end to produce video streams according to the demand of the consumer ends, and then sends the produced video streams to the consumer ends. By collecting the demand of the consumer ends and controlling the production end to push streams on demand, the apparatus dynamically adjusts the video streams to be published by the production end, thereby saving bandwidth resources of the production end, reducing power consumption, and improving the processing efficiency of video streams.

[0134] Figure 9 This is a block diagram illustrating a video processing apparatus according to another exemplary embodiment. (Refer to...) Figure 9 The video processing device may include: a production end acquisition module 901, a consumer end acquisition module 902, a demand aggregation module 903, a production control module 904, and a stream transmission module 905.

[0135] It should be noted that the production-end acquisition module 901, consumer-end acquisition module 902, demand aggregation module 903, production control module 904, and stream sending module 905 in this embodiment have the same structure and function as the production-end acquisition module 801, consumer-end acquisition module 802, demand aggregation module 803, production control module 804, and stream sending module 805, respectively.

[0136] In some embodiments of this application, the stream publishing signaling information includes the sub-stream category provided by the producer, the stream subscription signaling information includes the expected sub-stream value, and the demand aggregation module 903 includes:

[0137] The estimation unit 9031 is used to estimate the bandwidth of the current consumer among multiple consumers and determine the optimal sub-stream value within the bandwidth range based on the sub-stream category.

[0138] The comparison unit 9032 is used to take the smaller of the current optimal substream value and the expected substream value at the consumer end as the target substream value at the current consumer end;

[0139] The aggregation unit 9033 is used to aggregate the target sub-stream values ​​of multiple consumer terminals to obtain a target sub-stream value set.

[0140] In some embodiments of this application, the estimation unit 9031 is further configured to:

[0141] In response to insufficient bandwidth to transmit the lowest quality sub-stream in the sub-stream category, the lowest quality sub-stream is selected as the optimal sub-stream value.

[0142] In some embodiments of this application, the production control module 904 includes:

[0143] The timed detection unit 9041 is used to periodically detect whether the set of sub-streams currently uploaded by the production end is consistent with the set of target sub-stream values;

[0144] The production end sending unit 9042 is used to respond to the inconsistency between the sub-stream set currently uploaded by the production end and the target sub-stream value set, and send the target sub-stream value set to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set.

[0145] In some embodiments of this application, the streaming module 905 includes:

[0146] The sub-stream acquisition unit 9051 is used to determine the target sub-stream corresponding to each target sub-stream value in the target sub-stream value set based on the sub-stream set currently uploaded by the production end;

[0147] The consumer-side sending unit 9052 is used to send the target substream corresponding to the target substream value to the consumer-side corresponding to the target substream value.

[0148] In some embodiments of this application, the sub-stream acquisition unit 9051 is specifically used for:

[0149] In response to the fact that the set of sub-streams currently uploaded by the production end includes the first sub-stream corresponding to the current target sub-stream value in the set of target sub-stream values, the first sub-stream is taken as the target sub-stream.

[0150] In some embodiments of this application, the substream acquisition unit 9051 is further configured to:

[0151] In response to the fact that the set of substreams currently uploaded by the production end does not include the first substream corresponding to the current target substream value in the target substream value set, the highest quality substream among at least one substream in the set of substreams currently uploaded by the production end whose quality is lower than the first substream is taken as the target substream.

[0152] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0153] The video processing apparatus of this application first collects the demands of the consumer end, and then controls the production end to push the stream on demand, thereby dynamically adjusting the video stream to be released by the production end, saving bandwidth resources of the production end, reducing power consumption loss, and improving the processing efficiency of the video stream.

[0154] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0155] like Figure 10 The diagram shown is a block diagram of an electronic device for implementing a video processing method according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0156] like Figure 10 As shown, the electronic device includes one or more processors 1001, a memory 1002, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take processor 1001 as an example.

[0157] The memory 1002 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the video processing method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the video processing method provided in this application.

[0158] Memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the video processing method in the embodiments of this application (e.g., attached...). Figure 8 The shown modules are: production-side acquisition module 801, consumer-side acquisition module 802, demand aggregation module 803, production control module 804, and stream sending module 805, or, supplemented by... Figure 9 The processor 1001 includes a production-side acquisition module 901, a consumer-side acquisition module 902, a demand aggregation module 903, a production control module 904, and a stream transmission module 905. The processor 1001 executes various server functions by running non-transient software programs, instructions, and modules stored in the memory 1002.

[0159] The memory 1002 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the video processing electronic device. Furthermore, the memory 1002 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include memory remotely located relative to the processor 1001, and these remote memories can be connected to the video processing electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] The electronic device for implementing the video processing method may further include an input device 1003 and an output device 1004. The processor 1001, memory 1002, input device 1003, and output device 1004 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0161] Input device 1003 can receive input digital or character information, and generate key signal inputs related to user settings and function control of the video processing electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1004 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0162] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0165] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0166] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0167] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the above-described method.

[0168] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0169] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only.

[0170] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Obtain streaming signaling information sent by the production end; The stream publishing signaling information includes the sub-stream category provided by the production end; Based on the stream publishing signaling information, obtain stream subscription signaling information sent by multiple consumer terminals; The stream subscription signaling information includes the expected substream value; Based on the stream publishing signaling information and the stream subscription signaling information, a target sub-stream value set for the plurality of consumers is determined, including: estimating the bandwidth of the current consumer among the plurality of consumers, and determining the optimal sub-stream value within the bandwidth range based on the sub-stream category; taking the smaller of the optimal sub-stream value and the expected sub-stream value of the current consumer as the target sub-stream value of the current consumer; and aggregating the target sub-stream values ​​of the plurality of consumers to obtain a target sub-stream value set. The target sub-stream value set is sent to the production end, which encodes the sub-stream set based on the target sub-stream value set. The production end periodically checks whether the sub-stream set currently uploaded by the production end is consistent with the target sub-stream value set. In response to the inconsistency between the sub-stream set currently uploaded by the production end and the target sub-stream value set, the target sub-stream value set is sent to the production end, which encodes the sub-stream set based on the target sub-stream value set. Sending the set of substreams uploaded by the producer to the plurality of consumers includes: determining a target substream corresponding to each target substream value in the target substream value set based on the set of substreams currently uploaded by the producer; and sending the target substream corresponding to the target substream value to the consumer corresponding to the target substream value.

2. The method according to claim 1, characterized in that, The step of estimating the bandwidth of the current consumer among the plurality of consumers and determining the optimal sub-stream value within the bandwidth range based on the sub-stream category further includes: In response to the fact that the bandwidth is insufficient to transmit the sub-stream with the lowest quality in the sub-stream category, the sub-stream with the lowest quality is taken as the optimal sub-stream value.

3. The method according to claim 1, characterized in that, The step of determining the target sub-stream corresponding to each target sub-stream value in the target sub-stream value set based on the sub-stream set currently uploaded by the production end includes: In response to the fact that the set of sub-streams currently uploaded by the production end includes the first sub-stream corresponding to the current target sub-stream value in the set of target sub-stream values, the first sub-stream is taken as the target sub-stream.

4. The method according to claim 3, characterized in that, The step of determining the target sub-stream corresponding to each target sub-stream value in the target sub-stream value set based on the sub-stream set currently uploaded by the production end further includes: In response to the fact that the set of sub-streams currently uploaded by the production end does not include the first sub-stream corresponding to the current target sub-stream value in the target sub-stream value set, the highest quality sub-stream among at least one sub-stream in the set of sub-streams currently uploaded by the production end whose quality is lower than the first sub-stream is selected as the target sub-stream.

5. A video processing apparatus, characterized in that, include: The production-side acquisition module is used to acquire the streaming broadcast signaling information sent by the production end; The stream publishing signaling information includes the sub-stream category provided by the production end; The consumer acquisition module is used to acquire stream subscription signaling information sent by multiple consumers based on the stream publishing signaling information; The stream subscription signaling information includes the expected substream value; The demand aggregation module is used to determine the target sub-stream value set of the multiple consumer terminals based on the stream publishing signaling information and the stream subscription signaling information; The production control module is used to send the target sub-stream value set to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set; The stream sending module is used to send the set of substreams uploaded by the production end to the multiple consumer ends. The production control module also includes: A timed detection unit is used to periodically detect whether the set of sub-streams currently uploaded by the production end is consistent with the set of target sub-stream values; A production-end sending unit is configured to, in response to a discrepancy between the sub-stream set currently uploaded by the production end and the target sub-stream value set, send the target sub-stream value set to the production end, so that the production end encodes the sub-stream set based on the target sub-stream value set; The demand aggregation module includes: An estimation unit is used to estimate the bandwidth of the current consumer among the plurality of consumer terminals, and to determine the optimal sub-stream value within the bandwidth range based on the sub-stream category; The comparison unit is used to take the smaller of the optimal substream value and the expected substream value of the current consumer as the target substream value of the current consumer. The aggregation unit is used to aggregate the target sub-stream values ​​of the multiple consumer terminals to obtain a target sub-stream value set. The stream sending module includes: The sub-stream acquisition unit is used to determine the target sub-stream corresponding to each target sub-stream value in the target sub-stream value set based on the sub-stream set currently uploaded by the production end. The consumer-end sending unit is used to send the target sub-stream corresponding to the target sub-stream value to the consumer end corresponding to the target sub-stream value.

6. The apparatus according to claim 5, characterized in that, The estimation unit is further configured to: In response to the fact that the bandwidth is insufficient to transmit the sub-stream with the lowest quality in the sub-stream category, the sub-stream with the lowest quality is taken as the optimal sub-stream value.

7. The apparatus according to claim 5, characterized in that, The sub-stream acquisition unit is specifically used for: In response to the fact that the set of sub-streams currently uploaded by the production end includes the first sub-stream corresponding to the current target sub-stream value in the set of target sub-stream values, the first sub-stream is taken as the target sub-stream.

8. The apparatus according to claim 7, characterized in that, The sub-stream acquisition unit is further configured to: In response to the fact that the set of sub-streams currently uploaded by the production end does not include the first sub-stream corresponding to the current target sub-stream value in the target sub-stream value set, the highest quality sub-stream among at least one sub-stream in the set of sub-streams currently uploaded by the production end whose quality is lower than the first sub-stream is selected as the target sub-stream.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the video processing method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the video processing method according to any one of claims 1 to 4.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

Images