Video conference processing method, system and related device

Abstract

The invention relates to the technical field of video communication, and discloses a video conference processing method, system and related device, and the method comprises the steps: a distribution server obtains video source information of each participating terminal, and broadcasts the video source information to other participating terminals; the distribution server obtains subscription information of each participating terminal; the distribution server sends a corresponding video stream pushing request to the subscribed end according to the supply video source type in the subscription information, and receives a video stream sent by the subscribed end; if the coding formats supported by the subscribing ends cannot be supported by the subscribing ends with the subscribing relationship, or the coding formats supported by the corresponding subscribing ends are not the formats with the lowest propagation bandwidth occupation, the distribution server controls the graphic acceleration card to transcode the received video streams into the coding formats with the lowest propagation bandwidth occupation supported by the subscribing ends; and the distribution server distributes the video streams to the corresponding subscription ends according to the subscription information. According to the method, the bandwidth pressure of the server can be reduced, and the number of participants is increased.

Description

Technical Field

[0001] This invention relates to the field of video communication technology, and specifically to a video conferencing processing method, system, and related apparatus. Background Technology

[0002] Audio and video conferencing, due to its excellent communication efficiency and effectiveness, has become an indispensable tool in modern work and life, widely used in scenarios such as healthcare, education, and firefighting. With the development of hardware technology, the clarity of video captured by various participating terminals has also greatly improved, leading to a significant increase in the computing power and bandwidth pressure on servers and terminal devices. Furthermore, different participating terminals may support different encoding formats, requiring the server to transcode different formats according to subscription relationships, further increasing the server's computing power burden. Additionally, some participating terminals support encoding formats with low compression rates and high transmission bandwidth occupancy; when the video streams from such terminals are widely subscribed to, this also significantly increases the server's bandwidth pressure. Therefore, a new video conferencing processing solution is needed to overcome these technical problems. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a video conferencing processing method that can reduce the computing power burden and bandwidth pressure on the server, ensure the smoothness of the video conference, and increase the acceptable number of participating terminals.

[0004] To solve the above problems, the technical solution adopted by the present invention is as follows: A video conferencing processing method, comprising: The distribution server obtains the video source information of each participating terminal and broadcasts the video source information to other participating terminals. The video source information includes the types of video sources that each participating terminal can provide, the types of video sources that can be viewed, and the supported encoding formats. The distribution server obtains the subscription information of each participating terminal. The subscription information includes the subscription relationship between the subscriber and the subscribed terminal, as well as the type of video source provided by the subscribed terminal that the subscriber needs to watch. The distribution server sends the corresponding video push request to the subscribed end based on the type of video source provided in the subscription information, and receives the video stream sent by the subscribed end; If the encoding format supported by the subscriber is not supported by the subscriber with whom it has a subscription relationship, or is not the encoding format with the lowest propagation bandwidth among the encoding formats supported by the corresponding subscriber, the distribution server sends the video stream of the subscriber to the graphics accelerator card, and the graphics accelerator card transcodes the received video stream into the encoding format with the lowest propagation bandwidth supported by the subscriber. The distribution server distributes each video stream to the corresponding subscriber based on the subscription relationship in the subscription information.

[0005] Compared to existing technologies, the advantages of this invention are as follows: This method uses a graphics accelerator card to uniformly transcode video streams that require transcoding, reducing the computational burden on the distribution server and allowing it to focus more on video stream distribution, thus ensuring the smoothness of video conferencing. For encoding formats with low compression rates and high transmission bandwidth consumption provided by subscribed clients, the graphics accelerator card uniformly converts them to encoding formats with higher compression rates supported by the subscribed clients, which are then reused and distributed by the distribution server to the corresponding subscribed clients. This significantly reduces the bandwidth requirements for the distribution server to send video streams, allowing the server to connect more participating terminals. Furthermore, this method requests video streams from subscribed clients based on subscription relationships, rather than directly requesting video streams from all participating terminals, avoiding bandwidth consumption by video streams from unsubscribed participating terminals and further reducing the server's bandwidth requirements.

[0006] The aforementioned video conferencing processing methods also include: If the difference between the maximum resolution of the available video source type on the subscribed end and the minimum resolution of the viewable video source type on the subscribed end with a subscription relationship is greater than a preset threshold, the distribution server will send the video stream from the subscribed end to the graphics accelerator card, and the graphics accelerator card will improve the resolution of the received video stream through a super-resolution model.

[0007] In the above-described video conferencing processing method, in the steps of the distribution server sending a corresponding video streaming request to the subscribed end based on the supply video source type in the subscription information and receiving the video stream sent by the subscribed end, if multiple supply video source types of the subscribed end are subscribed to by multiple subscribing ends, the distribution server requests the subscribed end to push the video stream of the supply video source type with the highest resolution among the subscribed types, and requires the subscribed end to perform multi-resolution layer encoding on the video stream; in the step of the distribution server distributing each video stream to the corresponding subscribing end according to the subscription relationship in the subscription information, the distribution server sends the corresponding resolution layer to the corresponding subscribing end according to the resolution of the video stream subscribed to by the subscribing end.

[0008] The aforementioned video conferencing processing methods also include: The distribution server sends the received video stream to the graphics accelerator card, which then divides the video frames of the received video stream into multiple time layers with different frame rates. The distribution server monitors the real-time transmission rate with each participating terminal; In the step where the distribution server distributes each video stream to the corresponding subscriber based on the subscription relationship in the subscription information, the distribution server sends the time layer required for the number of frames corresponding to the real-time transmission rate to the corresponding subscriber based on the mapping relationship between the transmission rate and the frame rate.

[0009] The aforementioned video conferencing processing methods also include: The distribution server sends the received video stream to the graphics accelerator card, which uses the ROI recognition model to identify the region of interest in each frame of the video stream. The distribution server monitors the real-time transmission rate with each participating terminal; If the real-time transmission rate is less than the preset rate threshold, the distribution server will remove information from the non-interest areas in the high-resolution layer of the video stream.

[0010] The aforementioned video conferencing processing methods also include: The distribution server obtains user interaction information from each participating terminal; The distribution server inputs user interaction information into a pre-trained subscription prediction model to obtain predicted subscription relationships; The predicted subscription relationships are compared with the current subscription relationships to obtain new subscription relationships; The distribution server sends a minimum resolution video streaming request to the subscribed end in the newly added subscription relationship, and distributes the video stream to the corresponding subscriber based on the newly added subscription relationship.

[0011] A video conferencing system includes participating terminals and a conferencing server. The conferencing server includes a distribution server and a graphics accelerator card. The participating terminals communicate with the distribution server via a network, and the distribution server is electrically connected to the graphics accelerator card. The distribution server is used to obtain video source information and subscription information of each participating terminal, broadcast the video source information to other participating terminals, send corresponding video stream requests to subscribed terminals according to the available video source types in the subscription information, and distribute the video streams to subscribed terminals according to the subscription relationship. The graphics accelerator card is used to transcode the video stream sent by the distribution server to the encoding format with the lowest propagation bandwidth supported by the subscribed terminal when the encoding format supported by the subscribed terminal is not supported by the subscribed terminals with a subscription relationship, or is not the encoding format with the lowest propagation bandwidth among the encoding formats supported by the corresponding subscribed terminals, under the control of the distribution server. The video source information includes the available video source types, viewable video source types, and supported encoding formats of each participating terminal. The subscription information includes the subscription relationship between the subscribing terminal and the subscribed terminal, and the available video source types of the subscribed terminal that the subscribing terminal needs to view.

[0012] In the aforementioned video conferencing system, the graphics accelerator card includes a decoding / encoding module and a super-resolution module. The decoding / encoding module is used to decode and encode the received video stream under the control of the distribution server, thereby realizing the transcoding of the video stream. The super-resolution module is used to improve the resolution of the video stream of the subscribed end by means of a super-resolution model when the difference between the maximum resolution of the available video source type on the subscribed end and the minimum resolution of the viewable video source type on the subscribed end with a subscription relationship is greater than a preset threshold.

[0013] A computer device includes a processor, a memory, and a bus. The memory stores a computer program, and the processor communicates with the memory via the bus. When the computer program is executed by the processor, it implements the steps of the video conferencing processing method described above.

[0014] A storage medium storing a computer program, which, when executed by a processor, implements the steps of the video conferencing processing method described above.

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a video conferencing processing method according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the multi-resolution hierarchical encoding process according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the time layer division process in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the video conferencing system according to an embodiment of the present invention.

[0020] Figure 5 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention are described in detail below, with reference to... Figure 1 The present invention provides a video conferencing processing method, comprising the following steps: The distribution server obtains the video source information of each participating terminal and broadcasts the video source information to other participating terminals. The video source information includes the types of video sources that each participating terminal can provide, the types of video sources that can be viewed, and the supported encoding formats. The distribution server obtains the subscription information of each participating terminal. The subscription information includes the subscription relationship between the subscriber and the subscribed terminal, as well as the type of video source provided by the subscribed terminal that the subscriber needs to watch. The distribution server sends the corresponding video push request to the subscribed end based on the type of video source provided in the subscription information, and receives the video stream sent by the subscribed end; If the encoding format supported by the subscribed end is different from the encoding format supported by the subscriber with whom it has a subscription relationship, or is not the encoding format with the lowest propagation bandwidth among the encoding formats supported by the corresponding subscriber, the distribution server sends the video stream of the subscribed end to the graphics accelerator card, and the graphics accelerator card transcodes the received video stream into the encoding format with the lowest propagation bandwidth supported by the subscriber. The distribution server distributes each video stream to the corresponding subscriber based on the subscription relationship in the subscription information.

[0022] This method uses a graphics accelerator card to transcode the video stream, avoiding the computational burden on the distribution server and ensuring the stability of the distribution process, thus guaranteeing smooth video conferencing. The graphics accelerator card uniformly transcodes the video stream into an encoding format supported by the subscribers, which consumes less bandwidth, before the distribution server reuses and distributes it to the corresponding subscribers. This reduces bandwidth consumption and improves bandwidth utilization, allowing the system to connect more participants. Since each participating terminal typically displays only the screens of a few other participants, some terminals may not have their video sources subscribed to by other terminals. If these unsubscribed terminals continue to push video streams, it would waste bandwidth. Therefore, this method also obtains the subscription relationship first and then requests video streams from subscribed terminals based on the subscription relationship, preventing unsubscribed terminals from pushing streams to the server and consuming bandwidth, further avoiding bandwidth waste.

[0023] It is understandable that participating terminals can be smartphones, personal computers, high-definition surveillance cameras, ordinary cameras, or drone controllers, etc. After receiving video source information from other participating terminals broadcast by the distribution server, each participating terminal determines the resolution of the video sources it needs to subscribe to based on its own screen resolution and the window size of different users on the screen. If the highest resolution among the available video source types on the subscribed terminal is lower than the resolution required by the subscribing terminal, then it subscribes to the highest resolution among the available video source types. For example, if user A's participating terminal is a personal computer with a 1080P monitor resolution, and the conference interface can display a main window and a nine-grid group of smaller windows, then user A's participating terminal can subscribe to a 720P video source type from the participating terminal corresponding to the user displayed in the main window, and subscribe to a 360P video source type from the participating terminals of the nine users displayed in the nine-grid group of smaller windows.

[0024] After receiving subscription information from each participating terminal, the distribution server integrates the types of video sources subscribed to by all subscribed terminals and sends corresponding video streaming requests to the subscribed terminals based on the subscribed video source types, requesting the participating terminals to stream video streams at the appropriate resolution. If, after a subscription update, it is found that some participating terminals currently streaming are no longer subscribed to by any other participating terminals, video streaming requests are stopped being sent to the unsubscribed participating terminals, thus halting video streaming on the unsubscribed terminals and conserving bandwidth resources.

[0025] The distribution server's memory can store a bandwidth usage score for each encoding format. A higher score indicates a higher bandwidth usage. If a participating terminal's video stream encoding format is subscribed to by multiple participating terminals that support it with lower scores, such as a participating terminal that can only provide 2K-4K H264 encoded video streams, and its video stream is subscribed to by multiple participating terminals that support H265 encoding, the distribution server will transmit the participating terminal's video stream to the graphics accelerator card. The graphics accelerator card will then transcode it into an H265 encoded video stream, and the distribution server will then multiplex and distribute the transcoded video stream to the corresponding subscribers. In practice, since most terminal devices often support multiple encoding formats, the encoding format supported by multiple participating terminals that have subscribed to the same video source may not be the one with the lowest propagation bandwidth among the encoding formats supported by some of the participating terminals. Transcoding also requires significant computing power. In this case, converting all video streams to the encoding format with the lowest propagation bandwidth supported by the corresponding subscriber would significantly increase the system's computing cost. To reduce bandwidth usage while avoiding excessive computing power consumption, some embodiments require listing all transcoding strategies based on subscription relationships to form a transcoding strategy space. Then, optimization algorithms such as ant colony optimization or greedy selection algorithms are used to find the transcoding strategy with the lowest overall cost in the transcoding strategy space, and the video stream is transcoded based on the transcoding strategy with the highest return. The overall cost can be the weighted sum of the bandwidth cost and computing power cost after transcoding. The weights of bandwidth cost and computing power cost can be dynamic. The weight coefficient of bandwidth cost can be proportional to the total bandwidth load rate of the system, so that the weight coefficient of bandwidth cost can be increased when the network is busy. This allows the optimization algorithm to find transcoding strategies that reduce bandwidth consumption. The weight coefficient of computing power cost can be proportional to the utilization rate of the graphics accelerator card, so that the weight coefficient of computing power cost can be increased when the graphics accelerator card is under high load. This allows the optimization algorithm to find transcoding strategies with lower computing power requirements.

[0026] In practice, some emergency meetings, such as rescue meetings, only have ordinary sampling equipment at the rescue site, while the sampled video source needs to be played on a large screen in the dispatch center. Forcing a low-resolution video stream onto the large screen would result in extremely poor display quality, affecting the meeting's effectiveness. To solve this technical problem, in this embodiment, if the difference between the maximum resolution of the available video source type on the subscribed end and the minimum resolution of the viewable video source type on the subscribed end is greater than a preset threshold, the distribution server sends the video stream from the subscribed end to the graphics accelerator card. The super-resolution model in the graphics accelerator card performs anti-blur distortion, intelligent noise reduction, or artifact removal on the video stream, converting it to a higher resolution to improve its display effect on the large screen in the dispatch center. In this embodiment, the difference can be measured by calculating the ratio of the minimum resolution of the viewable video source type on the subscribed end to the maximum resolution of the available video source type on the subscribed end. If the ratio is greater than 9 times, the video stream needs to be sent to the graphics accelerator card for super-resolution processing. A super-resolution model can be a neural network model trained on a training set consisting of a large number of high-resolution original images and lower-resolution images obtained by downsampling.

[0027] In practice, participating terminals often provide multiple video source types with different resolutions, and these resolutions may be subscribed to by multiple different participating terminals. This results in participating terminals needing to push multiple resolution video streams to the distribution server, which not only increases the transcoding pressure on the participating terminals but also increases the bandwidth burden on the server. In this embodiment, to further reduce the server's bandwidth burden, when multiple subscribers subscribe to multiple video source types with different resolutions, the distribution server requests the subscriber to push the video stream of the highest resolution video source type among the subscribed types. The subscriber is also required to use scalable video coding technology to perform multi-resolution layer encoding on the video stream, encoding the video stream into multiple spatial layers L0, L1 to LN corresponding to different resolutions. L0 is the basic spatial layer, which can be decoded into a complete video stream on its own, but the decoded video stream has a lower resolution, corresponding to a lower resolution. L1 to LN are all enhancement spatial layers, corresponding to higher resolutions in sequence. Each enhancement spatial layer reuses common texture and motion information with the lower resolution enhancement layer and the basic spatial layer to avoid duplicate encoding. When distributing video streams to different subscribers, the distribution server can send the corresponding number of layers based on the resolution of the video source subscribed to by each subscriber. For example, if a participating terminal can provide three video source types—360P, 720P, and 1080P—and all three types are subscribed to by other participating terminals, the distribution server requires that participating terminal to encode the video stream into three layers: L0, L1, and L2. L0 corresponds to 360P, L1 to 720P, and L2 to 1080P. The distribution server only sends layer L0 to subscribers subscribed to 360P, sends layers L0 and L1 to subscribers subscribed to 720P, and sends all three layers to subscribers subscribed to 1080P. By performing multi-resolution layer encoding on the video stream, multiple video sources with different resolutions can be supplied with a single encoding, reducing the encoding burden on participating terminals. Furthermore, because different spatial layers reuse common texture and motion information, the encoded data packets are only slightly larger than those of ordinary encoding. Compared to encoding and streaming different resolutions separately, this reduces bandwidth consumption.

[0028] Understandably, participating terminals can achieve spatial layering of the video stream using two-dimensional wavelet transform or downsampling. Two-dimensional wavelet transform generates different wavelet sub-bands by performing multiple levels of two-dimensional wavelet transform on the original video frames. The number of wavelet transform levels corresponds to the number of spatial layers, and different wavelet sub-bands correspond to different video resolutions. Downsampling methods are as follows... Figure 2As shown, by progressively downsampling the original video frames, video frames of different resolutions are obtained. During encoding, the lower-resolution video frames are used to perform inter-layer prediction on the higher-resolution video frames, thereby reducing the residual data that needs to be encoded in the high-resolution layers and improving the compression rate of the video stream. Finally, the layers with different resolutions are encoded separately to obtain a bitstream containing multiple different resolution layers.

[0029] In practice, bandwidth may fluctuate due to network conditions. To avoid stuttering and screen tearing when network conditions are poor, in this embodiment, the distribution server sends the video stream to the graphics accelerator card. The graphics accelerator card divides the received video stream into multiple time layers T0, T1 to T2 with different frame rates. N T0 is the base time layer, corresponding to a lower frame rate, and T1 to T... N These are all enhanced time layers, corresponding sequentially to higher frame rates. Video frames in high frame rate time layers are dependent on those in low frame rate time layers, and can be decoded in conjunction with the basic time layer and the low frame rate enhanced time layer to obtain higher frame rate video. The distribution server monitors the real-time transmission rate with each participating terminal and, during the video stream distribution process, sends the required number of time layers corresponding to the real-time transmission rate to the corresponding subscribers based on the mapping relationship between the real-time transmission rate and the frame rate. That is, when the distribution server detects a decrease in transmission rate, it only sends the basic time layer or the basic time layer and the lower frame rate enhanced time layer, reducing the amount of data in the transmitted data stream and lowering the video frame rate to avoid stuttering and screen tearing, ensuring smooth video playback. When the transmission rate recovers, it transmits all time layers in full, providing a high frame rate, high-quality video stream. The distribution server's memory stores the range of transmission rates corresponding to different frame rates. The distribution server queries the corresponding video frame rate based on the real-time transmission rate and selects the corresponding time layer to send to the subscribers based on the corresponding video frame rate. For example, if T0 corresponds to 24 frames per second, T1 corresponds to 45 frames per second, and T2 corresponds to 60 frames per second, when a video of 24 frames per second is required, the distribution server only sends time layer T0; when a video of 45 frames per second is required, the distribution server sends time layers T0 and T1; when a video of 60 frames per second is required, the distribution server sends time layers T0, T1, and T2.

[0030] Understandably, graphics accelerator cards can divide a video stream into multiple temporal layers with different frame rates using either Motion Compensating Temporal Filter (MCTF) or graded B-frame prediction techniques. MCTF performs wavelet-based lifting on the video frame sequence at a temporal temperature, incorporating motion estimation during the lifting process to improve video compression efficiency. The wavelet lifting process generates a low-frequency sub-band and multiple high-frequency sub-bands; the low-frequency sub-band serves as the base temporal layer, while the high-frequency sub-bands serve as enhancement temporal layers for each layer. Graded B-frame prediction techniques, on the other hand, involve... Figure 3 As shown, video frames in each image group are evenly divided into multiple time layers according to a certain pattern. During the encoding process, upper-layer video frames use neighboring lower-layer video frames as reference frames for prediction, thus creating a dependency between upper-layer and lower-layer video frames. During encoding, each higher-layer video frame can select one preceding and one following lower-layer video frame as reference frames, such as... Figure 3 As shown in Figures (a) and (b), this method aims to improve coding efficiency, but it also requires selecting one frame before and one frame after the previous frame as a reference during decoding, resulting in a significant decoding delay. Higher-level video frames can also be predicted by selecting only one previous lower-level video frame as a reference frame, such as... Figure 3 As shown in Figure (c), this encoding method has relatively low efficiency but fast decoding time and low latency. Therefore, in practice, the distribution server can choose to use either two-frame prediction or single-frame prediction encoding to encode the video stream based on the bandwidth load. If the bandwidth load is higher than a preset load threshold, the two-frame prediction method is used to divide the video stream into time layers; when the bandwidth load is lower than the preset load threshold, the single-frame prediction encoding method is used to encode the video stream to reduce video latency and improve video real-time performance.

[0031] In some embodiments, to further improve video smoothness during bandwidth fluctuations while ensuring the quality of key elements in the video and thus guaranteeing the transmission of critical information, the distribution server sends the received video stream to a graphics accelerator card. The graphics accelerator card identifies the Region of Interest (ROI) in each frame of the video stream using an ROI recognition model. When the real-time transmission rate between the distribution server and the participating terminals is less than a preset rate threshold, the distribution server removes information from the non-ROI regions in the high-resolution augmented spatial layer of the video stream. This further improves the compression rate of the video stream, reduces its bandwidth requirements, and ensures transmission efficiency and smoothness. Since only the non-ROI regions in the high-resolution augmented spatial layer are removed, deeper information about visually noticeable points such as faces, text, or vehicles is preserved. This further compresses the video stream's data volume while allowing the decoded video to be displayed at different resolutions in different areas. Important areas such as human figures or text are displayed at higher resolutions, while less important areas such as backgrounds or walls are displayed at lower resolutions. This ensures video transmission efficiency while preventing the transmission of important messages in the video. ROI recognition models can be obtained by training a lightweight neural network model on a training set consisting of a large number of manually indexed images of visually important regions such as faces, text, or flames.

[0032] In practice, participating users may adjust the number and type of other participating users displayed on their terminal's screen according to the meeting's progress or their own needs, leading to modifications in subscription relationships. In this embodiment, to improve the video loading speed of newly displayed subscribed terminals after a participating user modifies the meeting interface, the distribution server collects user interaction information generated by each user's actions on their terminal and inputs this information into a pre-trained subscription prediction model. The subscription prediction model then predicts possible future changes in subscription relationships based on the input user interaction information and outputs the predicted subscription relationships. The distribution server compares the predicted subscription relationships with the current subscription relationships to identify newly added subscription relationships and requests the lowest resolution video source type that the subscribed terminals in these new subscription relationships can provide for streaming. Based on the new subscription relationships, the distribution server distributes these lowest resolution video streams to the corresponding subscribed terminals to preload the video feeds of potentially newly added participating users, preventing black screens of newly displayed video feeds of other participating users after user actions. If subsequent user actions on the video conferencing interface change the subscription relationship, thus confirming some of the predicted new subscription relationships, the distribution server will then subscribe to the video stream of the subscribed end based on the modified actual subscription relationship. It will then request the video stream at the resolution actually needed by the subscribed end and replace the pre-loaded low-resolution video stream. If, after prediction, the user does not interact with the conference interface within a preset time window (e.g., 30 seconds) causing a change in the subscription relationship, the distribution server will stop requesting the lowest resolution stream from the subscribed end in the new subscription relationship and will continue to request streams according to the actual subscription relationship. It is understandable that collecting user interaction information can include mouse actions, UI actions, and touchscreen positions. A neural network model is trained using a training set comprised of previously collected historical user interaction information and changes in subscription relationships after interactions to obtain the subscription prediction model. During system operation, user interaction information and changes in subscription relationships after interactions can be continuously collected and recorded to form interaction subscription records. The subscription prediction model can then periodically learn from newly added interaction subscription records to improve the model's accuracy in predicting changes in subscription relationships.

[0033] Based on the same inventive concept, referring to Figure 4This invention also provides a video conferencing system, including participating terminals and a conferencing server. The conferencing server includes a distribution server and a graphics accelerator card. Participating terminals communicate with the distribution server via a network, and the distribution server and graphics accelerator card are electrically connected via a data bus. The distribution server acquires video source information and subscription information from each participating terminal, broadcasts the video source information to other participating terminals, sends corresponding video stream requests to subscribed terminals based on the available video source type in the subscription information, and distributes the video stream to subscribed terminals according to the subscription relationship. The graphics accelerator card, under the control of the distribution server, transcodes the video stream sent by the distribution server into the encoding format with the lowest propagation bandwidth supported by the subscribed terminal when the encoding format supported by the subscribed terminal is not supported by the subscribed terminals with which there is a subscription relationship, or is not the encoding format with the lowest propagation bandwidth among the encoding formats supported by the corresponding subscribed terminal. The video source information includes the available video source types, viewable video source types, and supported encoding formats available from each participating terminal. The subscription information includes the subscription relationship between the subscribing terminal and the subscribed terminal, and the available video source types from the subscribed terminal that the subscribing terminal needs to view.

[0034] In some embodiments, the graphics accelerator card includes a decoding / encoding module and a super-resolution module. The decoding / encoding module, under the control of the distribution server, performs decoding and encoding operations on the received video stream, thereby transcoding the video stream. The super-resolution module is used to enhance the resolution of the video stream through a super-resolution model when there is a large resolution difference between the subscriber and the subscribed end, thereby improving the display effect of the low-resolution video stream on a large screen.

[0035] In some embodiments, the distribution server includes an acquisition module, a push stream request module, a distribution module, and a transcoding strategy module. The acquisition module acquires video source information, subscription information, and video streams from each participating terminal. The push stream request module sends corresponding video push stream requests to the subscribed terminals based on the subscription relationships in the subscription information, requesting the subscribed terminals to push video streams of the subscribed resolution's supplied video source type. The distribution module distributes the received video streams, or video streams transcoded by a graphics accelerator card, to the corresponding subscribed terminals based on the subscription relationships, and broadcasts the video source information of each participating terminal to other participating terminals. The transcoding strategy module lists all transcoding strategies to form a transcoding strategy space based on the subscription relationships and the encoding format types supported by the subscribing and subscribed terminals, solves for the optimal transcoding strategy in the transcoding strategy space that minimizes the weighted sum of bandwidth and computing power costs after transcoding, and controls the decoding / encoding module to transcode each video stream according to the optimal transcoding strategy.

[0036] Based on the same inventive concept, referring to Figure 5Embodiments of the present invention also provide a computer device, including a processor, a memory, and a bus. The memory stores a computer program, and the processor communicates with the memory via the bus. When the computer program is executed by the processor, it implements the steps of the video conferencing processing method described above.

[0037] In some embodiments, the processor may include one or more processing units. The processor and memory may be implemented on the same chip or on separate chips. The processor may be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the video conferencing processing method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0038] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory in the embodiments of this application can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0039] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described video conferencing processing method.

[0040] In some possible implementations, various aspects of the video conferencing processing method provided by the present invention can also be implemented as a program product comprising program code that, when the program product is run on a device, causes the control device to perform the steps of the video conferencing processing method according to the various exemplary embodiments of the present application described above.

[0041] By designing and programming the processor, the code corresponding to the video conferencing processing method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the video conferencing processing method shown in the embodiments of the present invention during operation. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0042] It should be noted that in the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first" or "second" is mentioned, it is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0043] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0044] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0046] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0047] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method of processing a video conference, characterized by, The application comprises the following steps: The distribution server acquires video source information of each participating terminal and broadcasts the video source information to other participating terminals, wherein the video source information comprises supplyable video source types, viewable video source types and supported encoding formats of each participating terminal; The distribution server acquires subscription information of each participating terminal, wherein the subscription information comprises a subscription relationship between a subscription terminal and a subscribed terminal and a supplyable video source type of the subscribed terminal that needs to be viewed by the subscription terminal; The distribution server sends a corresponding video push request to the subscribed terminal according to the supplyable video source type in the subscription information and receives a video stream sent by the subscribed terminal; If the encoding format supported by the subscribed terminal cannot be supported by the subscription terminal having the subscription relationship or is not the format with the lowest propagation bandwidth occupation among the encoding formats supported by the corresponding subscription terminal, the distribution server sends the video stream of the subscribed terminal to a graphics acceleration card, and the graphics acceleration card transcodes the received video stream into the encoding format with the lowest propagation bandwidth occupation supported by the subscription terminal; The distribution server distributes each video stream to the corresponding subscription terminal according to the subscription relationship in the subscription information.

2. The method of claim 1, wherein, The application further comprises the following steps: If the difference between the maximum resolution of the supplyable video source type of the subscribed terminal and the minimum resolution of the viewable video source type of the subscription terminal having the subscription relationship is greater than a preset threshold value, the distribution server sends the video stream of the subscribed terminal to the graphics acceleration card, and the graphics acceleration card improves the resolution of the received video stream through a super-resolution model.

3. The method of claim 1, wherein, In the step of sending a corresponding video push request to the subscribed terminal according to the supplyable video source type in the subscription information and receiving a video stream sent by the subscribed terminal, if multiple supplyable video source types of the subscribed terminal are subscribed by multiple subscription terminals, the distribution server requests the subscribed terminal to push the video stream of the supplyable video source type with the highest resolution among the subscribed types and requires the subscribed terminal to perform multi-resolution level coding on the video stream; In the step of distributing each video stream to the corresponding subscription terminal according to the subscription relationship in the subscription information, the distribution server sends the corresponding resolution layer to the corresponding subscription terminal according to the resolution of the video stream subscribed by the subscription terminal.

4. The method of claim 1, wherein, The application further comprises the following steps: The distribution server sends the received video stream to the graphics acceleration card, and the graphics acceleration card divides the video frames of the received video stream into multiple time layers with different frame rates; The distribution server monitors real-time transmission rates between the participating terminals; In the step of distributing each video stream to the corresponding subscription terminal according to the subscription relationship in the subscription information, the distribution server sends the time layer required by the frame number corresponding to the real-time transmission rate to the corresponding subscription terminal according to the mapping relationship between the transmission rate and the frame rate.

5. The method of claim 3, wherein, The application further comprises the following steps: The distribution server sends the received video stream to the graphics acceleration card, and the graphics acceleration card identifies the region of interest in each frame of image in the video stream through a ROI identification model; The distribution server monitors real-time transmission rates between the participating terminals; If the real-time transmission rate is less than a preset rate threshold value, the distribution server removes the information of the non-region of interest in the high-resolution layer in the video stream.

6. The method of claim 1, wherein, The application further comprises the following steps: The distribution server acquires user interaction information of each participating terminal; The distribution server inputs the user interaction information into a pre-trained subscription prediction model to obtain a predicted subscription relationship; The predicted subscription relationship is compared with a current subscription relationship to obtain a new subscription relationship; The distribution server sends a video push request with the lowest resolution to a subscribed end in the new subscription relationship, and distributes a video stream to the corresponding subscribed end according to the new subscription relationship.

7. A video conferencing system characterized by The video conference includes a participating terminal and a conference server, the conference server includes a distribution server and a graphics acceleration card, the participating terminal is in communication connection with the distribution server through a network, the distribution server is in electrical connection with the graphics acceleration card, the distribution server is used to obtain video source information and subscription information of each participating terminal, broadcast the video source information to other participating terminals, send a corresponding video stream request to a subscribed end according to a supply video source type in the subscription information, and distribute a video stream to the subscribed end according to a subscription relationship, the graphics acceleration card is used to, when an encoding format supported by the subscribed end cannot be supported by a subscription end with an existing subscription relationship, or when the encoding format is not supported by the corresponding subscribed end and a format with the lowest propagation bandwidth occupation is supported by the subscribed end, under the control of the distribution server, transcode the video stream sent by the distribution server into an encoding format with the lowest propagation bandwidth occupation supported by the subscribed end, the video source information includes a supply video source type, a watchable video source type and a supported encoding format that can be supplied by each participating terminal, the subscription information includes a subscription relationship between the subscribed end and the subscribed end, and a supply video source type of the subscribed end that needs to be watched by the subscribed end.

8. The video conferencing system of claim 7, wherein, The graphics acceleration card includes a decoding / encoding module and a super-resolution module, the decoding / encoding module is used to decode and encode the received video stream under the control of the distribution server, thereby realizing transcoding of the video stream, and the super-resolution module is used to, when a difference between a maximum value of a resolution of a supply video source type of the subscribed end and a minimum value of a resolution of a watchable video source type of a subscription end with an existing subscription relationship is greater than a preset threshold, improve the resolution of the video stream of the subscribed end through a super-resolution model.

9. A computer device, comprising: The computer program is executed by the processor to realize the steps of the video conference processing method according to any one of claims 1 to 6.

10. A storage medium storing a computer program, characterized by The computer program is executed by the processor to realize the steps of the video conference processing method according to any one of claims 1 to 6.

Images