A method for handling group meetings
By pre-analyzing video data with a distributed encoder, identifying feature pixels and calculating video change representation coefficients, and selecting appropriate encoding strategies and adjusting parameters, the problems of video stuttering and latency in group meetings were solved, achieving efficient encoding and low bandwidth usage.
Patent Information
- Application Number
- CN202311344345.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-17
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-17
AI Technical Summary
In group meetings, a large number of participants can easily lead to video stuttering, latency, and audio-visual desynchronization. Existing technologies have not been able to effectively adjust video encoding parameters to reduce bandwidth consumption and improve encoding efficiency.
By pre-analyzing video data using a distributed encoder, identifying the feature pixels of video frames, calculating video change representation coefficients, selecting encoding strategies based on these coefficients, and adjusting encoding parameters and frame extraction intervals to achieve efficient encoding.
While ensuring the viewing experience, reduce the amount of encoding, decrease video transmission latency and bandwidth usage, and improve encoding efficiency.
Smart Images

Figure CN117978943B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video conferencing encoding, and more particularly to a method for processing group meetings. Background Technology
[0002] Online video conferencing technology is a technology that enables remote real-time video communication and collaboration via the Internet. With the increasing demand, some online conferencing systems are now able to switch from overall meetings to breakout sessions. However, each participant in each breakout session needs to be equipped with a separate server for video encoding or audio mixing to support multiple breakout sessions simultaneously.
[0003] For example, Chinese Patent Publication No. CN114827522A discloses a method, device, and storage medium for group interaction in video conferencing, implemented based on a video conferencing system. This method includes the following steps: Step S1, after obtaining the group interaction instructions for the current video conference, parsing the group interaction instructions to obtain the group interaction information for the current group; Step S2, calculating the group interaction network requirements for the current group based on the group interaction information; Step S3, constructing a service group for the current video conference based on the group interaction network requirements, with the service group providing services to the video terminals; the service group includes several distribution servers and interaction servers. This invention provides a low-latency, high-quality, and low-load video conferencing environment, ensuring the stability of group interaction in video conferencing and meeting user needs.
[0004] However, the following problems still exist in the existing technology.
[0005] In practice, if too many people join a group meeting, it can consume bandwidth and easily cause video stuttering, video delays, and audio-visual asynchrony. However, video conferences are often accompanied by explanations, and the video content shared by the sharing party may remain static for a long time. Current technologies do not take this into account. Therefore, it is necessary to adaptively adjust the video encoding parameters based on the content shared by the sharing party, thereby reducing the amount of encoding, improving encoding efficiency, reducing video transmission latency, and reducing bandwidth consumption while ensuring video quality. Summary of the Invention
[0006] Therefore, the present invention provides a method for processing group meetings to overcome the problems in the prior art where too many participants in a group meeting consume bandwidth, easily leading to video stuttering, video delay, and audio-visual asynchrony.
[0007] To achieve the above objectives, the present invention provides a method for processing group meetings, comprising:
[0008] Step S1: Set up several distributed encoders to call the corresponding distributed encoders to connect the participants after grouping;
[0009] Step S2: The distributed encoder with pre-established connection relationship receives the video data transmitted by the sharing party participating in screen sharing, performs pre-analysis on the video data to identify the feature pixels of the video frames in each video segment, and calculates the video change characterization coefficient of each video segment based on the coordinate distance difference of the feature pixels in adjacent video frames.
[0010] Step S3: Select the encoding strategy to be applied to the video segment using the video change characterization coefficients. The encoding strategy includes:
[0011] According to the predetermined priority, the remaining distributed encoders are called in conjunction with the current distributed encoder to execute the two-end encoding strategy, so as to call video frames from both ends of the video segment for encoding at the same time. During encoding, a frame-sampling encoding method is adopted, the frame-sampling interval is adjusted in real time, and only the extracted video frames are used as reference frames to generate encoded data.
[0012] or,
[0013] Obtain basic video parameters of the video segment to adjust the encoding parameters of the current distributed encoder, the encoding parameters including the encoder's target bit rate;
[0014] Step S4: Transmit the encoded data to several participating parties connected to the distributed encoder.
[0015] Furthermore, the process of identifying the feature pixels of video frames in each video segment includes,
[0016] The closed contours existing in the video frame are identified, the smallest convex polygon surrounding the closed contour is obtained, the centroid of the smallest convex polygon is calculated, and the centroid is determined as the feature pixel point.
[0017] Further, in step S2, the video change representation coefficient is calculated according to formula (1).
[0018]
[0019] In formula (1), K represents the video change characterization coefficient, Ne represents the number of adjacent video frames in the video segment, and Kj represents the difference parameter of the j-th adjacent video frame. Kj is calculated according to formula (2).
[0020]
[0021] In formula (2), n represents the number of identical closed contours in adjacent video frames, and Di represents the coordinate distance difference between the feature pixels corresponding to the i-th identical closed contour in adjacent video frames.
[0022] Furthermore, the process of selecting the coding strategy to be applied to the video segment using the video change characterization coefficients includes,
[0023] If the video change characterization coefficient is less than the predetermined video change characterization coefficient comparison threshold, then the two-end encoding strategy is selected;
[0024] If the video change characterization coefficient is greater than or equal to a predetermined video change characterization coefficient comparison threshold, then the encoding parameters of the current distributed encoder are adjusted.
[0025] Furthermore, in step S3, the process of invoking the remaining distributed encoders according to a predetermined priority includes:
[0026] Obtain the computational load of the remaining distributed encoders and prioritize calling the distributed encoder with the smallest computational load.
[0027] Furthermore, in step S3, the process of generating encoded data by using the extracted video frame as a reference frame includes generating pixel values of the prediction frame based on the reference frame and encoding them.
[0028] Furthermore, in step S3, the process of adjusting the frame skipping interval includes,
[0029] Several adjustment methods are pre-set to adjust the frame interval according to the video change characterization coefficient, and the adjustment amount of the frame interval is different in each adjustment method.
[0030] Furthermore, in step S3, the basic video parameters of the video segment include the number of frames and the resolution.
[0031] Furthermore, step S3 also includes calculating video representation parameters based on the basic video parameters according to formula (3).
[0032]
[0033] In formula (3), G represents the number of frames, G0 represents the preset frame comparison threshold, P represents the resolution, P0 represents the preset resolution comparison threshold, γ represents the first proportional coefficient, and β represents the second proportional coefficient.
[0034] Furthermore, specifically, in step S3, during the process of adjusting the encoding parameters of the current distributed encoder,
[0035] The target bit rate adjustment is positively correlated with the video characterization parameters.
[0036] Compared with existing technologies, this invention receives video data transmitted by the sharing parties participating in screen sharing through a distributed encoder with pre-established connections. It pre-analyzes the video data, identifies the feature pixels of each video frame in each video segment, calculates the video variation representation coefficient of each video segment based on the coordinate distance difference of the feature pixels in adjacent video frames, and selects the encoding strategy to be executed on the video segment based on the video variation representation coefficient. Through the above process, considering the content of the video data transmitted by the sharing parties, a two-end encoding strategy is adopted when the video variation representation coefficient is low, and the encoding parameters of the distributed encoder are adjusted when the video variation representation coefficient is high. This reduces the impact of adjusting the encoding parameters on the video viewing experience, reduces the amount of encoding while ensuring the video viewing experience, improves encoding efficiency, reduces video transmission latency, and reduces bandwidth consumption.
[0037] In particular, this invention pre-analyzes the video data transmitted by the sharing party before encoding, calculating a video change characterization coefficient. This coefficient is calculated based on the difference in coordinate distance between feature pixels in a video frame and adjacent video frames, characterizing the degree of contour change between two adjacent video frames and the overall video variation. In practice, when contours in a video move quickly and with large amplitude, viewers are more sensitive to moving objects. Adjusting encoding parameters and reducing video quality significantly impacts the viewing experience. However, video conferencing often involves presentations, and the shared video content may remain static for extended periods. Consequently, adjusting encoding parameters and reducing video quality has a smaller impact on the viewing experience. Therefore, calculating the video change characterization coefficient provides data support for subsequent adjustments to encoding parameters, minimizing the impact of these adjustments on the video's visual appeal. This reduces the amount of encoding required, improves encoding efficiency, reduces video transmission latency, and minimizes bandwidth usage while maintaining a good viewing experience.
[0038] In particular, this invention selects the coding strategy to be performed on video segments through the video change characterization coefficient. When the video change characterization coefficient is low, an end-to-end coding strategy is adopted, and inter-frame prediction is performed, which significantly reduces the amount of data that needs to be transmitted, thereby achieving efficient video compression. Furthermore, since the inter-frame prediction effect is good when the video change characterization coefficient is low, and the extraction interval of the reference frame is related to the video change characterization coefficient, the final video effect is further guaranteed. Thus, while ensuring the video viewing experience, the amount of coding is reduced, coding efficiency is improved, video transmission latency is reduced, and bandwidth usage is reduced.
[0039] In particular, when the video change characterization coefficient is high, the video dynamic changes are obvious, the inter-frame prediction effect is poor, and the computing power is consumed. Therefore, the present invention adaptively reduces the target bit rate by the basic video parameters of the video segment to adjust the overall effect of the encoded video, thereby reducing video transmission latency and bandwidth consumption. Attached Figure Description
[0040] Figure 1 This is a flowchart illustrating the steps of a group conference processing method according to an embodiment of the invention.
[0041] Figure 2 A logic diagram is selected for the encoding strategy of the embodiments of the invention;
[0042] Figure 3 This is a schematic diagram of the distributed encoder arrangement according to an embodiment of the invention;
[0043] Figure 4 This is a schematic diagram of the encoding order at both ends of an embodiment of the invention;
[0044] In the diagram, 1 represents a video frame. Detailed Implementation
[0045] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.
[0046] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0047] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0048] Please see Figure 1 , Figure 2 as well as Figure 3 As shown, Figure 1 This is a flowchart illustrating the steps of a group meeting processing method according to an embodiment of the present invention. Figure 2 A logic diagram is selected for the encoding strategy of the embodiments of the invention. Figure 3 A schematic diagram of the distributed encoder layout of an embodiment of the invention. A method for processing group conferences according to the present invention includes:
[0049] Step S1: Set up several distributed encoders to call the corresponding distributed encoders to connect the participants after grouping;
[0050] Step S2: The distributed encoder with pre-established connection relationship receives the video data transmitted by the sharing party participating in screen sharing, performs pre-analysis on the video data to identify the feature pixels of the video frames in each video segment, and calculates the video change characterization coefficient of each video segment based on the coordinate distance difference of the feature pixels in adjacent video frames.
[0051] Step S3: Select the encoding strategy to be applied to the video segment using the video change characterization coefficients. The encoding strategy includes:
[0052] According to the predetermined priority, the remaining distributed encoders are called in conjunction with the current distributed encoder to execute the two-end encoding strategy, so as to call video frames from both ends of the video segment for encoding at the same time. During encoding, a frame-sampling encoding method is adopted, the frame-sampling interval is adjusted in real time, and only the extracted video frames are used as reference frames to generate encoded data.
[0053] or,
[0054] Obtain basic video parameters of the video segment to adjust the encoding parameters of the current distributed encoder, the encoding parameters including the encoder's target bit rate;
[0055] Step S4: Transmit the encoded data to several participating parties connected to the distributed encoder.
[0056] Specifically, the present invention does not limit the specific structure of the distributed encoder. Those skilled in the art can select the corresponding encoder according to the format of the video data, such as an H.264 encoder, an H.265 encoder, or a VP9 encoder.
[0057] Specifically, the present invention does not limit the image processing algorithm for analyzing video data. For example, the functions of identifying closed contours and determining feature pixels can be achieved by pre-constructing an image processing model or algorithm that can realize the above functions, importing it into the computer to process the corresponding image data, and realizing the corresponding functions.
[0058] Specifically, the process of identifying the feature pixels of video frames in each video segment includes,
[0059] The closed contours existing in the video frame are identified, the smallest convex polygon surrounding the closed contour is obtained, the centroid of the smallest convex polygon is calculated, and the centroid is determined as the feature pixel point.
[0060] It is understandable that the methods described above for obtaining the minimum convex polygon that encloses the closed contour and for determining the centroid are based on mathematical calculations and can be implemented using image processing libraries (such as OpenCV), which will not be elaborated here.
[0061] Specifically, in step S2, the video change representation coefficient is calculated according to formula (1).
[0062]
[0063] In formula (1), K represents the video change characterization coefficient, Ne represents the number of adjacent video frames in the video segment, and Kj represents the difference parameter of the j-th adjacent video frame. Kj is calculated according to formula (2).
[0064]
[0065] In formula (2), n represents the number of identical closed contours in adjacent video frames, and Di represents the coordinate distance difference between the feature pixels corresponding to the i-th identical closed contour in adjacent video frames.
[0066] Specifically, it can be understood that adjacent video frames are two adjacent video frames. If an object is moving in the two video frames, the coordinates of the feature pixels corresponding to the same closed contour in the two video frames will be different, and the coordinate distance difference can be calculated. Of course, the coordinate system reference for determining the coordinates of the feature pixels should be the same, and the center of the video frame can be selected as the coordinate system reference.
[0067] Specifically, the process of selecting the coding strategy to be applied to a video segment using the video change characterization coefficients includes,
[0068] If the video change characterization coefficient is less than the predetermined video change characterization coefficient comparison threshold, then the two-end encoding strategy is selected;
[0069] If the video change characterization coefficient is greater than or equal to a predetermined video change characterization coefficient comparison threshold, then the encoding parameters of the current distributed encoder are adjusted.
[0070] Specifically, in this embodiment, the video change characterization coefficient comparison threshold K0 is obtained in advance. A video segment of a predetermined duration is obtained as a sample, and the video change characterization coefficient corresponding to the video segment of the predetermined duration is calculated and recorded as Ke. K0 is set to α × Ke, where α represents the precision coefficient and is set to 0.4 < α < 0.7.
[0071] This invention pre-analyzes the video data transmitted by the sharing party before encoding, calculating a video change characterization coefficient. This coefficient, based on the coordinate distance difference between feature pixels in adjacent video frames, characterizes the degree of contour change between two adjacent frames, reflecting the overall video dynamics. In practice, when contours move quickly and significantly, viewers are more sensitive to moving objects. Adjusting encoding parameters and reducing video quality significantly impacts viewing experience. However, during video conferences, presentations are often included, and the shared video content may remain static for extended periods. Therefore, adjusting encoding parameters and reducing video quality has a smaller impact on viewing experience. By calculating the video change characterization coefficient, data support is provided for subsequent encoding parameter adjustments, minimizing the impact on video viewing experience. This reduces encoding volume, improves encoding efficiency, reduces video transmission latency, and minimizes bandwidth usage while maintaining video quality.
[0072] Specifically, in step S3, the process of invoking the remaining distributed encoders according to a predetermined priority includes:
[0073] Obtain the computational load of the remaining distributed encoders and prioritize calling the distributed encoder with the smallest computational load.
[0074] It is understandable that encoders consume computing resources, including CPU and memory, when performing encoding processing. The computing load here refers to the computing load of the GPU.
[0075] Specifically, in step S3, the process of generating encoded data by using the extracted video frame as a reference frame includes generating pixel values of the prediction frame based on the reference frame and encoding them.
[0076] Specifically, it can be understood that the basic idea of inter-frame prediction is that the pixel value of the current frame can be predicted by referring to some previous frames. The relevant prediction algorithms are already quite mature, so they will not be elaborated here.
[0077] Specifically, in step S3, the process of adjusting the frame skipping interval includes,
[0078] Several adjustment methods are pre-set to adjust the frame interval according to the video change characterization coefficient, and the adjustment amount of the frame interval is different in each adjustment method.
[0079] In this embodiment, at least three adjustment methods for adjusting the frame skipping interval are provided, including:
[0080] The first method for adjusting the frame extraction interval is to increase the initial frame extraction interval by 1.
[0081] The second method for adjusting the frame extraction interval is to increase the initial frame extraction interval by 2.
[0082] The third method for adjusting the frame extraction interval is to increase the initial frame extraction interval by 3.
[0083] The initial frame skipping interval is set within the interval [2,4]. The first adjustment method requires 0.75K0 < K, the second adjustment method requires 0.6K0 < K ≤ 0.75K0, and the third adjustment method requires K ≤ 0.6K0.
[0084] Specifically, please refer to Figure 4 As shown, Figure 4 This is a schematic diagram of the encoding order at both ends of an embodiment of the present invention. When video frame 1 is called from both ends of a video segment for encoding, video frames are called every frame interval starting from the video frames at both ends of the video frame.
[0085] Specifically, in step S3, the basic video parameters of the video segment include the number of frames and the resolution.
[0086] This invention selects the encoding strategy for video segments based on the video change characterization coefficient. When the video change characterization coefficient is low, an end-to-end encoding strategy is adopted, and inter-frame prediction is performed, significantly reducing the amount of data that needs to be transmitted, thereby achieving efficient video compression. Furthermore, since the inter-frame prediction effect is good under the condition of low video change characterization coefficient, and the extraction interval of the reference frame is related to the video change characterization coefficient, the final video effect is further guaranteed. Thus, while ensuring the video viewing experience, the amount of encoding is reduced, the encoding efficiency is improved, the video transmission latency is reduced, and the bandwidth usage is reduced.
[0087] Specifically, step S3 further includes calculating video representation parameters based on the basic video parameters according to formula (3).
[0088]
[0089] In formula (3), G represents the number of frames, G0 represents the preset frame comparison threshold, P represents the resolution, P0 represents the preset resolution comparison threshold, γ represents the first proportional coefficient, β represents the second proportional coefficient, β+γ=1, and in this embodiment, β=0.35 and γ=0.65 are set.
[0090] G0 is set within the interval [30, 60]. In this embodiment, G0 is set to 30, and P0 is set to 1280x720.
[0091] Specifically, in step S3, during the process of adjusting the encoding parameters of the current distributed encoder,
[0092] The target bit rate adjustment is positively correlated with the video characterization parameters.
[0093] In this embodiment, optionally,
[0094] If F < 0.6, adjust the target coding rate to 2 Mbps;
[0095] If 0.6 ≤ F < 2, adjust the target coding rate to 5 Mbps;
[0096] If 2 ≤ F, adjust the target coding rate to 10 Mbps.
[0097] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A method for processing group meetings, characterized in that, include: Step S1: Set up several distributed encoders to call the corresponding distributed encoders to connect the participants after grouping; Step S2: The distributed encoder with pre-established connection relationship receives the video data transmitted by the sharing party participating in screen sharing, performs pre-analysis on the video data to identify the feature pixels of the video frames in each video segment, and calculates the video change characterization coefficient K of each video segment based on the coordinate distance difference of the feature pixels in adjacent video frames. Step S3: Select the encoding strategy to be applied to the video segment using the video change characterization coefficient K. The encoding strategy includes: According to the predetermined priority, the remaining distributed encoders are invoked in conjunction with the current distributed encoder to execute the two-end encoding strategy, so as to simultaneously call video frames from both ends of the video segment for encoding. During encoding, a frame-skipping encoding method is adopted, the frame-skipping interval is adjusted in real time, and only the extracted video frames are used as reference frames to generate encoded data. The frame-skipping interval adjustment method includes: the first frame-skipping interval adjustment method is to increase the initial frame-skipping interval by 1; the second frame-skipping interval adjustment method is to increase the initial frame-skipping interval by 2; and the third frame-skipping interval adjustment method is to increase the initial frame-skipping interval by 3. The initial frame-skipping interval is set within the interval [2,4]. The first adjustment method must satisfy 0.75K0 < K, the second adjustment method must satisfy 0.6K0 < K ≤ 0.75K0, and the third adjustment method must satisfy K ≤ 0.6K0, where K0 is the video change characterization coefficient comparison threshold. or, Obtain basic video parameters of the video segment to adjust the encoding parameters of the current distributed encoder, the encoding parameters including the encoder's target bit rate; Step S4: Transmit the encoded data to several participating parties connected to the distributed encoder.
2. The method for processing group meetings according to claim 1, characterized in that, In step S2, the process of calibrating the feature pixel points of video frames in each video segment includes: The closed contours existing in the video frame are identified, the smallest convex polygon surrounding the closed contour is obtained, the centroid of the smallest convex polygon is calculated, and the centroid is determined as the feature pixel point.
3. The method for processing group meetings according to claim 2, characterized in that, In step S2, the video change representation coefficient is calculated according to formula (1). In formula (1), K represents the video change characterization coefficient, and Ne represents the number of adjacent video frames in the video segment. Let Kj represent the difference parameter of the j-th adjacent video frame, calculated according to formula (2). In formula (2), n represents the number of identical closed contours in adjacent video frames, and Di represents the coordinate distance difference between the feature pixels corresponding to the i-th identical closed contour in adjacent video frames.
4. The method for processing group meetings according to claim 1, characterized in that, In step S3, the process of selecting the coding strategy to be applied to the video segment using the video change characterization coefficients includes the following: If the video change characterization coefficient is less than the predetermined video change characterization coefficient comparison threshold, then the two-end encoding strategy is selected; If the video change characterization coefficient is greater than or equal to a predetermined video change characterization coefficient comparison threshold, then the encoding parameters of the current distributed encoder are adjusted.
5. The method for processing group meetings according to claim 1, characterized in that, In step S3, the process of calling the remaining distributed encoders according to a predetermined priority includes: Obtain the computational load of the remaining distributed encoders and prioritize calling the distributed encoder with the smallest computational load.
6. The method for processing group meetings according to claim 1, characterized in that, In step S3, the process of using the extracted video frame as a reference frame to generate encoded data includes generating the pixel values of the prediction frame based on the reference frame and encoding it.
7. The method for processing group meetings according to claim 1, characterized in that, In step S3, the basic video parameters of the video segment include the number of frames and the resolution.
8. The method for processing group meetings according to claim 1, characterized in that, Step S3 further includes calculating video representation parameters based on the basic video parameters according to formula (3). In formula (3), G represents the number of frames, G0 represents the preset frame comparison threshold, P represents the resolution, P0 represents the preset resolution comparison threshold, γ represents the first proportional coefficient, β represents the second proportional coefficient, and the sum of β and γ is 1.
9. The method for processing group meetings according to claim 1, characterized in that, In step S3, during the process of adjusting the encoding parameters of the current distributed encoder, Adjusting the target bit rate is positively correlated with the video characterization parameters.
Citation Information
Patent Citations
Grouping interaction method and equipment for video conference and storage medium
CN114827522A
Regional distributed video conference device
CN213126228U
Behavior recognition method and apparatus, terminal device, and readable storage medium
WO2022134983A1
