Network online conference method based on video monitoring

Abstract

The invention discloses a network online conference method based on video monitoring, which relates to the technical field of network conferences, and comprises the following steps: pre-configuring conference participating terminals to realize connection with a conference server and receive and display audio and video data in a conference, comprising a camera used for collecting video images, a microphone used for collecting sound and a loudspeaker used for playing audios in a conference. And the conference server is used for managing a conference and processing, distributing and storing audio and video data of the conference participating terminals, and the conference server detects the video fluency of the conference participating terminals and performs adaptive adjustment. Transmission of audio and video data can be preferentially guaranteed through self-adaptive adjustment, it is ensured that participants can clearly hear speaking content, meanwhile, manual intervention is reduced through automatic adjustment, adjustment is automatically conducted according to the video fluency, and the workload of conference organizers or manual adjustment of video parameters is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of network conferences, and in particular to a network online conference method based on video monitoring. Background Art

[0002] Online conferencing refers to a form of conference that uses Internet technology to allow people in different geographical locations to communicate, collaborate and exchange ideas in real time.

[0003] Video conferencing is the most intuitive form of online conferencing. Participants can see each other's facial expressions, body language and other non-verbal information, which enhances the authenticity of communication. For example, in a remote interview scenario, the interviewer can better observe the applicant's status through video conferencing, and the applicant can also show his image and communication ability more directly. Video conferencing has relatively high requirements for network bandwidth and equipment performance because it needs to transmit and process multiple video streams at the same time. In order to ensure video quality, video encoding technology (such as H.264, H.265) is usually used to compress the video before transmission.

[0004] At present, a certain amount of bandwidth is required to transmit video data during video conferencing. If the bandwidth is not enough, the transmission of video data will be hindered, just like a narrow road cannot accommodate enough vehicles. For example, high-definition video conferencing usually requires at least 1-2Mbps bandwidth to ensure basic fluency. If multiple high-resolution video streams are turned on at the same time, or there are other devices in the network that occupy a lot of bandwidth (such as someone downloading files, watching high-definition videos, etc.), the available bandwidth for video conferencing will be insufficient, resulting in jams. At the same time, if the time required for data to be transmitted from the sender to the receiver is too long, delays will occur. Network congestion, long distances between servers, and other factors may cause delays. When the delay is too high, the video data cannot reach the receiver in time, which will cause jams. For example, in a cross-border video conference, data may need to be transmitted through multiple network nodes and submarine optical cables. The long distance leads to increased delays, and the picture and sound may be out of sync, causing jams, thereby affecting the quality of online meetings.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In view of the problems in the related art, the present invention proposes a network online conference method based on video monitoring to overcome the above technical problems existing in the existing related art.

[0007] The technical solution of the present invention is achieved in this way:

[0008] A network online conference method based on video surveillance includes the following steps:

[0009] Pre-configure the conference terminal to connect to the conference server for receiving and displaying audio and video data in the conference, wherein the conference terminal includes: a camera for capturing video images, a microphone for capturing sound, and a speaker for playing audio in the conference; the conference server is used to manage the conference and process, distribute, and store the audio and video data of the conference terminal;

[0010] The conference server detects the video fluency of the participating terminals and performs adaptive adjustments, including: adjusting the video quality or frame rate according to the video dynamics of the current participating terminals.

[0011] Furthermore, the conference server detects the video fluency of the participating terminals, including the following steps:

[0012] Obtaining the video dynamics of the current client video includes the following steps:

[0013] The resolution of the calibrated video frame is N×M. For each pixel position (x, y), the pixel value of the current frame is set to I t (x,y), the pixel value of the previous frame is I t-1 (x, y), then the inter-frame difference is expressed as: D(x, y) = |I t (x,y)-I t-1 (x,y)|;

[0014] Get the sum of inter-frame differences

[0015] A dynamic threshold T is set. When S>T, the video is judged to have a high dynamic level. When S≤T, the video is judged to have a low dynamic level.

[0016] Furthermore, the adaptive adjustment comprises the following steps:

[0017] If the current video has high dynamic level, adjust the video quality;

[0018] If the current video has low dynamic level, adjust the video frame rate.

[0019] Further, the video quality adjustment includes: adjusting the resolution, including the following steps:

[0020] Pre-set the network bandwidth interval threshold and set the corresponding resolution level;

[0021] When detecting that the online conference video is stuck, obtain the current client network bandwidth value, and obtain the network bandwidth interval threshold corresponding to the current network bandwidth value;

[0022] According to the network bandwidth interval threshold, adjust the current client video resolution to meet the resolution level;

[0023] Further, the adjusting the current client video resolution to conform to the resolution level comprises the following steps:

[0024] Pre-set the jam level threshold and time threshold;

[0025] The video resolution of the client at the current resolution level is periodically tested for the degree of freeze, and the resolution level is adjusted according to the degree of freeze.

[0026] Furthermore, adjusting the resolution level according to the degree of freeze includes the following steps:

[0027] During the adjustment period, if the current client video resolution has no lag, the resolution level will be increased by one level, and the lag level will be tested during the next adjustment period.

[0028] During the adjustment period, if the current client video resolution is occasionally stuck and the stuck time is less than the time threshold, no processing will be done;

[0029] During the adjustment period, if the current client video resolution is frequently stuck and the stuck time is greater than the time threshold, the resolution level will be reduced by one level, and the stuck level will be detected in the next adjustment period;

[0030] Furthermore, the video frame rate adjustment includes the following steps:

[0031] Pre-set the video frame rate adjustment interval threshold;

[0032] Get the current video frame rate, and adjust the interval threshold according to the video frame rate to adjust the current video frame rate to lower one video frame rate.

[0033] Beneficial effects of the present invention:

[0034] The present invention pre-configures a conference participant terminal to achieve connection with a conference server for receiving and displaying audio and video data in a conference, wherein the conference participant terminal comprises: a camera for capturing video images, a microphone for capturing sound, and a loudspeaker for playing audio in a conference; the conference server is used to manage the conference and process, distribute, and store audio and video data of the conference participant terminal. The conference server detects the video fluency of the conference participant terminal and performs adaptive adjustment to realize a network online conference based on video monitoring. The adaptive adjustment can give priority to the transmission of audio and video data to ensure that the participants can clearly hear the speech content. Even if the video quality is reduced due to network problems, the stable audio can enable the participants to keep up with the rhythm of the conference, understand the key points of the conference, and improve the quality of the conference. At the same time, the automatic adjustment reduces manual intervention, and it automatically adjusts according to the video fluency, reducing the workload of the conference organizer or manually adjusting the video parameters.

[0035] In addition, to adapt to different network environments, whether in a wired network or a wireless network environment, whether in the case of high, low or large fluctuations in network bandwidth, the adaptive adjustment strategy based on video fluency detection can enable the network conferencing system to respond quickly, which can reduce the amount of video data for each participant, so that the meeting can maintain a relatively stable operating state in a congested network environment and reduce the risk of meeting interruption. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 The present invention is a flowchart of a network online conference method based on video surveillance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0039] According to an embodiment of the present invention, a network online conference method based on video monitoring is provided.

[0040] like Figure 1 As shown, the network online conference method based on video monitoring according to an embodiment of the present invention includes the following steps:

[0041] Step S1, pre-configure the conference terminal to connect to the conference server for receiving and displaying audio and video data in the conference, wherein the conference terminal includes: a camera for capturing video images, a microphone for capturing sound, and a speaker for playing audio in the conference; the conference server is used to manage the conference and process, distribute and store the audio and video data of the conference terminal;

[0042] Step S2, the conference server detects the video fluency of the participating terminal and performs adaptive adjustment, which includes: adjusting the video quality or frame rate according to the video dynamics of the current participating terminal.

[0043] The conference server detects the video fluency of the participating terminals, including the following steps:

[0044] Obtaining the video dynamics of the current client video includes the following steps:

[0045] The resolution of the calibrated video frame is N×M. For each pixel position (x, y), the pixel value of the current frame is set to I t (x,y), the pixel value of the previous frame is I t-1 (x, y), then the inter-frame difference is expressed as: D(x, y) = |I t (x,y)-I t-1 (x,y)|;

[0046] Add up the D(x,y) of the entire video frame to get the sum of the inter-frame differences

[0047] A dynamic threshold T is set. When S>T, the video is judged to have a high dynamic level. When S≤T, the video is judged to have a low dynamic level.

[0048] If the current video has a high degree of dynamics, the video quality is adjusted, including: adjusting the resolution, as follows:

[0049] Pre-set the network bandwidth interval threshold and set the corresponding resolution level;

[0050] When the technical solution is applied, for example, when the network bandwidth is lower than 1 Mbps, the video resolution is 720×480; when the bandwidth is between 1-2 Mbps, the resolution is 1280×720.

[0051] When detecting that the online conference video is stuck, obtain the current client network bandwidth value, and obtain the network bandwidth interval threshold corresponding to the current network bandwidth value;

[0052] According to the network bandwidth interval threshold, adjust the current client video resolution to meet the resolution level;

[0053] Specifically, when applied, for example, the resolution is reduced from high definition (1920×1080) to standard definition (1280×720) or even lower.

[0054] In addition, adjusting the current client video resolution to match the resolution level also includes the following steps:

[0055] Pre-set the jamming degree threshold and time threshold, wherein the jamming degree threshold includes: occasional and frequent, and the jamming degree threshold is set according to the number of jamming times in a period;

[0056] Periodically detect the degree of freeze of the client video resolution at the current resolution level, and adjust the resolution level according to the degree of freeze, including:

[0057] During the adjustment period, if the current client video resolution has no lag, the resolution level will be increased by one level, and the lag level will be tested during the next adjustment period.

[0058] During the adjustment period, if the current client video resolution is occasionally stuck and the stuck time is less than the time threshold, no processing will be done;

[0059] During the adjustment period, if the current client video resolution is frequently stuck and the stuck time is greater than the time threshold, the resolution level will be reduced by one level, and the stuck level will be detected in the next adjustment period;

[0060] Specifically, when applying, if the video resolution of the client at the current resolution level freezes only occasionally and for a short period of time, no processing will be done; if the freezes are frequent and last for a long time, the resolution will be reduced by one level. For example, if the freeze is detected for less than 5 seconds for the first time, no processing will be done; if the freeze is detected for more than 5 seconds for the second time, it will be reduced from 1080p to 720p; if the freeze is still not improved, and the freeze rate (freeze frame number / total frame number) exceeds 30% in the next 10 seconds, it will be reduced to 480p.

[0061] If the current video has a low dynamic level, the video frame rate is adjusted, including the following steps:

[0062] Pre-set the video frame rate adjustment interval threshold;

[0063] Obtain the current video frame rate, and adjust the interval threshold according to the video frame rate to adjust the current video frame rate to a lower video frame rate;

[0064] In this technical solution, the frame rate determines the smoothness of the video. Under normal circumstances, the video frame rate may be 30fps or 60fps. When the network is not good and there are signs of freezing, the system can reduce the frame rate to 15fps or lower. This is because after the frame rate is reduced, the number of video frames that need to be transmitted per second is reduced, thereby reducing the burden on network transmission.

[0065] With the help of the above technical solution, for scenes with more dynamic images (such as people frequently walking around in a meeting, and more demonstrations), a relatively high frame rate is maintained to adjust the video quality; while for scenes with more static images (such as mainly document presentations and PPT explanations), the frame rate is reduced more. For example, during the document presentation stage, if a freeze is detected, the frame rate is reduced from 30fps to 10fps to ensure video quality.

[0066] In summary, with the help of the above technical solution of the present invention, the following effects can be achieved:

[0067] The present invention pre-configures a conference participant terminal to achieve connection with a conference server for receiving and displaying audio and video data in a conference, wherein the conference participant terminal comprises: a camera for capturing video images, a microphone for capturing sound, and a loudspeaker for playing audio in a conference; the conference server is used to manage the conference and process, distribute, and store audio and video data of the conference participant terminal. The conference server detects the video fluency of the conference participant terminal and performs adaptive adjustment to realize a network online conference based on video monitoring. The adaptive adjustment can give priority to the transmission of audio and video data to ensure that the participants can clearly hear the speech content. Even if the video quality is reduced due to network problems, the stable audio can enable the participants to keep up with the rhythm of the conference, understand the key points of the conference, and improve the quality of the conference. At the same time, the automatic adjustment reduces manual intervention, and it automatically adjusts according to the video fluency, reducing the workload of the conference organizer or manually adjusting the video parameters.

[0068] In addition, to adapt to different network environments, whether in a wired network or a wireless network environment, whether in the case of high, low or large fluctuations in network bandwidth, the adaptive adjustment strategy based on video fluency detection can enable the network conferencing system to respond quickly, which can reduce the amount of video data for each participant, so that the meeting can maintain a relatively stable operating state in a congested network environment and reduce the risk of meeting interruption.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. After considering the disclosure of the specification and the examples, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and the examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0070] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A network online conference method based on video surveillance, characterized in that: The following steps are involved: Pre-configure the conference terminal to connect to the conference server for receiving and displaying audio and video data in the conference, wherein the conference terminal includes: a camera for capturing video images, a microphone for capturing sound, and a speaker for playing audio in the conference; the conference server is used to manage the conference and process, distribute, and store the audio and video data of the conference terminal; The conference server detects the video fluency of the participating terminals and performs adaptive adjustments, including: adjusting the video quality or frame rate according to the video dynamics of the current participating terminals.

2. The method for online conference based on video surveillance according to claim 1, characterized in that: The conference server detects the video fluency of the participating terminals, including the following steps: Obtaining the video dynamics of the current client video includes the following steps: The resolution of the calibrated video frame is N×M. For each pixel position (x, y), the pixel value of the current frame is set to I t (x,y), the pixel value of the previous frame is I t-1 (x, y), then the inter-frame difference is expressed as: D(x, y) = |I t (x,y)-I t-1 (x,y)|; Get the sum of inter-frame differences A dynamic threshold T is set. When S>T, the video is judged to have a high dynamic level. When S≤T, the video is judged to have a low dynamic level.

3. The network online conference method based on video monitoring according to claim 2 is characterized in that: The adaptive adjustment comprises the following steps: If the current video has high dynamic level, adjust the video quality; If the current video has low dynamic level, adjust the video frame rate.

4. The method for online conference based on video surveillance according to claim 3 is characterized in that: The video quality adjustment includes: adjusting the resolution, including the following steps: Pre-set the network bandwidth interval threshold and set the corresponding resolution level; When detecting that the online conference video is stuck, obtain the current client network bandwidth value, and obtain the network bandwidth interval threshold corresponding to the current network bandwidth value; According to the network bandwidth range threshold, adjust the current client video resolution to meet the resolution level.

5. The method for online conference based on video surveillance according to claim 4, characterized in that: The step of adjusting the current client video resolution to match the resolution level comprises the following steps: Pre-set the jam level threshold and time threshold; The video resolution of the client at the current resolution level is periodically tested for the degree of freeze, and the resolution level is adjusted according to the degree of freeze.

6. The method for online conference based on video surveillance according to claim 5, characterized in that: The step of adjusting the resolution level according to the degree of jamming includes the following steps: During the adjustment period, if the current client video resolution has no lag, the resolution level will be increased by one level, and the lag level will be tested during the next adjustment period. During the adjustment period, if the current client video resolution is occasionally stuck and the stuck time is less than the time threshold, no processing will be done; During the adjustment period, if the current client video resolution is frequently stuck and the stuck time is greater than the time threshold, the resolution level is reduced by one level, and the stuck level is detected in the next adjustment period.

7. The method for online conference based on video surveillance according to claim 3, characterized in that: The video frame rate adjustment comprises the following steps: Pre-set the video frame rate adjustment interval threshold; Get the current video frame rate, and adjust the interval threshold according to the video frame rate to adjust the current video frame rate to lower one video frame rate.

Images