Remote video conference system

Abstract

The invention relates to the technical field of remote video conferences, in particular to a remote video conference system which comprises a data acquisition module, an image recognition module, a voice recognition module, a message recognition module, a data center module and a data feedback module. The data acquisition module is used for collecting data information of participants; the image recognition module is used for recognizing images of participants; the voice recognition module is used for voice recognition of participants; the message identification module is used for identifying text messages of participants; the data center module is used for storing data and calculating an index according to the data information; the indexes comprise a video conference attention index, a video conference interaction index and a video conference evaluation index; and the data feedback module feeds back the quality and the effect of the video conference according to the related indexes. According to the method, the video conference quality is effectively evaluated by providing the video conference evaluation index, the video conference attention index and the video conference interaction index.

Description

Technical Field

[0001] This invention relates to the field of remote video conferencing technology, specifically to a remote video conferencing system. Background Technology

[0002] The rapid development of online video technology is constantly empowering traditional industries and scenarios, injecting new vitality. Nowadays, online video technology is no longer limited to video conferencing. In addition to shining in scenarios such as academic forums, lectures, and reports, it is also showing its potential in areas such as live streaming, online games, and online courses.

[0003] Video conferencing can overcome spatial limitations, allowing members located in different places to communicate face-to-face. In video conferencing, the meeting host can impart knowledge and skills to participants through online video, or exchange and discuss opinions and viewpoints, which greatly saves the time and energy spent organizing meetings.

[0004] However, because video conferencing systems cannot accurately and completely convey the emotional state of participants during online image and voice transmission, and the perspective for focusing on and analyzing the behavior of participants is relatively limited, the host cannot clearly perceive the state and reactions of participants during remote meetings, and therefore cannot obtain feedback on the quality of the meeting.

[0005] Based on the above factors, the present invention provides a remote video conferencing system. Summary of the Invention

[0006] The purpose of this invention is to provide a remote video conferencing system for quality feedback in remote online meetings. By measuring changes in the number of participants, total meeting duration, participants' attention index, and participants' interaction index, a video conferencing evaluation index is derived to provide feedback on the quality of the video conferencing and further identify areas where problems have occurred.

[0007] To achieve the above objectives, the present invention provides a remote video conferencing system;

[0008] The remote video conferencing system includes a data acquisition module, an image recognition module, a voice recognition module, a message recognition module, a data middleware module, and a data feedback module.

[0009] The data acquisition module is used to collect data information of participants during video conferences, including the participants' client video and the participants' registered identity information.

[0010] The image recognition module is used for image recognition of the participants to obtain the time the participants appear on the screen during the video conference, the time the participants gaze at the screen, and the time the participants take notes; the image recognition module includes a participant attention recognition model;

[0011] The speech recognition module is used for speech recognition of the participants and to calculate the duration of speech interactions; the speech recognition module includes a speech recognition model.

[0012] The message recognition module is used to recognize the text messages left by the participants and to count and store the number of messages.

[0013] The data platform module is used for data storage and index calculation; the indices include a video conference attention index, a video conference interaction index, and a video conference evaluation index; the video conference attention index measures the level of participation of attendees; the video conference interaction index indicates the level of interaction among attendees during the video conference; and the video conference evaluation index is used to evaluate the quality of the video conference.

[0014] The data feedback module provides feedback on the video conference based on relevant indices.

[0015] The process for calculating the video conference attention index in the image recognition module is as follows:

[0016] The process involves: acquiring the client-side video of participants from the data acquisition module as the initial video file; reading the initial video file frame by frame to obtain a set of video frame images; identifying the set of video frame images containing faces in the video file using a convolutional neural network and recording and converting them into durations; organizing the video frame images containing faces into a separate set of facial video frame images; recognizing the set of facial video frame images using the participant attention recognition model; obtaining the set of video frame images in the set of facial video frame images where participants are looking directly at the screen and recording and converting them into durations; obtaining the set of video frame images in the set of facial video frame images where participants are writing notes and recording and converting them into durations; and calculating the video conference attention index based on the collected data.

[0017] The image recognition module includes a participant attention recognition model;

[0018] The attendee attention recognition model comprises: an image input layer, a first training module, a second training module, a third training module, a fully connected layer, and a result output layer; wherein the first training module comprises two 3x3 convolutional layers, an ECA optimization layer, and a max pooling layer; the second training module comprises two 3x3 convolutional layers and a max pooling layer; and the third training module comprises two 3x3 convolutional layers, an ECA optimization layer, and a max pooling layer.

[0019] The process by which the participant attention recognition model identifies the facial video frame image set is as follows:

[0020] The set of facial video frames is imported into the image input layer to obtain image set M1; image set M1 is imported into the first training module for training to obtain image set M2; image set M2 is imported into the second training module for training to obtain image set M3; image set M3 is imported into the third training module for training to obtain image set M4; image set M4 is imported into the fully connected layer for processing to obtain image set M5; image set M5 is imported into the result output layer for judgment, and the judgment result is output.

[0021] The video conference attention index measures the level of engagement of participants through their performance during the video conference; the formula for the video conference attention index is:

[0022]

[0023] Where β is the video conference attention index; i represents the number of participants; n is the total number of participants; t0 represents the total duration of the video conference; t i1 The duration of time that participant i appears on the screen during a video conference; t i2 The duration of time that participant i's gaze is fixed on the screen when i appears on the screen; t i3 The time for taking notes when participant i appears on the screen; α1 is the first adjustment parameter; α2 is the second adjustment parameter.

[0024] The process by which the speech recognition module performs speech recognition on the video from the participants' client devices is as follows:

[0025] The process involves: acquiring audio information from participants' client-side videos and organizing it into an audio dataset; preprocessing the audio dataset to obtain a preprocessed audio dataset; identifying the meeting host's voice in the preprocessed audio dataset and masking the host's voice to obtain a participant audio dataset; identifying the participant audio dataset using a participant speech recognition model; wherein, if the identified participant audio dataset contains a voice that has not appeared before, the voice is encoded and timed; if the identified participant audio dataset contains a voice that has appeared before and been encoded, the voice is timed and added to the previous time record; and calculating the video conference interaction index based on the collected data.

[0026] The preprocessing steps of the speech recognition module for the audio dataset include pre-emphasis, frame segmentation, windowing, Fourier transform, Mel filter processing, discrete cosine transform, and difference processing.

[0027] The audio dataset is pre-emphasized to obtain audio dataset P1; audio dataset P1 is then segmented into frames to obtain audio dataset P2; audio dataset P2 is then windowed to obtain audio dataset P3; audio dataset P3 is then subjected to Fourier transform to obtain audio dataset P4; audio dataset P4 is then processed by a Mel filter to obtain audio dataset P5; audio dataset P5 is then subjected to discrete cosine transform to obtain audio dataset P6; and audio dataset P6 is then subjected to differential processing to obtain a preprocessed audio dataset.

[0028] The process by which the message recognition module identifies the interaction index of participants' messages during a video conference is as follows:

[0029] The system retrieves the message boards of participants during the video conference to obtain message data; it then groups and statistically analyzes the message data of participants by their IDs to obtain grouped message data; it performs emotional text recognition on the grouped message data to obtain a positive coefficient; and it calculates the message interaction index based on the number of messages and the positive coefficient.

[0030] The video conferencing interaction index is calculated using the voice interaction index and the message interaction index, taking into account the synergistic effect of the voice interaction and the message interaction.

[0031] The formula for the video conferencing interaction index is:

[0032] γ=μ1*γ1+μ2*γ2+μ3*γ1*γ2;

[0033]

[0034]

[0035] Where γ is the video conferencing interaction index, and γ1 represents the voice interaction index of the participants; u i γ represents the speaking time of participant i during the video conference; u0 is the set audio duration threshold; γ2 represents the participant's comment interaction index; c i c0 represents the number of text messages left by participant i during the video conference; c0 is the set threshold for the number of messages; ω i μ1 is the active participation coefficient for participant i; μ2 is the voice adjustment parameter; μ3 is the comment adjustment parameter; e is the natural constant.

[0036] The video conferencing evaluation index considers the impact of changes in the number of participants during the video conference, the average duration of the video conference, the video conference attention index, and the video conference interaction index; it is used to measure the quality of the video conference.

[0037] The formula for calculating the video conferencing evaluation index is as follows:

[0038]

[0039]

[0040] in, yt1 is the video conference evaluation index; n1 is the number of participants at the start; n2 is the number of participants at the end; yt1 is the average meeting duration; yt1 is the set meeting duration threshold; β is the video conference attention index; γ is the video conference interaction index; t i The duration of the meeting for participant i.

[0041] The data feedback module is used for the evaluation and measurement of video conferences;

[0042]

[0043] in For evaluation level, when A value of -1 indicates a poor evaluation level for the video conference; when... A value of 0 indicates that the video conference evaluation level is good; when... A value of 1 indicates that the video conference evaluation level is excellent; and The evaluation threshold is set.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention constructs a video conferencing attention index to measure the level of engagement of participants during video conferences. The video conferencing attention index includes three factors: the duration of participation in the video conference, the duration of screen-gazing, and the duration of note-taking. It also calculates the ratio of screen-gazing and note-taking time to the total participation time, introduces an exponential function to give this ratio a non-linear variation, and adjusts the weights of screen-gazing and note-taking time by adjusting the index to accurately measure participants' attention levels during video conferences. This invention, by acquiring the duration of different participant behaviors and introducing linear and non-linear variations, constructs a video conferencing attention index that can accurately measure the level of engagement of each participant.

[0046] 2. This invention measures the video conference interaction index by using the voice interaction index and the message interaction index of participants during the video conference. The voice interaction index is determined by comparing the duration of voice interaction with a set voice duration threshold; the message interaction index is determined by comparing the number of text messages left by participants with a set voice duration threshold. The voice interaction index is weighted using voice adjustment parameters, the message interaction index is weighted using message adjustment parameters, and the product of the voice interaction index and message interaction index is weighted using cross-adjustment parameters to obtain the final video conference interaction index. By calculating the video conference interaction index using the voice interaction index and message interaction index, and considering their cross-influence, the interaction status of the video conference can be accurately reflected.

[0047] 3. This invention proposes a video conference evaluation index to measure the quality of video conferences. This index considers the impact of changes in the number of participants, average participation time, attention index, and interaction index during the video conference. Regarding changes in the number of participants, the ratio of the difference between the number of participants at the start and end of the video conference to the total number of participants is used as the number of participants influencing factor. Regarding the average participation time, the ratio of the average participation time of participants during the video conference to a set threshold is used as the influencing factor. These factors are then combined with the attention index and the interaction index. The attention index can effectively and comprehensively reflect the overall quality of the video conference, providing feedback to the host of the video conference. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the remote video conferencing system of the present invention;

[0049] Figure 2 This is a schematic diagram of the video conferencing attention index calculation process of the present invention;

[0050] Figure 3 This is a schematic diagram of the participant attention recognition model structure of the present invention;

[0051] Figure 4 This is a schematic diagram of the ECA layer in the participant attention recognition model of the present invention;

[0052] Figure 5 This is a schematic diagram of the video conferencing voice interaction index calculation process of the present invention;

[0053] Figure 6 This is a schematic diagram of the video conferencing message interaction index calculation process of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The rapid development of network video technology is constantly empowering traditional industries, injecting new vitality into the survival and development of enterprises. When network video technology first became popular, it primarily focused on video conferencing. Video conferencing could overcome spatial limitations, allowing members located in different places to communicate face-to-face, greatly saving the time and effort spent organizing meetings. It also facilitated knowledge dissemination and learning by enabling participants from all over the world to exchange ideas. However, due to the limited information transmission capabilities of the video conferencing systems used—primarily consisting of video images and audio—and the lack of attention and analysis of participant behavior, the host could not clearly perceive the participants' status and reactions during remote meetings, resulting in a lack of effective feedback on the meeting's quality.

[0056] Based on the above factors, the present invention provides a remote video conferencing system.

[0057] The remote video conferencing system includes a data acquisition module, an image recognition module, a voice recognition module, a message recognition module, a data middleware module, and a data feedback module.

[0058] The data acquisition module is used to collect data information of participants during video conferences, including the participants' client video and the participants' registered identity information.

[0059] The image recognition module is used for image recognition of the participants to obtain the time the participants appear on the screen during the video conference, the time the participants gaze at the screen, and the time the participants take notes; the image recognition module includes a participant attention recognition model;

[0060] The speech recognition module is used for speech recognition of the participants and to calculate the duration of speech interactions; the speech recognition module includes a speech recognition model.

[0061] The message recognition module is used to recognize the text messages left by the participants and to count and store the number of messages.

[0062] The data platform module is used for data storage and index calculation; the indices include a video conference attention index, a video conference interaction index, and a video conference evaluation index; the video conference attention index measures the level of participation of participants; the video conference interaction index indicates the level of interaction among participants during the video conference; and the video conference evaluation index is used to evaluate the quality of the video conference.

[0063] The data feedback module provides feedback on the video conference based on relevant indices.

[0064] Example 1;

[0065] This invention identifies and calculates the attention index of video conferences using a participant attention recognition model.

[0066] The calculation process for the video conferencing attention index is as follows: Figure 2 As shown:

[0067] The system acquires video data from attendees' clients via the data acquisition module as the initial video file; it reads the initial video file frame by frame to obtain a set of video frame images; it then uses a convolutional neural network to identify the set of video frame images containing human faces in the video file, records them, and converts them into a duration t. i1 The video frames showing faces are organized into a separate facial video frame image set; the facial video frame image set is then identified using the participant attention recognition model; the set of direct-view video frame images showing participants looking directly at the screen is obtained, recorded, and converted into a duration t. i2 ; Obtain a set of video frames of attendees writing notes from a set of facial video frames, record and convert them into a duration t i3 ;

[0068] The collected data is used to measure the level of attention during video conferences.

[0069] The image recognition module includes a participant attention recognition model; the structure of the model is as follows: Figure 3 As shown.

[0070] The attendee attention recognition model comprises an image input layer, a first training module, a second training module, a third training module, a fully connected layer, and a result output layer; wherein the first training module comprises two 3x3 convolutional layers, an ECA optimization layer, and a max pooling layer; the second training module comprises two 3x3 convolutional layers and a max pooling layer; and the third training module comprises two 3x3 convolutional layers, an ECA optimization layer, and a max pooling layer.

[0071] The process by which the participant attention recognition model identifies the facial video frame image set is as follows:

[0072] The set of facial video frames is imported into the image input layer to obtain image set M1; image set M1 is imported into the first training module for training to obtain image set M2; image set M2 is imported into the second training module for training to obtain image set M3; image set M3 is imported into the third training module for training to obtain image set M4; image set M4 is imported into the fully connected layer for processing to obtain image set M5; image set M5 is imported into the result output layer for judgment, and the judgment result is output.

[0073] The ECA optimization layer includes an ECA attention mechanism module. This module effectively avoids dimensionality reduction and identifies cross-channel interactions. The core idea is to introduce channel attention into convolutional operations to capture the relationships between different channels, thereby improving feature representation capabilities. The goal of the channel attention mechanism is to adaptively adjust the weights of channel features, allowing the network to better focus on important features and suppress unimportant ones. This mechanism effectively enhances the network's representational capabilities without increasing the number of parameters or computational cost. The structure of the ECA optimization layer is as follows: Figure 4 As shown.

[0074] The implementation process of the ECA attention mechanism module is as follows:

[0075] The feature map to be trained is globally averaged to transform it into a feature vector. An adaptive one-dimensional convolution kernel size is calculated based on the number of channels in the feature vector. The one-dimensional convolution kernel size is used to calculate the weights for each channel of the feature vector. The normalized weights are multiplied by the input feature map according to the channels to obtain a weighted feature map.

[0076] The video conferencing attention index is calculated based on the data collected through the above process.

[0077] The formula for the video conferencing attention index is:

[0078]

[0079] Where β is the video conference attention index; i represents the number of participants; n is the total number of participants; t0 represents the total duration of the video conference; t i1 The duration of time that participant i appears on the screen during a video conference; t i2 The duration of time that participant i's gaze is fixed on the screen when i appears on the screen; t i3 The time for taking notes when participant i appears on the screen; α1 is the first adjustment parameter; α2 is the second adjustment parameter.

[0080] Table 1 shows the participation information of three participants, A, B, and C, in a video conference.

[0081] Table 1. Example Data for Calculating Video Attention Index

[0082] Total video duration Time of arrival of attendees Screen time Notes recording time A 1800s 1800s 1262s 350s B 1800s 1400s 658s 400s C 1800s 804s 628s 120s

[0083] Specifically, the first adjustment parameter is set to 0.7, and the second adjustment parameter is set to 0.3.

[0084] Based on the constructed formula for the video conferencing attention index, we can derive:

[0085] The video conference attention index for participant A is:

[0086]

[0087] The video conference attention index for participant B is:

[0088]

[0089] The video conference attention index for participant C is:

[0090]

[0091] The overall video conferencing attention index is:

[0092]

[0093] This invention constructs a video conferencing attention index to measure the level of engagement of participants during video conferences. The video conferencing attention index considers three factors: the duration of participation in the video conference, the duration of screen-gazing, and the duration of note-taking. It also calculates the ratio of screen-gazing and note-taking time to the total duration of participation, and introduces an exponential function to give this ratio a non-linear variation. Furthermore, by adjusting the index, the weights of screen-gazing and note-taking time are adjusted to accurately measure participants' attention levels during video conferences.

[0094] Example 2;

[0095] This invention measures the video conference interaction index by measuring the voice interaction index and message interaction index of participants during the video conference.

[0096] The process of the speech recognition module for recognizing the video and speech of the participants' clients is as follows: Figure 5 As shown:

[0097] Obtain the audio information from the participants' video feeds and compile it into an audio dataset;

[0098] The audio dataset is preprocessed and features are extracted to obtain a preprocessed audio dataset;

[0099] Identify the voice of the meeting host in the preprocessed audio dataset and mask the host's voice to obtain the audio dataset of the participants;

[0100] The audio dataset of attendees was identified using a speech recognition model for attendees.

[0101] If the sound of the identified participant audio dataset has not appeared before, the sound is encoded and timed.

[0102] If the sound of the identified participant audio dataset has already appeared and been encoded, then the sound is timed and added to the previous time record;

[0103] The voice interaction index is calculated based on the obtained data.

[0104] The speech recognition model for the attendees was an improved ResNet34 model, which introduced an SE attention mechanism module after each residual structure of the classic ResNet34 model.

[0105] When recognizing video and audio from attendees' clients, issues such as aliasing and high-frequency distortion can easily occur due to interference from network equipment, the surrounding environment, and the speaker. Directly recognizing the original audio data can lead to inaccurate recognition and low data utilization. Therefore, it is necessary to clean and preprocess the audio data to improve the accuracy and effectiveness of speech recognition.

[0106] The preprocessing steps of the speech recognition module for the audio dataset include pre-emphasis, frame segmentation, windowing, Fourier transform, Mel filter processing, discrete cosine transform, and difference processing.

[0107] The audio dataset is pre-emphasized to obtain audio dataset P1;

[0108] The audio dataset P1 is divided into frames to obtain the audio dataset P2;

[0109] The audio dataset P2 is windowed to obtain the audio dataset P3;

[0110] The audio dataset P3 is subjected to Fourier transform to obtain the audio dataset P4;

[0111] The audio dataset P4 is processed by a Mel filter to obtain the audio dataset P5;

[0112] The audio dataset P5 is subjected to discrete cosine transform to obtain audio dataset P6.

[0113] The audio dataset P6 is differentially processed to obtain a preprocessed audio dataset.

[0114] Pre-emphasis: Voice signal transmission lines exhibit low-pass filtering characteristics, where high-frequency components of the signal attenuate significantly during transmission, while low-frequency components attenuate less. Furthermore, the signal frequency is primarily determined by the rate of change of the signal level, so high-frequency components mainly appear at the rising and falling edges of the signal. The idea behind pre-emphasis technology is to enhance the high-frequency components of the signal at the beginning of the transmission line to compensate for the excessive attenuation of high-frequency components during transmission.

[0115] Frame segmentation: Since speech signals have time-varying characteristics and short-term stationarity, their characteristics can be approximated as constant over a short period of time. By analyzing and extracting feature parameters segment by segment, stable feature data can be obtained. Generally, 10ms-30ms is considered a reasonable frame length.

[0116] In practice, P sampling points are combined into one frame, and the value of P is related to the sampling frequency of the signal; for a sampling rate of QkHz, P / Q is generally set to fall within the range of 10 to 30. To avoid excessive variation between adjacent frames, an overlapping region is created between two adjacent frames. This overlapping region contains L sampling points, and the value of L is generally about 1 / 2 or 1 / 3 of P.

[0117] Windowing: After framing, in order to enhance the continuity of the data at both ends and to prevent spectrum leakage, the voice data needs to be windowed. Common windowing methods include Hanning window, rectangular window, Hamming window and Hamming window.

[0118] The Hamming window uses the Hamming function as the window function. In the time domain, it presents a shape similar to a partially symmetrical sine wave with two identical amplitude structures. The Hamming window attenuates the oscillations on both sides of the window and is relatively smooth in the frequency domain. It has better frequency response suppression and amplitude sidelobe attenuation effects than the rectangular window.

[0119]

[0120] Where N is the length of each frame; n is a constant; ω n π is the Hamming window function; π is a mathematical constant.

[0121] Discrete Fourier Transform: Since it is usually difficult to see the characteristics of a signal by its transformation in the time domain, it is usually converted into an energy distribution in the frequency domain for observation. Different energy distributions can represent the characteristics of different speech. Therefore, after multiplying by the Hamming window, each frame must undergo a Fast Fourier Transform to obtain the energy distribution in the spectrum.

[0122] Mel filtering: Compared to normal sound frequencies, the Mel value is closer to the human ear's hearing mechanism. It increases rapidly in the low-frequency range and slowly in the high-frequency range.

[0123] Each frequency value corresponds to a Mel value, and the correspondence is as follows:

[0124]

[0125] Where f is the input Hertz frequency and M is the corresponding output Mel frequency.

[0126] The obtained speech processing results are subjected to discrete cosine transform to obtain the characteristic coefficients of MFCC.

[0127] The characteristic coefficients of MFCC can only reflect the characteristics of speech under static conditions. In order to further identify the dynamic characteristics of speech and realize the combination of static and dynamic speech information, it is necessary to perform first-order and second-order derivative operations to obtain a preprocessed dataset.

[0128] The message recognition module's process for recognizing messages left by participants during video conferences is as follows: Figure 6 As shown:

[0129] Obtain the message board of participants during the video conference and get the message data;

[0130] The message data of the participants is grouped and statistically analyzed by their IDs to obtain group message data;

[0131] Emotional text recognition is performed on the grouped message data to obtain a positive coefficient, which is used to measure the positive relevance of the message to the meeting.

[0132] The comment interaction index is calculated by the number of comments and the activity level.

[0133] The formula for the video conferencing interaction index is:

[0134] γ=μ1*γ1+μ2*γ2+μ3*γ1*γ2;

[0135]

[0136]

[0137] Where γ is the video conferencing interaction index, μ1 is the voice adjustment parameter, μ2 is the message adjustment parameter, and μ3 is the cross-adjustment parameter;

[0138] γ1 represents the voice interaction index of the participants; u i This represents the speaking time of participant i during the video conference; u0 is the set audio duration threshold; e is a natural constant.

[0139] γ2 represents the interaction index of participants' comments; c i c0 represents the number of text messages left by participant i during the video conference; c0 is the set threshold for the number of messages; ω i The positive response rate of participant i's comments.

[0140] Table 2 shows the interaction data of participants A, B, and C in a video conference:

[0141] Table 2. Examples of Data for Calculating Video Conferencing Interaction Index

[0142]

[0143]

[0144] Set the voice adjustment parameter to 0.7; set the message adjustment parameter to 0.3; set the cross adjustment parameter to 1;

[0145] According to the above formula for the video conferencing voice interaction index, we can obtain:

[0146] The formula and result for calculating the video conference voice interaction index for participant A are as follows:

[0147]

[0148] The formula and result for calculating the video conference voice interaction index for participant B are as follows:

[0149]

[0150] The formula and result for calculating the video conference voice interaction index for participant C are as follows:

[0151]

[0152] The overall video conferencing voice interaction index is:

[0153]

[0154] Based on the above formula for video conference message interaction index, we can obtain:

[0155] The formula and result for calculating the video conference comment interaction index of participant A are as follows:

[0156]

[0157] The formula and result for calculating the video conference comment interaction index of participant B are as follows:

[0158]

[0159] The formula and result for calculating the video conference comment interaction index of participant C are as follows:

[0160]

[0161] The overall video conferencing message interaction index is:

[0162]

[0163] Based on the above video conferencing voice interaction index and video conferencing message interaction index, the video conferencing interaction index can be obtained as follows:

[0164] γ=0.7*0.806+0.3*0.671+1*0.806*0.671=1.306;

[0165] This invention measures the video conference interaction index by using the voice interaction index and the message interaction index of participants during the video conference. The voice interaction index is determined by comparing the duration of voice interaction with a set voice duration threshold, while the message interaction index is determined by comparing the number of text messages left by participants with a set voice duration threshold. The video conference interaction index is obtained by weighting the voice interaction index with voice adjustment parameters, weighting the message interaction index with message adjustment parameters, and then weighting the product of the voice interaction index and the message interaction index with cross-adjustment parameters. This accurately reflects the interaction status of the video conference.

[0166] Example 3;

[0167] This invention proposes a video conferencing evaluation index to measure the quality of video conferencing.

[0168] The formula for calculating the video conferencing evaluation index is as follows:

[0169]

[0170]

[0171] in, yt1 is the video conference evaluation index; n1 is the number of participants at the start; n2 is the number of participants at the end; yt1 is the average meeting duration; yt1 is the set meeting duration threshold; β is the video conference attention index; γ is the video conference interaction index; t iThe duration of the meeting for participant i.

[0172] If a video conference starts with 30 participants and ends with 24, and the conference duration is 1800 seconds; the average participation time per participant is 1350 seconds, and the participation time threshold is 1080 seconds; the video conference attention index is 1.140; and the video conference interaction index is 1.306; then the evaluation index of the video conference is:

[0173]

[0174] The obtained video conferencing evaluation index is further rated;

[0175]

[0176] in For evaluation level, when A value of -1 indicates a poor evaluation level for the video conference; when... A value of 0 indicates that the video conference evaluation level is good; when... A value of 1 indicates that the video conference evaluation level is excellent;

[0177] The video conferencing rating is good.

[0178] The video conference evaluation index considers the impact of changes in the number of participants, average participation time, attention index, and interaction index during the video conference. Specifically, regarding changes in the number of participants, the ratio of the difference between the number of participants at the start and end of the video conference to the total number of participants is used as the number of participants influencing factor. Regarding the average participation time, the ratio of the average participation time of participants during the video conference to a set threshold is used as the influencing factor. Combined with the attention index and interaction index, this effectively reflects participants' feedback on the video conference and provides feedback for the video conference.

[0179] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A remote video conferencing system, characterized in that: The remote video conferencing system includes a data acquisition module, an image recognition module, a voice recognition module, a message recognition module, a data middleware module, and a data feedback module. The data acquisition module is used to collect data information of participants during video conferences, including the participants' client video data and the participants' registered identity information. The image recognition module is used for image recognition of the participants to obtain the time the participants appear on the screen during the video conference, the time the participants gaze at the screen, and the time the participants take notes; the image recognition module includes a participant attention recognition model; The speech recognition module is used for speech recognition of the participants and to calculate the duration of speech interactions; the speech recognition module includes a speech recognition model. The message recognition module is used to recognize the text messages left by the participants and to count and store the number of messages. The data platform module is used for data storage and index calculation; the indices include a video conference attention index, a video conference interaction index, and a video conference evaluation index; the video conference attention index measures the level of participation of attendees; the video conference interaction index indicates the level of interaction among attendees during the video conference; and the video conference evaluation index is used to evaluate the quality of the video conference. The data feedback module provides feedback on the video conference based on the video conference evaluation index.

2. The remote video conferencing system according to claim 1, characterized in that: The process for calculating the video conference attention index in the image recognition module is as follows: Obtain the video from the participants' client applications as the initial video file; The initial video file is read frame by frame to obtain a set of video frame images; The video frame image set containing a human face in the video file is identified by a convolutional neural network, and the duration of the video frame image set containing a human face is calculated. The video frames containing human faces are organized into a separate set of facial video frame images; The facial video frame image set is identified using the attendee attention recognition model. Obtain the set of video frames in the facial video frame image set where the participants' eyes are directly looking at the screen, and convert them into duration; Obtain the set of video frame images of participants writing notes in the facial video frame image set, and calculate the duration of the video frame image set of participants taking notes; The video conference attention index was calculated based on the obtained data.

3. A remote video conferencing system according to claim 2, characterized in that: The image recognition module includes a participant attention recognition model; The attendee attention recognition model comprises: an image input layer, a first training module, a second training module, a third training module, a fully connected layer, and a result output layer; wherein the first training module comprises two 3x3 convolutional layers, an ECA optimization layer, and a max pooling layer; the second training module comprises two 3x3 convolutional layers and a max pooling layer; and the third training module comprises two 3x3 convolutional layers, an ECA optimization layer, and a max pooling layer. The process by which the participant attention recognition model identifies the facial video frame image set is as follows: The set of facial video frame images is imported into the image input layer to obtain image set M1; The image set M1 is imported into the first training module for training to obtain the image set M2; The image set M2 is imported into the second training module for training to obtain the image set M3; The image set M3 is imported into the third training module for training to obtain the image set M4; The image set M4 is imported into the fully connected layer for processing to obtain the image set M5; The image set M5 is imported into the result output layer for judgment, and the judgment result is output.

4. A remote video conferencing system according to claim 1, characterized in that: The video conference attention index measures the level of engagement of participants through their performance during the video conference; the formula for the video conference attention index is: wherein β is a video conference attention index; i represents a participant; n is the total number of participants; t0represents the total duration of the video conference; t i1 is the time when the participant i appears on the screen during the video conference; t i2 is the time when the participant i gazes at the screen when appearing on the screen; t it is the time when the participant i takes notes when appearing on the screen; α1is a first adjustment parameter; and α2is a second adjustment parameter.

5. A remote video conferencing system according to claim 1, characterized in that: The process by which the speech recognition module performs speech recognition on the video from the participants' client devices is as follows: Obtain the audio information from the participants' video feeds and compile it into an audio dataset; The audio dataset is preprocessed to obtain a preprocessed audio dataset; Identify the voice features of the meeting host in the preprocessed audio dataset, and mask the voice features of the host to obtain the audio dataset of the participants; The audio dataset of attendees was identified using a speech recognition model for attendees. If the sound of the identified participant audio dataset has not appeared before, the sound is encoded and timed; if the sound of the identified participant audio dataset has appeared and been encoded, the sound is timed and added to the previous time record. The video conferencing interaction index is calculated based on the collected data.

6. A remote video conferencing system according to claim 5, characterized in that: The preprocessing steps of the speech recognition module for the audio dataset include pre-emphasis, frame segmentation, windowing, Fourier transform, Mel filter processing, discrete cosine transform, and difference processing. The audio dataset is pre-emphasized to obtain audio dataset P1; The audio dataset P1 is divided into frames to obtain the audio dataset P2; The audio dataset P2 is windowed to obtain the audio dataset P3; The audio dataset P3 is subjected to Fourier transform to obtain the audio dataset P4; The audio dataset P4 is processed by a Mel filter to obtain the audio dataset P5; The audio dataset P5 is subjected to discrete cosine transform to obtain audio dataset P6. The audio dataset P6 is differentially processed to obtain a preprocessed audio dataset.

7. A remote video conferencing system according to claim 1, characterized in that: The process by which the message recognition module identifies the interaction index of participants' messages during a video conference is as follows: Obtain the message board of participants during the video conference and get the message data; The message data of the participants is grouped and statistically analyzed by their IDs to obtain group message data; The grouped message data is subjected to sentiment text recognition to obtain a positive coefficient; The comment interaction index is calculated by the number of comments and the activity level.

8. A remote video conferencing system according to claim 1, characterized in that: The video conferencing interaction index is calculated using the voice interaction index and the message interaction index, taking into account the synergistic effect of the voice interaction and the message interaction. The formula for the video conferencing interaction index is: γ=μ1*γ1+μ2*γ2+μ t *γ1*γ2; Where γ is the video conferencing interaction index, and γ1 represents the voice interaction index of the participants; u i γ represents the speaking time of participant i during the video conference; u0 is the set audio duration threshold; γ2 represents the participant's comment interaction index; c i c0 represents the number of text messages left by participant i during the video conference; c0 is the set threshold for the number of messages; ω i μ1 represents the engagement coefficient of participant i in leaving comments; μ2 represents the voice adjustment parameter; μ3 represents the comment adjustment parameter; μ4 represents the voice adjustment parameter. t is the cross-adjustment parameter; e is the natural constant.

9. A remote video conferencing system according to claim 1, characterized in that: The video conferencing evaluation index takes into account the changes in the number of participants during the video conference, the average participation time of the video conference, the video conference attention index, and the video conference interaction index, and is used to measure the quality of the video conference. The formula for calculating the video conferencing evaluation index is as follows: Where θ is the video conference evaluation index; n1 is the number of participants at the start; n2 is the number of participants at the end; yt1 is the average meeting duration; yt1 is the set meeting duration threshold; β is the video conference attention index; γ is the video conference interaction index; t i The duration of the meeting for participant i.

10. A remote video conferencing system according to claim 1, characterized in that: The data feedback module is used for the evaluation and measurement of video conferences; in For evaluation level, when A value of -1 indicates a poor evaluation level for the video conference; when... A value of 0 indicates that the video conference evaluation level is good; when... A value of 1 indicates that the video conference evaluation level is excellent; θ1 and θ2 are the set evaluation thresholds.

Images