Audio and video system integrated conference system

Abstract

The invention discloses an audio and video system integrated conference system, which belongs to the technical field of remote conferences, and comprises an information authentication module, an audio transmission module, a video transmission module, a conference effect detection module and a comprehensive summary module, the audio transmission module comprises audio acquisition equipment which is responsible for capturing, transmitting and processing audio signals in a conference so as to ensure that participants can clearly hear the sound of each other, and the video transmission module comprises a high-definition camera which is responsible for capturing, transmitting and displaying video signals. According to the audio and video system integrated conference system, audio and video transmission, concentration degree detection and conference content summarization can be realized, the interactivity and effect of a conference are effectively improved, the effect of the audio and video conference is improved, and the participation degree of participants in the conference process is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote conferences, and in particular is an audio and video system integrated conference system. Background Art

[0002] Audio and video system integrated conference is a conference system that uses various advanced equipment and technologies to achieve clear audio transmission, high-quality video display and interactive conference functions. Audio and video system integrated conference is widely used in enterprises, governments, education and other fields for remote meetings, training, seminars and other occasions to save time and cost and improve work efficiency.

[0003] The existing audio and video system integrated conference system, when in use, due to the large number of meeting participants, the speaker of the meeting cannot grasp the concentration of each participant during the meeting, and cannot obtain the participation effect of the current meeting according to the concentration of the participants during the meeting, and cannot make timely adjustments according to the real-time situation on site, and has the problem of low practicality. In addition, it is impossible to properly remind participants with low concentration during the meeting, and it is impossible to make a real-time summary of the meeting content of the speaker, and has the problem of low functionality.

[0004] In view of the above, this case proposes an audio and video system integrated conference system to solve the above technical problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an audio and video system integrated conference system, which solves the above technical problems by improving the detection method and the processing method.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An audio and video system integrated conference system, including an information authentication module, an audio transmission module, a video transmission module, a conference effect detection module, and a comprehensive summary module;

[0008] The information authentication module registers a login account based on the personal information of the participants and sets a permission granting function;

[0009] The audio transmission module, including an audio acquisition device, is responsible for capturing, transmitting and processing audio signals during a conference to ensure that participants can clearly hear each other's voices;

[0010] The video transmission module includes a high-definition camera, which is responsible for capturing, transmitting and displaying video signals, ensuring that participants can clearly see each other's video images, and transmitting data to subsequent modules;

[0011] The conference effect detection module detects the conference effect of the participants based on the relevant information of the audio transmission module and the video transmission module, and evaluates the concentration of the participants by extracting the line of sight of the participants;

[0012] The comprehensive summary module collects the meeting content of the speaker and combines the information of the audio transmission module and the video transmission module to summarize and generalize the meeting content, and transmits the summary content to the participant's account after the meeting.

[0013] Furthermore, the conference effect detection module detects the conference effect of the participants based on the relevant information of the audio transmission module and the video transmission module, and evaluates the concentration of the participants by extracting the sight lines of the participants, including the following steps:

[0014] The high-definition camera of the participant's device is used to capture images in real time, and the participant's facial area is identified based on the facial detection algorithm. The eye area is located in the facial image, the pupil center is extracted, and the sight direction is calculated based on the eye position and the pupil center. At the same time, the speaker's screen area is determined on the conference interface, and a two-dimensional coordinate system is established to map the participant's sight direction to the corresponding position on the speaker's screen.

[0015] Based on the gaze duration and the duration of the meeting, the concentration of the relevant participants is judged and graded. At the same time, the concentration of all participants is collected, the concentration ratio is calculated and transmitted to the speaker interface, and the meeting effect is scored.

[0016] Furthermore, the high-definition camera of the participant's device is used to capture images in real time, the participant's facial area is identified based on the facial detection algorithm, the eye area is located in the facial image, the pupil center is extracted, and the line of sight direction is calculated based on the eye position and the pupil center. The specific steps are as follows:

[0017] Capture the participants’ images in real time using a high-definition camera F , extract the facial region R of the participant in the image based on the Harr cascade classifier F , and its algorithm formula is:

[0018]

[0019] Where λ represents the Harr eigenvalue, is the sum of pixel intensities in the rectangular area A, is the sum of the pixel intensities of the rectangular area B. The facial feature information is extracted by comparing the pixel values ​​of different areas. The cascade classifier is trained by the AdaBoost algorithm. The sliding window is moved in the image and the classifier is applied in each window. The feature value of each window area is passed to the cascade classifier for face detection and the facial area R in the image is identified. F ;

[0020] R by Harr cascade classifier F The image within the eye region R E Detection, where the eye region R E It is a rectangular box, indicating the position of the eyes in the facial area;

[0021] Extract the eye area R by using the Canny edge detection algorithm E The edge features of the pupil are detected, and the edge of the pupil is found in the edge detection result. The Hough transform is applied to the edge detection result to detect the circular edge and determine the center position of the pupil. The specific algorithm formula is:

[0022]

[0023] Among them, E 0 (x 0 ,y 0 ) represents the pupil center coordinates, x and y are the center coordinates of the circle in the Hough circle transform, r represents the radius of the circle in the Hough circle transform, and v (x,y,r) It is the accumulated votes of the center coordinates (x, y) and radius r in the Hough circle transform, which is used to detect the center position of the circle;

[0024] By calculating the center position of the eye and the center position of the pupil, the line of sight direction is obtained, and the eye area R is calculated. E The geometric center of the eye is obtained by 1 (x 1 ,y 1 ), the algorithm formula is:

[0025]

[0026] Among them, x left 、x right Represents the x-coordinate of the left edge of the eye area and the x-coordinate of the right edge of the eye area, y top ,y bottom Represent the y coordinates of the upper and lower boundaries of the eye area, respectively, and the sight direction is the vector between the center of the pupil and the center of the eye:

[0027] θ=(x 0 -x 1 ,y0 -y 1 );

[0028] Among them, θ is the line of sight direction, x 0 represents the x-coordinate of the pupil center, x 1 Represents the x-coordinate, y-coordinate of the geometric center of the eye area 0 represents the y coordinate of the pupil center, y 1 Represents the y-coordinate of the geometric center of the eye region.

[0029] Furthermore, the screen area of ​​the speaker is determined on the conference interface, a two-dimensional coordinate system is established, and the sight direction of the participant is mapped to the corresponding position on the speaker's screen, including the following steps:

[0030] Identify the speaker's screen area R from the conference screen S ,in:

[0031] R S =(x l ,y t ,x r ,y b );

[0032] Among them, x l ,y t ,x r ,y b Respectively represent the left boundary x-coordinate of the speaker area, the upper boundary y-coordinate of the speaker area, the right boundary x-coordinate of the speaker area, and the lower boundary y-coordinate of the speaker area, and establish a coordinate system;

[0033] Based on the line of sight θ, the line of sight of the participant is mapped to the position on the speaker's screen. The algorithm formula is:

[0034] Y=(x 原 + k·(x 0 -x 1 ),y 原 + k·(y 0 -y 1 ));

[0035] Among them, Y represents the mapping point, x 原 ,y 原 represents the origin of the coordinate system of the speaker's screen area, and k is the scaling factor used to adjust the size of the sight direction to match the size of the speaker's screen;

[0036] For the mapping point coordinates, the boundary is determined when:

[0037] x l ≤x 原 + k·(x 0 -x1 )≤x r And y t ≤y 原 + k·(y 0 -y 1 )≤y b When the above conditions are not met, it means that the participant's line of sight is within the speaker's screen area. When the above conditions are not met, it means that the participant's line of sight is not within the speaker's screen area.

[0038] Furthermore, based on the gaze duration and the duration of the meeting, the concentration of the relevant participants is judged and graded. At the same time, the concentration of all participants is collected, the concentration ratio is calculated and transmitted to the speaker interface, and the meeting effect is scored. The specific steps are as follows:

[0039] When the mapping point coordinates satisfy:

[0040] x l ≤x 原 + k·(x 0 -x 1 )≤x r And y t ≤y 原 + k·(y 0 -y 1 )≤y b When the participant's gaze starts and ends, the duration of the gaze Δt is obtained:

[0041] Δt=t 结束 -t 开始 ;

[0042] The cumulative duration of all gaze events of the current participant is calculated using the following algorithm:

[0043]

[0044] Among them, T represents the total fixation time of the relevant participants, Δt i represents the duration of the ith gaze event, n is the total number of gaze events, and records the current meeting duration Δt 会议 =t 会议开始 -t 当前时间 , calculate the current concentration of relevant participants:

[0045]

[0046] Among them, F is the current concentration of the relevant participants, and the concentration of the relevant participants is determined based on the value of F:

[0047] When 80<F≤100, it means that the relevant participants have high concentration;

[0048] When 60<F≤80, it means that the concentration of the relevant participants is moderate;

[0049] When F≤60, it means that the concentration of the relevant participants is low;

[0050] Record the number of people with high concentration A and the number of people with medium concentration B, and calculate the proportion L based on the total number of people participating in the meeting P to score the current meeting effect and transmit it to the speaker interface. The algorithm formula is:

[0051]

[0052] When 70<L≤100, it means the current meeting effect is excellent;

[0053] When 50<L≤70, it means the current meeting effect is average;

[0054] When L≤50, it means that the current meeting effect is very poor.

[0055] Furthermore, a reminder system is also provided in the meeting effect detection module, which collects meeting content during the meeting and generates simple judgment questions, which are sent to the interface of participants with low participation for interactive reminders. The specific steps are as follows:

[0056] Collect meeting content in real time during the meeting, and convert the content through audio-to-text technology and natural language processing technology. Use the TxtRank algorithm to identify key points and important information in the meeting content and extract important statements or information.

[0057] Generate judgment questions by pre-defining templates and filling in templates based on information or statements in the extracted text. For relevant participants with F≤60, questions are sent at 5-minute intervals and the accounts of relevant participants are marked. The specific steps are as follows:

[0058] The time when each participant receives a question is recorded, and each question is numbered. The time when any three questions are received is determined:

[0059]

[0060] When ΔT=10, it means that the relevant participant has received problem reminders three times in a row, and the abnormal signal is transmitted to the information authentication module, and the relevant participant is marked in red. Indicates the time when the question is currently received. Representative Move the time back twice to get the question;

[0061] Record the total number of questions K received by the relevant participants in the meeting 总 ,in:

[0062] K 总 =K 1 +...+K i ;

[0063] Among them, K i Represents the problem of obtaining, when K 总 When the value is greater than 6, the abnormal transmission signal enters the information authentication module and the relevant participants are marked in red.

[0064] Furthermore, the comprehensive summary module collects the meeting content of the speaker and combines the information of the audio transmission module and the video transmission module to summarize and generalize the meeting content, and transmits the summary content to the participant's account after the meeting. The specific steps are as follows:

[0065] The audio data of the speaker in the meeting is collected through the audio transmission module, and the tone in the audio is extracted for feature extraction. The intensity of emotion is analyzed based on the change of tone. Keywords are extracted using the natural language processing model and associated with the time point of the tone change. The information extracted from the tone and text is combined to generate the key summary text of the meeting using the natural language generation model.

[0066] According to the information of the information authentication module, the email address of the meeting participant is obtained, and the meeting key summary text is sent to the email address of the meeting participant via email after the meeting.

[0067] Furthermore, the audio data of the speaker in the meeting is collected through the audio transmission module, the tone in the audio is feature extracted, the intensity of emotion is analyzed based on the change of the tone, the keywords are extracted using the natural language processing model, and the keywords are associated with the time point of the tone change, and the information extracted from the tone and text is combined to generate the key summary text of the meeting using the natural language generation model. The specific steps are:

[0068] The audio data of the speaker in the conference is collected through the audio transmission module, and the tone extraction is performed on the audio data, including the following steps:

[0069] Divide the audio signal into short time windows, apply Fourier transform to each window, and get the spectrum information, involving the following formula:

[0070]

[0071] Among them, X(t,f) represents the spectrum information at time t and frequency f, x(n) represents the value of the audio signal at time point n, ω(nt) represents the window function, and e -2jπfn is a complex exponential function, n represents the discrete time index;

[0072] By analyzing the spectrum, we can get the fundamental frequency, that is, the pitch:

[0073]

[0074] Where R(τ) represents the autocorrelation function, x(t) is the value of the audio signal at time point t, x(t+τ) represents the value of the audio signal at time point t+τ, τ is the time delay, and N represents the total number of samples of the signal;

[0075] Find the first major peak of the autocorrelation function R(τ), i.e. the delay τ corresponding to the maximum value max , calculate the fundamental frequency F 0 :

[0076]

[0077] Based on the obtained fundamental frequency data, detect fluctuations and mutations in pitch:

[0078] ΔF 0 (t)=|F 0 (t)-F 0 (t-Δt)|;

[0079] Where, ΔF 0 (t) represents the pitch change at time t, F 0 (t) represents the fundamental frequency at time t, F 0 (t-Δt) is the fundamental frequency at time t-Δt, where Δt represents the time interval and is used to calculate the pitch change;

[0080] The conference audio is converted into text through the automatic speech recognition system ASR in the natural language processing model NLP. Keywords are extracted from the transcribed text through TF-IDF. The time point of the tone change is aligned with the time point of the text extracted by the keyword. The key content is identified by combining the tone change and keyword extraction results:

[0081] S(i)=α·ΔF 0 (i) + β·T(i);

[0082] Among them, S(i) represents the key content score of the i-th time period, ΔF 0 (i) represents the change in tone in the i-th time period, T(i) represents the TF-IDF score of the keyword in the i-th time period, α and β represent weight coefficients, and the main content of the meeting is extracted according to the key content score. A coherent summary is generated through the natural language generation model NLG, i.e., the key summary text of the meeting.

[0083] Furthermore, the information authentication module registers a login account based on the personal information of the participants, including mobile phone numbers and email addresses, issues permissions to special accounts through the administrator account, including speakers and special guests, and sets permissions granting functions, including speaking permissions and screen sharing permissions;

[0084] The audio transmission module includes an audio acquisition device and a microphone array, which is responsible for capturing, transmitting and processing audio signals during the meeting. It also integrates noise elimination and echo suppression technologies, and automatically adjusts audio transmission parameters according to network conditions during use to avoid delays and packet loss, ensuring that participants can clearly hear each other's voices;

[0085] The video transmission module includes a high-definition camera, which is responsible for capturing and transmitting high-definition video through the management and control of the high-definition camera, ensuring that the participants can clearly see each other's video images, and transmitting video data into the conference effect detection module.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] 1. In the present invention, by setting up a conference effect detection module, the line of sight direction of the participants in the conference process is obtained, and the line of sight position of the participants is analyzed in combination with the speaker's picture, and the concentration of the participants in the conference process is evaluated, so as to objectively evaluate the concentration of the participants in the conference process, and mobilize the subjective initiative of the participants through real-time monitoring;

[0088] 2. In the present invention, by collecting the concentration of all participants, the current meeting effect is scored during the meeting, the effect of the entire meeting is obtained in real time, and transmitted to the speaker interface, so that the speaker can grasp the rhythm of the meeting according to the actual situation of the meeting and adjust the depth and rhythm of the meeting content;

[0089] 3. In the present invention, by timely acquiring concentration data, the presenter can effectively control and maintain the concentration of the participants, reduce distractions during the meeting, further improve the quality of the meeting, and enhance the functionality and practicality of the system.

[0090] 4. In the present invention, by classifying the concentration of participants during the meeting, simple judgment questions related to the meeting are generated for participants with low participation at fixed time intervals and sent to the participant interface, which can attract the attention of participants when their concentration decreases and help them refocus their attention, thereby improving the participation of the meeting;

[0091] 5. In the present invention, for participants who have been inattentive for many times, the account files of the relevant participants are marked, which facilitates further communication between the speaker and these participants and helps the conference organizer and the speaker to better manage and adjust the conference;

[0092] 6. In the present invention, by setting up a comprehensive summary module, a summary text of the key points of the meeting is generated based on the speaker's tone and the meeting content and transmitted to the mailbox of each participant, providing the participants with a clear and concise meeting summary, which helps them to quickly review and understand the core content of the meeting;

[0093] The entire audio and video system integrates the conference system, which can realize audio and video transmission, concentration detection, and meeting content summary, effectively improving the interactivity and effectiveness of the meeting, helping to improve the effectiveness of audio and video conferencing, and increasing the participation of participants in the meeting. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 The present invention is a block diagram of an audio and video system integrated conference system. DETAILED DESCRIPTION

[0095] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0096] like Figure 1 As shown, an audio and video system integrated conference system includes an information authentication module, an audio transmission module, a video transmission module, a conference effect detection module, and a comprehensive summary module;

[0097] Information authentication module, registering and logging in accounts based on the personal information of participants, and setting up permission granting functions;

[0098] Information authentication module, based on the personal information of participants, register and log in accounts, including mobile phone numbers and email addresses, and grant permissions to special accounts, including speakers and special guests, through administrator accounts, and set permission granting functions, including speaking permissions and screen sharing permissions;

[0099] The audio transmission module, including the audio acquisition device, is responsible for capturing, transmitting and processing audio signals during the meeting to ensure that participants can hear each other clearly;

[0100] The audio transmission module, including audio acquisition equipment and microphone array, is responsible for capturing, transmitting and processing audio signals during the meeting. It also integrates noise elimination and echo suppression technologies. During use, it automatically adjusts audio transmission parameters according to network conditions to avoid delays and packet loss, ensuring that participants can hear each other clearly.

[0101] The video transmission module includes a high-definition camera. Through the management and control of the high-definition camera, it is responsible for capturing and transmitting high-definition videos, ensuring that participants can clearly see each other's video images, and transmitting video data to the conference effect detection module;

[0102] The conference effect detection module detects the conference effect of the participants based on the relevant information of the audio transmission module and the video transmission module, and evaluates the concentration of the participants by extracting the line of sight of the participants, including the following steps:

[0103] The high-definition camera of the participant's device is used to capture images in real time, and the participant's facial area is identified based on the facial detection algorithm. The eye area is located in the facial image, the pupil center is extracted, and the sight direction is calculated based on the eye position and the pupil center. At the same time, the speaker's screen area is determined on the conference interface, and a two-dimensional coordinate system is established to map the participant's sight direction to the corresponding position on the speaker's screen.

[0104] The high-definition camera of the participant's device captures images in real time, identifies the participant's facial area based on the facial detection algorithm, locates the eye area in the facial image, extracts the pupil center, and calculates the line of sight direction based on the eye position and pupil center. The specific steps are as follows:

[0105] Capture the participants’ images in real time using a high-definition camera F , extract the facial region R of the participant in the image based on the Harr cascade classifier F , and its algorithm formula is:

[0106]

[0107] Where λ represents the Harr eigenvalue, is the sum of pixel intensities in the rectangular area A, is the sum of the pixel intensities of the rectangular area B. The facial feature information is extracted by comparing the pixel values ​​of different areas. The cascade classifier is trained by the AdaBoost algorithm. The sliding window is moved in the image and the classifier is applied in each window. The feature value of each window area is passed to the cascade classifier for face detection and the facial area R in the image is identified. F ;

[0108] It should be noted that image I FIt is a two-dimensional image matrix that contains pixel information of the face. The Haar feature value is calculated in each sliding window and input into the cascade classifier to determine whether it is a facial area. The Haar feature describes different parts of the image based on the difference in pixel intensity in a rectangular area. The main features include edge features, line features, and rectangle features, which are used to identify the facial area in the image.

[0109] Among them, the edge feature contains two adjacent rectangular areas, where the sum of the pixels in one area is subtracted from the sum of the pixels in the other area;

[0110] The line feature consists of three adjacent rectangular areas, where the sum of pixels in the middle rectangular area is subtracted from the sum of pixels in the rectangular areas on both sides;

[0111] The center feature contains four adjacent rectangular areas, where the difference between the pixels of the two diagonal rectangular areas is obtained.

[0112] R by Harr cascade classifier F The image within the eye region R E Detection, where the eye region R E It is a rectangular box, indicating the position of the eyes in the facial area;

[0113] Extract the eye area R by using the Canny edge detection algorithm E The edge features of the pupil are detected, and the edge of the pupil is found in the edge detection result. The Hough transform is applied to the edge detection result to detect the circular edge and determine the center position of the pupil. The specific algorithm formula is:

[0114]

[0115] Among them, E 0 (x 0 ,y 0 ) represents the pupil center coordinates, x and y are the center coordinates of the circle in the Hough circle transform, r represents the radius of the circle in the Hough circle transform, and v (x,y,r) It is the accumulated votes of the center coordinates (x, y) and radius r in the Hough circle transform, which is used to detect the center position of the circle;

[0116] By calculating the center position of the eye and the center position of the pupil, the line of sight direction is obtained, and the eye area R is calculated. E The geometric center of the eye is obtained by 1 (x 1 ,y 1 ), the algorithm formula is:

[0117]

[0118] Among them, xleft 、x right Represents the x-coordinate of the left edge of the eye area and the x-coordinate of the right edge of the eye area, y top ,y bottom Represent the y coordinates of the upper and lower boundaries of the eye area, respectively, and the sight direction is the vector between the center of the pupil and the center of the eye:

[0119] θ=(x 0 -x 1 ,y 0 -y 1 );

[0120] Among them, θ is the line of sight direction, x 0 represents the x-coordinate of the pupil center, x 1 Represents the x-coordinate, y-coordinate of the geometric center of the eye area 0 represents the y coordinate of the pupil center, y 1 Represents the y-coordinate of the geometric center of the eye region.

[0121] Determine the screen area of ​​the speaker on the conference interface, establish a two-dimensional coordinate system, and map the sight direction of the participant to the corresponding position on the speaker's screen, including the following steps:

[0122] Identify the speaker's screen area R from the conference screen S ,in:

[0123] R S =(x l ,y t ,x r ,y b );

[0124] Among them, x l ,y t ,x r ,y b Respectively represent the left boundary x-coordinate of the speaker area, the upper boundary y-coordinate of the speaker area, the right boundary x-coordinate of the speaker area, and the lower boundary y-coordinate of the speaker area, and establish a coordinate system;

[0125] Based on the line of sight θ, the line of sight of the participant is mapped to the position on the speaker's screen. The algorithm formula is:

[0126] Y=(x 原 + k·(x 0 -x 1 ),y 原 + k·(y 0 -y 1 ));

[0127] Among them, Y represents the mapping point, x原 ,y 原 represents the origin of the coordinate system of the speaker's screen area, and k is the scaling factor used to adjust the size of the sight direction to match the size of the speaker's screen;

[0128] It should be noted that the scaling factor k is an adjustment coefficient to ensure that the mapping point is within a reasonable range of the speaker's screen area, and is determined in actual use according to the length of the sight line direction and the size of the speaker's screen.

[0129] For the mapping point coordinates, the boundary is determined when:

[0130] x l ≤x 原 + k·(x 0 -x 1 )≤x r And y t ≤y 原 + k·(y 0 -y 1 )≤y b When the above conditions are not met, it means that the participant's line of sight is within the speaker's screen area. When the above conditions are not met, it means that the participant's line of sight is not within the speaker's screen area.

[0131] It should be noted that by performing boundary determination on the mapping point, it can be determined whether the participant is paying attention to the screen of the conference speaker. If the mapping point is in the speaker area, the mapping point is retained. If not, the mapping point can be ignored or processed in other ways.

[0132] Based on the gaze duration and the duration of the meeting, the concentration of the relevant participants is judged and graded. At the same time, the concentration of all participants is collected, the concentration ratio is calculated and transmitted to the speaker interface, and the meeting effect is scored. The specific steps are as follows:

[0133] When the mapping point coordinates satisfy:

[0134] x l ≤x 原 + k·(x 0 -x 1 )≤x r And y t ≤y 原 + k·(y 0 -y 1 )≤y b When the participant's gaze starts and ends, the duration of the gaze Δt is obtained:

[0135] Δt=t 结束 -t 开始 ;

[0136] The cumulative duration of all gaze events of the current participant is calculated using the following algorithm:

[0137]

[0138] Among them, T represents the total fixation time of the relevant participants, Δt i represents the duration of the ith gaze event, n is the total number of gaze events, and records the current meeting duration Δt 会议 =t 会议开始 -t 当前时间 , calculate the current concentration of relevant participants:

[0139]

[0140] Among them, F is the current concentration of the relevant participants, and the concentration of the relevant participants is determined based on the value of F:

[0141] When 80<F≤100, it means that the relevant participants have high concentration;

[0142] When 60<F≤80, it means that the concentration of the relevant participants is moderate;

[0143] When F≤60, it means that the concentration of the relevant participants is low;

[0144] Record the number of people with high concentration A and the number of people with medium concentration B, and calculate the proportion L based on the total number of people participating in the meeting P to score the current meeting effect and transmit it to the speaker interface. The algorithm formula is:

[0145]

[0146] When 70<L≤100, it means the current meeting effect is excellent;

[0147] When 50<L≤70, it means the current meeting effect is average;

[0148] When L≤50, it means that the current meeting effect is very poor.

[0149] It should be noted that by calculating the concentration of relevant participants, the meeting effect of the current participants can be evaluated. At the same time, by collecting the number of people with high and medium concentration, the current meeting effect can be scored. At the same time, it helps the speaker to master the rhythm of the meeting and adjust the depth and rhythm of the meeting content. If there are many participants with medium or low concentration, the speaker can consider introducing more interactive sessions, questions or discussions to increase the sense of participation.

[0150] The meeting effect detection module also has a reminder system, which collects meeting content during the meeting and generates simple judgment questions, which are sent to the interface of participants with low participation for interactive reminders. The specific steps are as follows:

[0151] Collect meeting content in real time during the meeting, and convert the content through audio-to-text technology and natural language processing technology. Use the TxtRank algorithm to identify key points and important information in the meeting content and extract important statements or information.

[0152] Generate judgment questions by pre-defining templates and filling in templates based on information or statements in the extracted text. For relevant participants with F≤60, questions are sent at 5-minute intervals and the accounts of relevant participants are marked. The specific steps are as follows:

[0153] The time when each participant receives a question is recorded, and each question is numbered. The time when any three questions are received is determined:

[0154]

[0155] When ΔT=10, it means that the relevant participant has received problem reminders three times in a row, and the abnormal signal is transmitted to the information authentication module, and the relevant participant is marked in red. Indicates the time when the question is currently received. Representative Move the time back twice to get the question;

[0156] Record the total number of questions K received by the relevant participants in the meeting 总 ,in:

[0157] K 总 =K 1 +...+K i ;

[0158] Among them, K i Represents the problem of obtaining, when K 总 When the value is greater than 6, the abnormal transmission signal enters the information authentication module and the relevant participants are marked in red.

[0159] It should be noted that by real-time monitoring of participants' question reception and meeting participation, participants who may have problems such as inattention can be reminded and marked in time, so that managers can adjust the meeting status of participants in time. The predefined templates are:

[0160] [Declarative sentence]; (true / false);

[0161] Whether [the information] is accurate; (true / false).

[0162] The comprehensive summary module collects the meeting content of the speaker and combines the information of the audio transmission module and the video transmission module to summarize and generalize the meeting content. After the meeting, the summary content is transmitted to the participant's account. The specific steps are as follows:

[0163] The audio data of the speaker in the meeting is collected through the audio transmission module, and the tone in the audio is extracted for feature extraction. The intensity of emotion is analyzed based on the change of tone. The natural language processing model is used to extract keywords and associate them with the time point of the tone change. Combined with the information extracted from the tone and text, the natural language generation model is used to generate the key summary text of the meeting. The specific steps are as follows:

[0164] The audio data of the speaker in the conference is collected through the audio transmission module, and the tone extraction is performed on the audio data, including the following steps:

[0165] Divide the audio signal into short time windows, apply Fourier transform to each window, and get the spectrum information, involving the following formula:

[0166]

[0167] Among them, X(t,f) represents the spectrum information at time t and frequency f, x(n) represents the value of the audio signal at time point n, ω(nt) represents the window function, and e -2jπfn is a complex exponential function, n represents the discrete time index;

[0168] It should be noted that ω(nt) represents the window function, which means the weighted function at time point t, and is often used to reduce the boundary effect. -2jπfn The complex exponential function is used to transform the signal into the frequency domain.

[0169] By analyzing the spectrum, we can get the fundamental frequency, that is, the pitch:

[0170]

[0171] Where R(τ) represents the autocorrelation function, x(t) is the value of the audio signal at time point t, x(t+τ) represents the value of the audio signal at time point t+τ, τ is the time delay, and N represents the total number of samples of the signal;

[0172] Find the first major peak of the autocorrelation function R(τ), i.e. the delay τ corresponding to the maximum value max , calculate the fundamental frequency F 0 :

[0173]

[0174] It should be noted that R(τ) represents the autocorrelation function, which indicates the autocorrelation value under the time delay τ. The first peak of the autocorrelation function is the main period of the fundamental frequency, τ max is the position of the first major peak of the autocorrelation function R(τ), i.e., the periodic delay.

[0175] Based on the obtained fundamental frequency data, detect fluctuations and mutations in pitch:

[0176] ΔF 0 (t)=|F 0 (t)-F 0 (t-Δt)|;

[0177] Where, ΔF 0 (t) represents the pitch change at time t, F 0 (t) represents the fundamental frequency at time t, F 0 (t-Δt) is the fundamental frequency at time t-Δt, where Δt represents the time interval and is used to calculate the pitch change;

[0178] The conference audio is converted into text through the automatic speech recognition system ASR in the natural language processing model NLP. Keywords are extracted from the transcribed text through TF-IDF. The time point of the tone change is aligned with the time point of the text extracted by the keyword. The key content is identified by combining the tone change and keyword extraction results:

[0179] S(i)=α·ΔF 0 (i) + β·T(i);

[0180] Among them, S(i) represents the key content score of the i-th time period, ΔF 0 (i) represents the change in tone in the i-th time period, T(i) represents the TF-IDF score of the keyword in the i-th time period, α and β represent weight coefficients, and the main content of the meeting is extracted according to the key content score. A coherent summary is generated through the natural language generation model NLG, i.e., the key summary text of the meeting.

[0181] It should be noted that α and β represent weight coefficients. A larger α value indicates that the tone change has a greater impact on the score, and a larger β value indicates that the importance of keywords has a greater impact on the score. The specific weight value can be flexibly set according to the actual situation. The commonly used ratio is α: β = 1:1. S(i) is used to measure the importance of the content in this time period. ΔF 0 (i) indicates the change in pitch during the time period, which is usually used to measure the speaker's emotion or emphasis. T(i) is an indicator used to measure the importance of a word in a specific time period. The higher the score, the more important the keyword is in the time period.

[0182] According to the information of the information authentication module, the email address of the meeting participant is obtained, and the meeting key summary text is sent to the email address of the meeting participant via email after the meeting.

[0183] The present invention discloses an audio-video system integrated conference system. When in use, the personal information of the participants is used to register and log in to an account, and the administrator account is used to issue permissions to special accounts. The audio acquisition device and microphone array are used to capture, transmit and process audio signals in the conference. At the same time, noise elimination and echo suppression technologies are integrated. During use, audio transmission parameters are automatically adjusted according to network conditions to avoid delays and packet loss, ensuring that the participants can clearly hear each other's voices. Through the management and control of high-definition cameras, the high-definition video is captured and transmitted, ensuring that the participants can clearly see each other's video images.

[0184] The high-definition camera of the participant's device captures images in real time, identifies the participant's facial area based on the facial detection algorithm, locates the eye area in the facial image, extracts the pupil center, calculates the sight direction based on the eye position and the pupil center, and determines the speaker's screen area on the conference interface. A two-dimensional coordinate system is established to map the participant's sight direction to the corresponding position on the speaker's screen, and the concentration of the relevant participants is determined and graded. At the same time, the concentration of all participants is collected, the concentration ratio is calculated and transmitted to the speaker's interface, and the conference effect is scored;

[0185] Based on the speaker's meeting content, combined with the information from the audio transmission module, and according to the speaker's tone during the meeting, the meeting content is summarized and summarized, and the summary content is transmitted to the participant's account after the meeting.

[0186] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation; the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0187] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An audio and video system integrated conference system, characterized by: It includes information authentication module, audio transmission module, video transmission module, conference effect detection module and comprehensive summary module; The information authentication module registers a login account based on the personal information of the participants and sets a permission granting function; The audio transmission module, including an audio acquisition device, is responsible for capturing, transmitting and processing audio signals during a conference to ensure that participants can clearly hear each other's voices; The video transmission module includes a high-definition camera, which is responsible for capturing, transmitting and displaying video signals, ensuring that participants can clearly see each other's video images, and transmitting data to subsequent modules; The conference effect detection module detects the conference effect of the participants based on the relevant information of the audio transmission module and the video transmission module, and evaluates the concentration of the participants by extracting the line of sight of the participants; The comprehensive summary module collects the meeting content of the speaker and combines it with the information of the audio transmission module to summarize and generalize the meeting content, and transmits the summary content to the participant's account after the meeting.

2. The audio and video system integrated conference system according to claim 1, characterized in that: The conference effect detection module detects the conference effect of the participants based on the relevant information of the audio transmission module and the video transmission module, and evaluates the concentration of the participants by extracting the sight lines of the participants, including the following steps: The high-definition camera of the participant's device is used to capture images in real time, and the participant's facial area is identified based on the facial detection algorithm. The eye area is located in the facial image, the pupil center is extracted, and the sight direction is calculated based on the eye position and the pupil center. At the same time, the speaker's screen area is determined on the conference interface, and a two-dimensional coordinate system is established to map the participant's sight direction to the corresponding position on the speaker's screen. Based on the gaze duration and the duration of the meeting, the concentration of the relevant participants is judged and graded. At the same time, the concentration of all participants is collected, the concentration ratio is calculated and transmitted to the speaker interface, and the meeting effect is scored.

3. The audio and video system integrated conference system according to claim 2, characterized in that: The high-definition camera of the participant's device captures images in real time, identifies the participant's facial area based on the facial detection algorithm, locates the eye area in the facial image, extracts the pupil center, and calculates the line of sight direction based on the eye position and pupil center. The specific steps are as follows: Capture the participants’ images in real time using a high-definition camera F , extract the facial region R of the participant in the image based on the Harr cascade classifier F , and its algorithm formula is: Where λ represents the Harr eigenvalue, is the sum of pixel intensities in the rectangular area A, is the sum of the pixel intensities of the rectangular area B. The facial feature information is extracted by comparing the pixel values ​​of different areas. The cascade classifier is trained by the AdaBoost algorithm. The sliding window is moved in the image and the classifier is applied in each window. The feature value of each window area is passed to the cascade classifier for face detection and the facial area R in the image is identified. F ; R by Harr cascade classifier F The image within the eye region R E Detection, where the eye region R E It is a rectangular box, indicating the position of the eyes in the facial area; Extract the eye area R using the Canny edge detection algorithm E The edge features of the pupil are detected, and the edge of the pupil is found in the edge detection result. Hough transform is applied to the edge detection result to detect the circular edge and determine the center position of the pupil. The specific algorithm formula is: Among them, E0(x0,y0) represents the coordinates of the center of the pupil, x and y are the coordinates of the center of the circle in the Hough circle transform, r represents the radius of the circle in the Hough circle transform, and v (x,y,r) It is the accumulated votes of the center coordinates (x, y) and radius r in the Hough circle transform, which is used to detect the center position of the circle; By calculating the center position of the eye and the center position of the pupil, the line of sight direction is obtained, and the eye area R is calculated. E The geometric center of the eye is obtained by calculating the coordinates of the eye center position E1 (x1, y1). The algorithm formula is: Among them, x left 、x right Represents the x-coordinate of the left edge of the eye area and the x-coordinate of the right edge of the eye area, y top ,y bottom Represent the y coordinates of the upper and lower boundaries of the eye area, respectively, and the sight direction is the vector between the center of the pupil and the center of the eye: θ=(x0-x1,y0-y1); Among them, θ is the line of sight direction, x0 represents the x-coordinate of the center of the pupil, x1 represents the x-coordinate of the geometric center of the eye area, y0 represents the y-coordinate of the center of the pupil, and y1 represents the y-coordinate of the geometric center of the eye area.

4. The audio and video system integrated conference system according to claim 3, characterized in that: Determine the screen area of ​​the speaker on the conference interface, establish a two-dimensional coordinate system, and map the sight direction of the participant to the corresponding position on the speaker's screen, including the following steps: Identify the speaker's screen area R from the conference screen S ,in: R S =(x l ,y t ,x r ,y b ); Among them, x l ,y t ,x r ,y b Respectively represent the left boundary x-coordinate of the speaker area, the upper boundary y-coordinate of the speaker area, the right boundary x-coordinate of the speaker area, and the lower boundary y-coordinate of the speaker area, and establish a coordinate system; Based on the line of sight θ, the line of sight of the participant is mapped to the position on the speaker's screen. The algorithm formula is: Y=(x 原 +k·(x0−x1), y 原 +k·(y0−y1)): Among them, Y represents the mapping point, x 原 ,y 原 represents the origin of the coordinate system of the speaker's screen area, and k is the scaling factor used to adjust the size of the sight direction to match the size of the speaker's screen; For the mapping point coordinates, the boundary is determined when: x l ≤x 原 +k·(x0-x1)≤x r And y t ≤y 原 +k·(y0-y1)≤y b When the above conditions are not met, it means that the participant's line of sight is within the speaker's screen area. When the above conditions are not met, it means that the participant's line of sight is not within the speaker's screen area.

5. The audio and video system integrated conference system according to claim 4, characterized in that: Based on the gaze duration and the duration of the meeting, the concentration of the relevant participants is judged and graded. At the same time, the concentration of all participants is collected, the concentration ratio is calculated and transmitted to the speaker interface, and the meeting effect is scored. The specific steps are as follows: When the mapping point coordinates satisfy: x l ≤x 原 +k·(x0-x1)≤x r And y t ≤y 原 +k·(y0-y1)≤y b When the participant's gaze starts and ends, the duration of the gaze Δt is obtained: Δt=t 结束 -t 开始 ; The cumulative duration of all gaze events of the current participant is calculated using the following algorithm: Among them, T represents the total fixation time of the relevant participants, Δt i represents the duration of the ith gaze event, n is the total number of gaze events, and records the current meeting duration Δt 会议 =t 会议开始 -t 当前时间 , calculate the current concentration of relevant participants: Among them, F is the current concentration of the relevant participants, and the concentration of the relevant participants is determined based on the value of F: When 80<F≤100, it means that the relevant participants have high concentration; When 60<F≤80, it means that the concentration of the relevant participants is moderate; When F≤60, it means that the concentration of the relevant participants is low; Record the number of people with high concentration A and the number of people with medium concentration B, and calculate the proportion L based on the total number of people participating in the meeting P to score the current meeting effect and transmit it to the speaker interface. The algorithm formula is: When 70<L≤100, it means the current meeting effect is excellent; When 50<L≤70, it means the current meeting effect is average; When L≤50, it means that the current meeting effect is very poor.

6. The audio and video system integrated conference system according to claim 5, characterized in that: The meeting effect detection module also has a reminder system, which collects meeting content during the meeting and generates simple judgment questions, which are sent to the interface of participants with low participation for interactive reminders. The specific steps are as follows: Collect meeting content in real time during the meeting, and convert the content through audio-to-text technology and natural language processing technology. Use the TxtRank algorithm to identify key points and important information in the meeting content and extract important statements or information. Generate judgment questions by pre-defining templates and filling in templates based on information or statements in the extracted text. For relevant participants with F≤60, questions are sent at 5-minute intervals and the accounts of relevant participants are marked. The specific steps are as follows: The time when each participant receives a question is recorded, and each question is numbered. The time when any three questions are received is determined: When ΔT=10, it means that the relevant participant has received problem reminders three times in a row, and the abnormal signal is transmitted to the information authentication module, and the relevant participant is marked in red. Indicates the time when the question is currently received. Representative Move the time back twice to get the question; Record the total number of questions K received by the relevant participants in the meeting 总 ,in: K 总 =K1+...+K i ; Among them, K i Represents the problem of obtaining, when K 总 When the value is greater than 6, the abnormal transmission signal enters the information authentication module and the relevant participants are marked in red.

7. The audio and video system integrated conference system according to claim 1, characterized in that: The comprehensive summary module collects the meeting content of the speaker and combines the information of the audio transmission module and the video transmission module to summarize and generalize the meeting content, and transmits the summary content to the participant's account after the meeting. The specific steps are as follows: The audio data of the speaker in the meeting is collected through the audio transmission module, and the tone in the audio is extracted for feature extraction. The intensity of emotion is analyzed based on the change of tone. Keywords are extracted using the natural language processing model and associated with the time point of the tone change. The information extracted from the tone and text is combined to generate the key summary text of the meeting using the natural language generation model. According to the information of the information authentication module, the email address of the meeting participant is obtained, and the meeting key summary text is sent to the email address of the meeting participant via email after the meeting.

8. The audio and video system integrated conference system according to claim 7, characterized in that: The audio data of the speaker in the meeting is collected through the audio transmission module, the tone in the audio is extracted, the intensity of the emotion is analyzed based on the change of the tone, the keywords are extracted using the natural language processing model, and the keywords are associated with the time point of the tone change, and the information extracted from the tone and text is combined to generate the key summary text of the meeting using the natural language generation model. The specific steps are: The audio data of the speaker in the conference is collected through the audio transmission module, and the tone extraction is performed on the audio data, including the following steps: Divide the audio signal into short time windows, apply Fourier transform to each window, and get the spectrum information, involving the following formula: Among them, X(t,f) represents the spectrum information at time t and frequency f, x(n) represents the value of the audio signal at time point n, ω(nt) represents the window function, and e -2jπfn is a complex exponential function, n represents the discrete time index; By analyzing the spectrum, we can get the fundamental frequency, that is, the pitch: Where R(τ) represents the autocorrelation function, x(t) is the value of the audio signal at time point t, x(t+τ) represents the value of the audio signal at time point t+τ, τ is the time delay, and N represents the total number of samples of the signal; Find the first major peak of the autocorrelation function R(τ), i.e. the delay τ corresponding to the maximum value max , calculate the fundamental frequency F0: Based on the obtained fundamental frequency data, detect fluctuations and mutations in pitch: ΔF0(t)=|F0(t)-F0(t-Δt); Wherein, ΔF0(t) represents the pitch change at time t, F0(t) represents the fundamental frequency at time t, F0(t-Δt) is the fundamental frequency at time t-Δt, and Δt represents the time interval, which is used to calculate the pitch change; The conference audio is converted into text through the automatic speech recognition system ASR in the natural language processing model NLP. Keywords are extracted from the transcribed text through TF-IDF. The time point of the tone change is aligned with the time point of the text extracted by the keyword. The key content is identified by combining the tone change and keyword extraction results: S(i)=α·ΔF0(i)+β·T(i); Among them, S(i) represents the key content score of the i-th time period, ΔF0(i) represents the tone change of the i-th time period, T(i) represents the keyword TF-IDF score of the i-th time period, α and β represent weight coefficients. The main content of the meeting is extracted according to the key content score, and a coherent summary is generated through the natural language generation model NLG, that is, the key summary text of the meeting.

9. The audio and video system integrated conference system according to claim 1, characterized in that: The information authentication module registers and logs in accounts based on the personal information of participants, including mobile phone numbers and email addresses, issues permissions to special accounts through administrator accounts, including speakers and special guests, and sets permission granting functions, including speaking permissions and screen sharing permissions; The audio transmission module includes an audio acquisition device and a microphone array, which is responsible for capturing, transmitting and processing audio signals during the meeting. It also integrates noise elimination and echo suppression technologies, and automatically adjusts audio transmission parameters according to network conditions during use to avoid delays and packet loss, ensuring that participants can clearly hear each other's voices; The video transmission module includes a high-definition camera, which is responsible for capturing and transmitting high-definition video through the management and control of the high-definition camera, ensuring that the participants can clearly see each other's video images, and transmitting video data into the conference effect detection module.

Images