Dynamic collaborative management method and system for audio and video conferences

Abstract

The invention relates to the technical field of audio and video conference collaborative management, in particular to an audio and video conference dynamic collaborative management method and system. An abnormal attention focus screening module in the system is based on conference participating time intervals of expression states recognized by conference participating objects and associated deviation degrees relative to conference contents in the corresponding conference participating time intervals; and dynamically screening the conference abnormal attention focus content corresponding to the current time. According to the invention, a multi-modal sensor and an intelligent algorithm are used for replacing a traditional conference management mode depending on manual subjective judgment, so that the processes of permission migration, focus capture, feedback reminding and the like are automatic, the burden of a host is obviously reduced, and the conference fluency is ensured; according to the method, the expression states of the participants are analyzed in real time and are deeply associated with the conference content, so that the abnormal focus of attention in the conference can be accurately captured, and effective data support is provided for on-site intervention, subsequent discussion and accurate redisk.

Description

Technical Field

[0001] This invention relates to the field of audio and video conferencing collaborative management technology, specifically to a dynamic collaborative management method and system for audio and video conferencing. Background Technology

[0002] With the widespread adoption of remote work and online collaboration, audio and video conferencing has become an indispensable communication tool for modern enterprises. However, traditional meeting management systems have many limitations, leading to low meeting efficiency and insufficient collaboration. These limitations are mainly manifested in rigid access control; the transfer of meeting control permissions (such as speaking rights and sharing rights) usually relies on the host's subjective judgment or participants' manual requests (such as the "raise hand" function), failing to intelligently migrate permissions based on the actual dynamics of the meeting and the real-time status of participants, frequently interrupting the meeting flow. Simultaneously, existing systems lack effective capture and analysis of nonverbal cues, failing to identify participants' emotional states (such as confusion, agreement, excitement, and boredom) in real time. Hosts struggle to fully grasp the meeting atmosphere and each participant's focus, potentially missing key feedback and exhibiting a lack of engagement monitoring. Furthermore, existing meeting records are typically simple audio / video transcriptions or shared documents, unable to link meeting content with participants' real-time reactions (facial expressions, emotions) at corresponding points. During post-meeting debriefing, it's difficult to pinpoint the key issues of concern and the locations of content that caused confusion or disagreement among participants, resulting in a disconnect between meeting content and feedback. Therefore, existing technologies have significant shortcomings. Summary of the Invention

[0003] The purpose of this invention is to provide a dynamic collaborative management method and system for audio and video conferencing to solve the problems mentioned in the background art.

[0004] To address the aforementioned technical problems, the present invention provides the following technical solution: a dynamic collaborative management method for audio and video conferencing, the method comprising:

[0005] S1. Collect conference data streams in real time through multimodal sensors, import the collected conference data streams into the permission status model for dynamic analysis of conference control permission migration objects, and lock the migration objects of conference control permissions;

[0006] S2. In real time, identify the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process, and generate the participation time interval of each participant based on the identified facial expressions; combine the meeting content involved in the meeting data stream to analyze the correlation bias of each participant with the meeting content within the participation time interval based on the identified facial expressions.

[0007] S3. Based on the attendee's facial expression status, the time interval of the meeting, and the correlation bias with the meeting content within the corresponding time interval, dynamically filter the meeting's abnormal focus content corresponding to the current time.

[0008] S4. Based on the obtained meeting focus points, mark the meeting minutes and the pre-defined hierarchical relationships between the participants, and feed back the marked meeting minutes to the collaborating participants of the obtained meeting focus points.

[0009] Furthermore, the conference data stream in S1 includes audio stream, video stream, and text stream, and the data streams corresponding to the audio stream, video stream, and text stream are collections of corresponding types of collected data that change over time.

[0010] The process of importing the collected meeting data stream into the permission status model for dynamic analysis of meeting control permission migration objects involves the following calculation formulas:

[0011] If max{w i ·f i If (t)|i∈[1,2]}≥μ, then determine P(MA→MB). t =1;

[0012] Conversely, if the result is not positive, then determine P(MA→MB). t =0;

[0013] Where MA represents the object to which the meeting control permissions belonged before the dynamic analysis of the meeting control permission migration object; MB represents the object to which the meeting control permissions belonged after the dynamic analysis of the meeting control permission migration object; P(MA→MB) t The parameter represents the execution coefficient for the migration of meeting control permissions from MA to MB; μ represents the preset migration threshold; w i The w represents the weight of the i-th type of trigger. i The value is equal to the ratio of the frequency of times the trigger condition corresponding to the i-th type of trigger is met when the meeting control permission migration behavior is triggered in the historical data to the total frequency of meeting control permission migration behavior triggers; f i f1(t) represents the function value of the i-th type of trigger function at time t; f1(t) represents the agenda node switching function. If the screen-projected text in the text stream contains two agenda nodes within the preset time period before time t, and the meeting participants bound to the agenda nodes are MA and MB respectively, then f1(t) = 1; otherwise, f1(t) = 0.

[0014] f2(t) represents the voice keyword matching function. If the keyword frequency of the participant object bound to MB is the highest in the keyword extraction results of the audio stream based on the preset time period t, and the meeting control permission object before the dynamic analysis of the meeting control permission migration object belongs to MA, then f2(t) = 1 is determined; otherwise, f2(t) = 0 is determined.

[0015] The participant with the highest execution coefficient in the corresponding migration behavior is designated as the recipient of the meeting control permissions in the meeting control permission migration relationship.

[0016] This invention enables dynamic analysis of meeting control permission migration objects based on the collected meeting data stream, solving the problems of unintelligent and unsmooth meeting control permission migration; by comprehensively analyzing multimodal data streams through a permission status model, it can dynamically lock the most suitable permission migration object in real time, realizing the automation and intelligence of permission allocation, and avoiding meeting interruptions and delays caused by traditional methods of raising hands and calling roll.

[0017] Furthermore, in the process of generating the meeting time interval for each participant based on the recognized facial expression state in S2, the screen location regions of each participant in the meeting video frame at each time point in the meeting data stream during the meeting execution are extracted. Key features are extracted from the screen location regions of the participants using OpenFace, and the intensity values ​​of three key features are extracted: AU04 furrowed eyebrows, AU07 tightened eyelids, and AU12 pulled corners of the mouth. The obtained key features are input into a trained random forest classifier to identify whether the facial expression state corresponding to the screen location region of the participant is a confused state. The time interval formed by the time points when the facial expression state recognition result of each participant is a confused state is used as the meeting time interval of the corresponding participant based on the recognized confused state.

[0018] Furthermore, the analysis of the relevance bias of each participant to the meeting content within the time interval based on recognized facial expressions involves the following calculation formula:

[0019]

[0020] Where Pj represents the relevance bias of the j-th participant to the meeting content within the time interval of the meeting based on the confusion state of identification; the meeting content in the meeting record corresponding to the j-th participant within the time interval of the meeting based on the confusion state of identification is denoted as Hj; YHj represents the number of keywords belonging to the keyword library bound to the j-th participant in the keyword extraction result of the meeting content corresponding to Hj, and the keywords contained in the keyword library bound to each participant are preset; D (j，k)This represents the bias factor of the k-th keyword in the keyword extraction results for the meeting content corresponding to Hj, which belongs to the keyword library bound to the j-th participant. Let C be the keyword that belongs to the keyword library bound to the j-th participant in the keyword extraction results for the meeting content corresponding to Hj. (j，k) The D (j，k) The value is equal to the number of C keywords in the bound keyword library. (j，k) The reciprocal of the number of attendees; N represents the total number of attendees; This represents the association bias coefficient of the keyword extraction result for the meeting content corresponding to Hj, based on the keyword library bound to the j-th participant; max{} represents the maximum value function; max{Q (j,k) |j∈[1,N]} represents the maximum value of the association bias coefficient based on the keyword library bound to each participant, representing the result of keyword extraction for the meeting content corresponding to Hj.

[0021] This invention, by recognizing facial expressions and generating meeting time intervals, can reflect the effective participation and emotional trajectory of each participant to a certain extent. By combining this with the meeting content for analysis, the concept of "association bias" is introduced, effectively integrating the emotional reactions of participants with the meeting content. This provides quantifiable data for analyzing participants' true attitudes towards specific content (such as confusion), and provides a data foundation for subsequently screening out content that receives unusual attention during meetings.

[0022] Furthermore, in the process of dynamically filtering the abnormal focus content of the meeting corresponding to the current time in S3, the preset time period based on the current time is recorded as the abnormal analysis time interval; the summary set of all participants whose meeting time intervals based on the identified confusion state intersect with the abnormal analysis time interval is recorded as the abnormal object analysis set; the confusion state meeting time interval mapping set corresponding to the abnormal object analysis set is obtained, where each element in the confusion state meeting time interval mapping set is the meeting time interval where the most recent facial expression state recognition result of the corresponding participant in the abnormal object analysis set is a confused state and is continuous; the correlation bias degree of each element in the abnormal object analysis set relative to the meeting content is calculated within the meeting time interval corresponding to the corresponding element in the confusion state meeting time interval mapping set.

[0023] Get the g-th element and its corresponding associated element in the set of confused state meeting time interval mapping. The meeting time interval corresponding to any associated element of the g-th element in the set of confused state meeting time interval mapping is a subset of the meeting time interval corresponding to the g-th element in the set of confused state meeting time interval mapping.

[0024] The following formula is used to calculate the adaptation coefficient of the meeting's abnormal focus content for each element in the set of meeting time interval mappings for confused states:

[0025] SPg = EVg / EVmax

[0026] EVg=P[g]+P[g,sum]

[0027] Where SPg represents the matching coefficient of the meeting anomaly focus content corresponding to the g-th element in the set of confusion state and meeting time interval mapping; EVg represents the comprehensive bias evaluation value of the meeting content corresponding to the g-th element in the set of confusion state and meeting time interval mapping; P[g] represents the correlation bias degree of the corresponding element of the g-th element in the set of confusion state and meeting time interval mapping within the set of anomaly object analysis, relative to the meeting content; P[g,sum] represents the sum of the correlation bias degrees of each associated element of the g-th element in the set of confusion state and meeting time interval mapping within the set of anomaly object analysis, relative to the meeting content; EVmax represents the maximum value of the comprehensive bias evaluation value of the meeting content corresponding to each element in the set of confusion state and meeting time interval mapping.

[0028] In the set of time intervals for meeting participants in a state of confusion, select the element with the smallest adaptation coefficient for the abnormal focus content of the meeting among the elements whose adaptation coefficient is greater than the preset adaptation threshold. This element is denoted as the abnormal focus mapping time interval. The meeting content in the corresponding meeting record within the abnormal focus mapping time interval is taken as the abnormal focus content of the meeting at the current time.

[0029] Based on the data generated by S2 (the time interval of confused participants and the corresponding correlation bias), this invention can intelligently filter out the "abnormal focus of attention" at the current time. For example, when most participants show confused expressions about the topic being discussed, it can enable the moderator and manager to immediately identify the difficulties and points of contention in the meeting, thus providing an opportunity to resolve them on the spot.

[0030] Furthermore, during the process of S4 feeding back the marked meeting record content to the collaborative participants who have obtained the abnormal focus of the meeting, the scheduled meeting duration corresponding to the agenda node at the current time in the projected text stream and the actual meeting duration corresponding to the agenda node at the current time are obtained.

[0031] If the actual meeting duration of the agenda node corresponding to the current time is less than or equal to the product of the scheduled meeting duration and the preset control coefficient of the agenda node corresponding to the current time in the text stream and the projected text, then when it is determined that the marked meeting record content should be fed back to the collaborative participants of the obtained meeting abnormal focus content, the corresponding collaborative participants are the participants who have the host's authority at the current time. If the actual meeting duration of the agenda node corresponding to the current time is greater than the product of the scheduled meeting duration and the preset control coefficient of the agenda node corresponding to the current time in the text stream and the projected text, then when it is determined that the marked meeting record content should be fed back to the collaborative participants of the obtained meeting abnormal focus content, the corresponding collaborative participants are the direct superiors of each element in the abnormal object analysis set, according to the preset subordinate relationship between the participants.

[0032] This invention not only marks the content of unusual focus in the meeting, but also accurately feeds the information back to the most relevant "collaborating participants" based on a pre-set subordinate relationship map (such as automatically pushing the content of a subordinate's confusion to their superior manager). This targeted and closed-loop feedback mechanism ensures that problems can be followed up and dealt with quickly, effectively improving the decision-making execution and action efficiency of the meeting.

[0033] A dynamic collaborative management system for audio and video conferencing, the system comprising:

[0034] The permission migration management module collects meeting data streams in real time through multimodal sensors, imports the collected meeting data streams into the permission status model for dynamic analysis of meeting control permission migration objects, and locks the migration objects of meeting control permissions.

[0035] The meeting content association bias analysis module identifies the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process in real time, and generates the participation time interval of each participant based on the identified facial expressions; combined with the meeting content involved in the meeting data stream, it analyzes the association bias of each participant with the meeting content within the participation time interval based on the identified facial expressions.

[0036] An abnormal focus filtering module dynamically filters abnormal focus content of the meeting based on the time interval of the attendee's facial expression state and the correlation bias with the meeting content within the corresponding time interval.

[0037] The meeting anomaly intelligent feedback module marks the obtained meeting anomaly focus content in the meeting record content and the preset subordinate relationships between various participants, and feeds back the marked meeting record content to the collaborating participants of the obtained meeting anomaly focus content.

[0038] Furthermore, the meeting content association bias analysis module includes an attendance time analysis unit and a content association bias analysis unit.

[0039] The meeting time analysis unit identifies the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process in real time, and generates the meeting time interval for each participant based on the identified facial expressions.

[0040] The content association bias analysis unit combines the meeting content involved in the meeting data stream to analyze the association bias of each participant with the meeting content within the time interval based on the recognized facial expression state.

[0041] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0042] (1) This invention replaces the traditional meeting management method that relies on subjective human judgment with multimodal sensors and intelligent algorithms, automating processes such as permission transfer, focus capture, and feedback reminders, significantly reducing the burden on the host and ensuring the smoothness of the meeting;

[0043] (2) By analyzing the facial expressions of participants in real time and deeply relating them to the meeting content, this invention can accurately capture “abnormal focus of attention” in the meeting, providing effective data support for on-site intervention, subsequent discussion and accurate review. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a schematic diagram of the structure of an audio and video conferencing dynamic collaborative management system according to the present invention;

[0046] Figure 2 This is a flowchart illustrating a dynamic collaborative management method for audio and video conferencing according to the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Please see Figures 1-2 The present invention provides a technical solution: such as Figure 1 As shown, this embodiment provides a dynamic collaborative management system for audio and video conferencing, the system comprising:

[0049] The permission migration management module collects meeting data streams in real time through multimodal sensors, imports the collected meeting data streams into the permission status model for dynamic analysis of meeting control permission migration objects, and locks the migration objects of meeting control permissions.

[0050] The meeting content association bias analysis module includes an attendance time analysis unit and a content association bias analysis unit.

[0051] The meeting time analysis unit identifies the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process in real time, and generates the meeting time interval for each participant based on the identified facial expressions.

[0052] The content association bias analysis unit combines the meeting content involved in the meeting data stream to analyze the association bias of each participant relative to the meeting content within the meeting time interval based on the recognized facial expression state.

[0053] An abnormal focus filtering module dynamically filters abnormal focus content of the meeting based on the time interval of the attendee's facial expression state and the correlation bias with the meeting content within the corresponding time interval.

[0054] The meeting anomaly intelligent feedback module marks the obtained meeting anomaly focus content in the meeting record content and the preset subordinate relationships between various participants, and feeds back the marked meeting record content to the collaborating participants of the obtained meeting anomaly focus content.

[0055] like Figure 2 As shown, this embodiment provides a method for dynamic collaborative management of audio and video conferencing, the method including:

[0056] S1. Collect conference data streams in real time through multimodal sensors, import the collected conference data streams into the permission status model for dynamic analysis of conference control permission migration objects, and lock the migration objects of conference control permissions;

[0057] The conference data stream in S1 includes audio stream, video stream and text stream. The data streams corresponding to the audio stream, video stream and text stream are collections of corresponding types of collected data that change over time.

[0058] The process of importing the collected meeting data stream into the permission status model for dynamic analysis of meeting control permission migration objects involves the following calculation formulas:

[0059] If max{w i ·f i If (t)|i∈[1,2]}≥μ, then determine P(MA→MB). t =1;

[0060] Conversely, if the result is not positive, then determine P(MA→MB). t =0;

[0061] Where MA represents the object to which the meeting control permissions belonged before the dynamic analysis of the meeting control permission migration object; MB represents the object to which the meeting control permissions belonged after the dynamic analysis of the meeting control permission migration object; P(MA→MB) t The parameter represents the execution coefficient for the migration of meeting control permissions from MA to MB; μ represents the preset migration threshold; w i The w represents the weight of the i-th type of trigger. i The value is equal to the ratio of the frequency of times the trigger condition corresponding to the i-th type of trigger is met when the meeting control permission migration behavior is triggered in the historical data to the total frequency of meeting control permission migration behavior triggers; f i f1(t) represents the function value of the i-th type of trigger function at time t; f1(t) represents the agenda node switching function. If the screen-projected text in the text stream contains two agenda nodes within the preset time period before time t, and the meeting participants bound to the agenda nodes are MA and MB respectively, then f1(t) = 1; otherwise, f1(t) = 0.

[0062] f2(t) represents the voice keyword matching function. If the keyword frequency of the participant object bound to MB is the highest in the keyword extraction results of the audio stream based on the preset time period t, and the meeting control permission object before the dynamic analysis of the meeting control permission migration object belongs to MA, then f2(t) = 1 is determined; otherwise, f2(t) = 0 is determined.

[0063] The participant with the highest execution coefficient in the corresponding migration behavior is designated as the recipient of the meeting control permissions in the meeting control permission migration relationship.

[0064] S2. In real time, identify the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process, and generate the participation time interval of each participant based on the identified facial expressions; combine the meeting content involved in the meeting data stream to analyze the correlation bias of each participant with the meeting content within the participation time interval based on the identified facial expressions.

[0065] In step S2, during the process of generating the meeting time interval for each participant based on their recognized facial expression state, the screen location regions of each participant in the meeting video frame at each time point in the meeting data stream during the meeting execution are extracted. Key features are extracted from these screen location regions using OpenFace, specifically the intensity values ​​of three key features: AU04 furrowed eyebrows, AU07 tightened eyelids, and AU12 pulled corners of the mouth. These key features are then input into a pre-trained random forest classifier to identify whether the facial expression state corresponding to the participant's screen location region is a confused state. The time intervals formed by the times when the facial expression state of each participant is identified as a confused state are used as the meeting time interval for that participant based on their recognized confused state.

[0066] In this embodiment, during the training of the random forest classifier, the intensity values ​​of the screen locations of each participant in the meeting video frame at each time point in the meeting data stream during the historical meeting execution are extracted based on the three key features: AU04 furrowed eyebrows, AU07 eyelid tightening, and AU12 pulled corners of the mouth. The preset facial expression state (whether it is a confused state) corresponding to the screen location of the corresponding participant is also obtained. The dataset corresponding to the random forest model is constructed. The dataset corresponding to the random forest model includes a training set (80% of the corresponding dataset, used to train the random forest model), a validation set (10% of the corresponding dataset, used to adjust the parameters of the trained random forest model, such as tree depth and number of trees, to find the optimal configuration), and a test set (10% of the corresponding dataset, and the test set does not participate in the training process, used to evaluate the performance of the final model).

[0067] The analysis of the relevance bias of each participant to the meeting content within the time interval based on recognized facial expressions involves the following calculation formula:

[0068]

[0069] Where Pj represents the relevance bias of the j-th participant to the meeting content within the time interval of the meeting based on the confusion state of identification; the meeting content in the meeting record corresponding to the j-th participant within the time interval of the meeting based on the confusion state of identification is denoted as Hj; YHj represents the number of keywords belonging to the keyword library bound to the j-th participant in the keyword extraction result of the meeting content corresponding to Hj, and the keywords contained in the keyword library bound to each participant are preset; D (j，k) This represents the bias factor of the k-th keyword in the keyword extraction results for the meeting content corresponding to Hj, which belongs to the keyword library bound to the j-th participant. Let C be the keyword that belongs to the keyword library bound to the j-th participant in the keyword extraction results for the meeting content corresponding to Hj. (j，k) The D (j，k) The value is equal to the number of C keywords in the bound keyword library. (j，k) The reciprocal of the number of attendees; N represents the total number of attendees; This represents the association bias coefficient of the keyword extraction result for the meeting content corresponding to Hj, based on the keyword library bound to the j-th participant; max{} represents the maximum value function; max{Q (j,k) |j∈[1,N]} represents the maximum value of the association bias coefficient based on the keyword library bound to each participant, representing the result of keyword extraction for the meeting content corresponding to Hj.

[0070] S3. Based on the attendee's facial expression status, the time interval of the meeting, and the correlation bias with the meeting content within the corresponding time interval, dynamically filter the meeting's abnormal focus content corresponding to the current time.

[0071] In the process of dynamically filtering the abnormal focus content of the meeting corresponding to the current time in S3, the preset time period based on the current time is recorded as the abnormal analysis time interval; the summary set of all participants whose meeting time intervals based on the identified confusion state intersect with the abnormal analysis time interval is recorded as the abnormal object analysis set; the confusion state meeting time interval mapping set corresponding to the abnormal object analysis set is obtained, where each element in the confusion state meeting time interval mapping set is the meeting time interval where the most recent facial expression state recognition result of the corresponding participant in the abnormal object analysis set is a confused state and is continuous; the correlation bias degree of each element in the abnormal object analysis set relative to the meeting content is calculated within the meeting time interval corresponding to the corresponding element in the confusion state meeting time interval mapping set.

[0072] Get the g-th element and its corresponding associated element in the set of confused state meeting time interval mapping. The meeting time interval corresponding to any associated element of the g-th element in the set of confused state meeting time interval mapping is a subset of the meeting time interval corresponding to the g-th element in the set of confused state meeting time interval mapping.

[0073] The following formula is used to calculate the adaptation coefficient of the meeting's abnormal focus content for each element in the set of meeting time interval mappings for confused states:

[0074] SPg = EVg / EVmax

[0075] EVg=P[g]+P[g,sum]

[0076] Where SPg represents the matching coefficient of the meeting anomaly focus content corresponding to the g-th element in the set of confusion state and meeting time interval mapping; EVg represents the comprehensive bias evaluation value of the meeting content corresponding to the g-th element in the set of confusion state and meeting time interval mapping; P[g] represents the correlation bias degree of the corresponding element of the g-th element in the set of confusion state and meeting time interval mapping within the set of anomaly object analysis, relative to the meeting content; P[g,sum] represents the sum of the correlation bias degrees of each associated element of the g-th element in the set of confusion state and meeting time interval mapping within the set of anomaly object analysis, relative to the meeting content; EVmax represents the maximum value of the comprehensive bias evaluation value of the meeting content corresponding to each element in the set of confusion state and meeting time interval mapping.

[0077] In the set of time intervals for meeting participants in a state of confusion, select the element with the smallest adaptation coefficient for the abnormal focus content of the meeting among the elements whose adaptation coefficient is greater than the preset adaptation threshold. This element is denoted as the abnormal focus mapping time interval. The meeting content in the corresponding meeting record within the abnormal focus mapping time interval is taken as the abnormal focus content of the meeting at the current time.

[0078] S4. Based on the obtained meeting focus of attention, mark the meeting minutes and the pre-defined subordinate relationships between the participants, and feed back the marked meeting minutes to the collaborating participants of the obtained meeting focus of attention.

[0079] In step S4, during the process of feeding back the marked meeting record content to the collaborative participants who have obtained the abnormal focus content of the meeting, the scheduled meeting duration corresponding to the agenda node at the current time in the text stream and the actual meeting duration corresponding to the agenda node at the current time are obtained.

[0080] If the actual meeting duration of the agenda node corresponding to the current time is less than or equal to the product of the scheduled meeting duration and the preset control coefficient of the agenda node corresponding to the current time in the text stream and the projected text, then when it is determined that the marked meeting record content should be fed back to the collaborative participants of the obtained meeting abnormal focus content, the corresponding collaborative participants are the participants who have the host's authority at the current time. If the actual meeting duration of the agenda node corresponding to the current time is greater than the product of the scheduled meeting duration and the preset control coefficient of the agenda node corresponding to the current time in the text stream and the projected text, then when it is determined that the marked meeting record content should be fed back to the collaborative participants of the obtained meeting abnormal focus content, the corresponding collaborative participants are the direct superiors of each element in the abnormal object analysis set, according to the preset subordinate relationship between the participants.

[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0082] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamic collaborative management of audio and video conferencing, characterized in that, The method includes: S1. Collect conference data streams in real time through multimodal sensors, import the collected conference data streams into the permission status model for dynamic analysis of conference control permission migration objects, and lock the migration objects of conference control permissions; S2. In real time, identify the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process, and generate the participation time interval of each participant based on the identified facial expressions; combine the meeting content involved in the meeting data stream to analyze the correlation bias of each participant with the meeting content within the participation time interval based on the identified facial expressions. S3. Based on the attendee's facial expression status, the time interval of the meeting, and the correlation bias with the meeting content within the corresponding time interval, dynamically filter the meeting's abnormal focus content corresponding to the current time. S4. Based on the obtained meeting focus points, mark the meeting minutes and the pre-defined hierarchical relationships between the participants, and feed back the marked meeting minutes to the collaborating participants of the obtained meeting focus points.

2. The method for dynamic collaborative management of audio and video conferencing according to claim 1, characterized in that: The conference data stream in S1 includes audio stream, video stream and text stream. The data streams corresponding to the audio stream, video stream and text stream are collections of corresponding types of collected data that change over time. The process of importing the collected meeting data stream into the permission status model for dynamic analysis of meeting control permission migration objects involves the following calculation formulas: If max{w i ·f i If (t)|i∈[1,2]}≥μ, then determine P(MA→MB). t =1; Conversely, if the result is not positive, then determine P(MA→MB). t =0; Where MA represents the object to which the meeting control permissions belonged before the dynamic analysis of the meeting control permission migration object; MB represents the object to which the meeting control permissions belonged after the dynamic analysis of the meeting control permission migration object; P(MA→MB) t The parameter represents the execution coefficient for the migration of meeting control permissions from MA to MB; μ represents the preset migration threshold; w i The w represents the weight of the i-th type of trigger. i The value is equal to the ratio of the frequency of times the trigger condition corresponding to the i-th type of trigger is met when the meeting control permission migration behavior is triggered in the historical data to the total frequency of meeting control permission migration behavior triggers; f i f1(t) represents the function value of the i-th type of trigger function at time t; f1(t) represents the agenda node switching function. If the screen-projected text in the text stream contains two agenda nodes within the preset time period before time t, and the meeting participants bound to the agenda nodes are MA and MB respectively, then f1(t) = 1; otherwise, f1(t) = 0. f2(t) represents the voice keyword matching function. If the keyword frequency of the participant object bound to MB is the highest in the keyword extraction results of the audio stream based on the preset time period t, and the meeting control permission object before the dynamic analysis of the meeting control permission migration object belongs to MA, then f2(t) = 1 is determined; otherwise, f2(t) = 0 is determined. The participant with the highest execution coefficient in the meeting control permission migration relationship is selected as the recipient of the meeting control permission migration.

3. The method for dynamic collaborative management of audio and video conferencing according to claim 1, characterized in that: In step S2, during the process of generating the meeting time interval for each participant based on their recognized facial expression state, the screen location regions of each participant in the meeting video frame at each time point in the meeting data stream during the meeting execution are extracted. Key features are extracted from these screen location regions using OpenFace, specifically the intensity values ​​of three key features: AU04 furrowed eyebrows, AU07 tightened eyelids, and AU12 pulled corners of the mouth. These key features are then input into a pre-trained random forest classifier to identify whether the facial expression state corresponding to the participant's screen location region is a confused state. The time intervals formed by the times when the facial expression state of each participant is identified as a confused state are used as the meeting time interval for that participant based on their recognized confused state.

4. The method for dynamic collaborative management of audio and video conferencing according to claim 1, characterized in that: The analysis of the relevance bias of each participant to the meeting content within the time interval based on recognized facial expressions involves the following calculation formula: Where Pj represents the relevance bias of the j-th participant to the meeting content within the time interval of the meeting based on the confusion state of identification; the meeting content in the meeting record corresponding to the j-th participant within the time interval of the meeting based on the confusion state of identification is denoted as Hj; YHj represents the number of keywords belonging to the keyword library bound to the j-th participant in the keyword extraction result of the meeting content corresponding to Hj, and the keywords contained in the keyword library bound to each participant are preset; D (j，k) This represents the bias factor of the k-th keyword in the keyword extraction results for the meeting content corresponding to Hj, which belongs to the keyword library bound to the j-th participant. Let C be the keyword that belongs to the keyword library bound to the j-th participant in the keyword extraction results for the meeting content corresponding to Hj. (j，k) The D (j，k) The value is equal to the number of C keywords in the bound keyword library. (j，k) The reciprocal of the number of attendees; N represents the total number of attendees; This represents the association bias coefficient of the keyword extraction result for the meeting content corresponding to Hj, based on the keyword library bound to the j-th participant; max{} represents the maximum value function; max{Q (j,k) |j∈[1,N]} represents the maximum value of the association bias coefficient based on the keyword library bound to each participant, representing the result of keyword extraction for the meeting content corresponding to Hj.

5. The method for dynamic collaborative management of audio and video conferencing according to claim 1, characterized in that: In the process of dynamically filtering the abnormal focus content of the meeting corresponding to the current time in S3, the preset time period based on the current time is recorded as the abnormal analysis time interval; the summary set of all participants whose meeting time intervals based on the identified confusion state intersect with the abnormal analysis time interval is recorded as the abnormal object analysis set; the confusion state meeting time interval mapping set corresponding to the abnormal object analysis set is obtained, where each element in the confusion state meeting time interval mapping set is the meeting time interval where the most recent facial expression state recognition result of the corresponding participant in the abnormal object analysis set is a confused state and is continuous; the correlation bias degree of each element in the abnormal object analysis set relative to the meeting content is calculated within the meeting time interval corresponding to the corresponding element in the confusion state meeting time interval mapping set. Get the g-th element and its corresponding associated element in the set of confused state meeting time interval mapping. The meeting time interval corresponding to any associated element of the g-th element in the set of confused state meeting time interval mapping is a subset of the meeting time interval corresponding to the g-th element in the set of confused state meeting time interval mapping. The following formula is used to calculate the adaptation coefficient of the meeting's abnormal focus content for each element in the set of meeting time interval mappings for confused states: SPg = EVg / EVmax EVg=P[g]+P[g,sum] Where SPg represents the matching coefficient of the meeting anomaly focus content corresponding to the g-th element in the set of confusion state and meeting time interval mapping; EVg represents the comprehensive bias evaluation value of the meeting content corresponding to the g-th element in the set of confusion state and meeting time interval mapping; P[g] represents the correlation bias degree of the corresponding element of the g-th element in the set of confusion state and meeting time interval mapping within the set of anomaly object analysis, relative to the meeting content; P[g,sum] represents the sum of the correlation bias degrees of each associated element of the g-th element in the set of confusion state and meeting time interval mapping within the set of anomaly object analysis, relative to the meeting content; EVmax represents the maximum value of the comprehensive bias evaluation value of the meeting content corresponding to each element in the set of confusion state and meeting time interval mapping. In the set of time intervals for meeting participants in a state of confusion, select the element with the smallest adaptation coefficient for the abnormal focus content of the meeting among the elements whose adaptation coefficient is greater than the preset adaptation threshold. This element is denoted as the abnormal focus mapping time interval. The meeting content in the corresponding meeting record within the abnormal focus mapping time interval is taken as the abnormal focus content of the meeting at the current time.

6. The method for dynamic collaborative management of audio and video conferencing according to claim 5, characterized in that: In step S4, during the process of feeding back the marked meeting record content to the collaborative participants who have obtained the abnormal focus content of the meeting, the scheduled meeting duration corresponding to the agenda node at the current time in the text stream and the actual meeting duration corresponding to the agenda node at the current time are obtained. If the actual meeting duration of the agenda node corresponding to the current time is less than or equal to the product of the scheduled meeting duration and the preset control coefficient of the agenda node corresponding to the current time in the text stream and the projected text, then when it is determined that the marked meeting record content should be fed back to the collaborative participants of the obtained meeting abnormal focus content, the corresponding collaborative participants are the participants who have the host's authority at the current time. If the actual meeting duration of the agenda node corresponding to the current time is greater than the product of the scheduled meeting duration and the preset control coefficient of the agenda node corresponding to the current time in the text stream and the projected text, then when it is determined that the marked meeting record content should be fed back to the collaborative participants of the obtained meeting abnormal focus content, the corresponding collaborative participants are the direct superiors of each element in the abnormal object analysis set, according to the preset subordinate relationship between the participants.

7. A dynamic collaborative management system for audio and video conferencing, employing the dynamic collaborative management method for audio and video conferencing as described in any one of claims 1-6, characterized in that, The system includes: The permission migration management module collects meeting data streams in real time through multimodal sensors, imports the collected meeting data streams into the permission status model for dynamic analysis of meeting control permission migration objects, and locks the migration objects of meeting control permissions. The meeting content association bias analysis module identifies the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process in real time, and generates the participation time interval of each participant based on the identified facial expressions; combined with the meeting content involved in the meeting data stream, it analyzes the association bias of each participant with the meeting content within the participation time interval based on the identified facial expressions. An abnormal focus filtering module dynamically filters abnormal focus content of the meeting based on the time interval of the attendee's facial expression state and the correlation bias with the meeting content within the corresponding time interval. The meeting anomaly intelligent feedback module marks the obtained meeting anomaly focus content in the meeting record content and the preset subordinate relationships between various participants, and feeds back the marked meeting record content to the collaborating participants of the obtained meeting anomaly focus content.

8. The audio and video conferencing dynamic collaborative management system according to claim 7, characterized in that: The meeting content association bias analysis module includes an attendance time analysis unit and a content association bias analysis unit. The meeting time analysis unit identifies the facial expressions of each participant in the video frame at each time point in the meeting data stream during the meeting execution process in real time, and generates the meeting time interval for each participant based on the identified facial expressions. The content association bias analysis unit combines the meeting content involved in the meeting data stream to analyze the association bias of each participant with the meeting content within the time interval based on the recognized facial expression state.

Images