Conference session quality monitoring

Abstract

A method for monitoring the quality of a conference session between a plurality of participant devices is described. One or more data streams of a conference session are monitored. Presentor context information is determined for media sent by a presentor device of the plurality of participant devices over one or more data streams. A mismatch between the presenter context information and first participant context information for a first participant device of the plurality of participant devices is identified. For the identified mismatch, a mismatch notification is provided to the presenter device.

Description

Background Art

[0001] There are many common challenges that affect individual and team productivity on conference calls. Some challenges affect those connected to audio-visual functions, such as a user speaking but not being heard by others in a video conference, or referring to a slide that has not yet been shown or shared. These challenges may occur due to external constraints on the system (e.g., low network bandwidth), user errors (e.g., accidental muting), or software vulnerabilities or other problems with the communication software hosting the video conference. There are also scenarios where users simply want to know whether they can be seen and / or heard, or whether other participants on the call can see their slides or their shared screen. Current methods for handling these challenges rely on other participants to point out problems or presenters actively seeking confirmation from other participants, but these solutions take up time for effective conversations and reduce the overall quality of conference calls.

[0002] With respect to these and other general considerations, embodiments have been described. Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. Summary of the invention

[0003] Aspects of the present disclosure relate to monitoring the quality of a conference session.

[0004] In one aspect, a method for monitoring the quality of a conference session between multiple participant devices is provided. The method includes: monitoring one or more data streams of the conference session; determining presenter context information for media sent by a presenter device of the multiple participant devices over the one or more data streams; identifying a mismatch between the presenter context information and first participant context information for a first participant device of the multiple participant devices; and providing a mismatch notification to the presenter device for the identified mismatch.

[0005] In another aspect, a method for training a conference system is provided. The method includes: monitoring a data stream of a conference session between multiple participant devices, the data stream having one or more of an audio component, a video component, and a shared content component; determining presenter context information for a first media sent by a presenter device among the multiple participant devices through the data stream; determining first participant context information for a second media received by a first participant device among the multiple participant devices; marking a first segment of the data stream according to the presenter context information, and marking a second segment of the data stream according to the first participant context information; and training a machine learning model to provide a mismatch notification based on the marked first segment and the marked second segment, wherein overlapping portions of the marked first segment and the marked second segment have different tags.

[0006] In yet another aspect, a system for monitoring the quality of a conference session between multiple participant devices is provided. The system includes a data stream processor configured to monitor one or more data streams of the conference session. The system also includes a first context processor configured to: determine presenter context information for media sent by a presenter device in the multiple participant devices over the one or more data streams, identify a mismatch between the presenter context information and first participant context information for a first participant device in the multiple participant devices, and provide a mismatch notification to the presenter device for the identified mismatch.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0009] Figure 1 A block diagram of an example of a conferencing system for a conference session according to an example embodiment is shown.

[0010] Figure 2 A block diagram illustrating data flow for a conferencing session according to an example embodiment is shown.

[0011] Figure 3 A diagram illustrating an example notification for a conference session according to an example embodiment is shown.

[0012] Figure 4 A diagram illustrating another example notification for a conference session according to an example embodiment.

[0013] Figure 5 A flow chart illustrating an example method of monitoring the quality of a conferencing session between multiple participant devices according to an example embodiment.

[0014] Figure 6 A flow chart is shown of another example method of training a conferencing system according to an example embodiment.

[0015] Figure 7 is a block diagram illustrating example physical components of a computing device in which aspects of the present disclosure may be practiced.

[0016] Figure 8 is a simplified block diagram of a mobile computing device that can be used to practice aspects of the present disclosure. DETAILED DESCRIPTION

[0017] In the following specific embodiments, reference is made to the accompanying drawings that form a part of this article, and specific embodiments or examples are shown therein by way of illustration. Without departing from the present disclosure, these aspects can be combined, other aspects can be utilized, and structural changes can be made. The embodiment can be practiced as a method, system or device. Accordingly, the embodiment can take the form of hardware implementation, complete software implementation, or combination software and hardware implementation. Therefore, the following specific embodiments should not be regarded as restrictive, and the scope of the present disclosure is limited by the attached claims and their equivalents.

[0018] The present disclosure describes various aspects of monitoring the quality of conference sessions and training conference systems that support conference sessions. In some examples, the processor monitors data streams (such as audio or video output streams) from a presenter device on a conference session, and based on feedback from a participant device, a notification can be provided to the presenter device when one or more data streams in the data stream are not received or are inconsistent with each other. For example, when a participant device cannot hear an audio stream (e.g., when a mute function is activated while the presenter is speaking), a notification provides feedback to the presenter device. As another example, when a participant device cannot see shared content from a presenter device (e.g., when the presenter forgets to share its screen or shares the wrong document), another notification provides feedback to the presenter device. In some examples, the system is configured to propose a remediation strategy for handling the identified problem. For example, the system can prompt a user to unmute a microphone, go to a specific page in a shared document, disable video to save bandwidth, etc.

[0019] This and many other aspects of computing devices are described herein. For example, Figure 1 A block diagram of an example of a conference system 100 for a conference session according to an example aspect is shown. The conference system 100 includes various computing devices 110, 120, and 130 that can be used by participants in a conference session, a host of a conference session, or both a host and participants in a conference session. Figure 1 In the illustrated embodiment, computing device 110 is used by a presenter (i.e., one of the participants providing media to other participants, such as by sharing content, video, or speech), computing device 130 is configured as a moderator for a conference session (e.g., by relaying and / or processing data streams between participants), and one or more instances of computing device 120 are used by participants in the conference session.

[0020] Computing device 110 may be any type of computing device, including a mobile computer or a mobile computing device (e.g., devices, laptops, notebooks, smartphones, such as Apple iPad TMIn some aspects, computing device 110 is a cable set-top box, a streaming video box, or a console gaming device. Computing device 110 may be configured to execute one or more software applications (or "apps") and / or services, and / or manage hardware resources (e.g., processor, memory, etc.) that may be utilized by a user of computing device 110.

[0021] Computing device 110 includes conferencing module 112, data stream processor 114, and context processor 116. Computing device 120 is similar in some aspects to computing device 110 (e.g., a mobile computer, a laptop computer, etc.) and includes conferencing module 122, data stream processor 124, and context processor 126, generally corresponding to conferencing module 112, data stream processor 114, and context processor 116, respectively.

[0022] Computing device 130 may include data stream processor 134, context processor 136, and machine learning model 138. Data stream processor 134 and context processor 136 may generally correspond to data stream processor 114 and context processor 116, respectively. In some examples, computing device 130 is a network server, cloud server, or other suitable network device.

[0023] The conference module 112 (and the conference module 122) generally provide conference features to the user of the computing device 110. The conference features support participation in conference sessions, such as conference call sessions, video call sessions, collaboration sessions, etc. The conference module 112 can be implemented as a software program (e.g., Microsoft Teams, Zoom, WebEx), a hardware-based circuit or processor, or a combination thereof. The conference module 112 includes one or more of an image sensor or camera, a microphone, a speaker, a user interface (e.g., a keyboard, a mouse, buttons) that supports interaction with the conference features, or communicates with one or more of an image sensor or camera, a microphone, a speaker, a user interface (e.g., a keyboard, a mouse, buttons) that supports interaction with the conference features. The conference module 112 can be configured to generate one or more data streams having various components, such as an audio component (e.g., an audio signal or a text transcript of speech or sound within an audio signal), a video component (e.g., pixel information for displaying a video), or a content sharing component (e.g., information for sharing documents, applications, screens, etc.), and send the data streams to the computing device 120 or the computing device 130. The user of computing device 110 can select or provide media for transmission through data stream, for example, by speaking into a microphone, appearing in front of a webcam, or interacting with a document to be shared with the participants. In some examples, conference module 112 generates a single data stream including an audio component, a video component, and a shared content component. In other examples, conference module 112 generates two or more data streams with different components. As an example, a first data stream includes audio, and a second data stream includes video and shared content. As another example, a first data stream includes audio and video, and a second data stream includes shared content. In other examples, different data streams are used for each component in the audio, video, and shared content components.

[0024] Data stream processor 114 (as well as data stream processor 124 and data stream processor 134) is configured to monitor the data streams generated by conference module 112 and conference module 122. In some examples, data stream processor 114 separates or extracts audio, video, or shared content components from the data streams. Data stream processor 114 can then monitor the components separately, provide the components to other processors (e.g., context processor 116, context processor 126, or context processor 136), or perform other suitable processing.

[0025] The context processor 116 (as well as the context processor 126 and the context processor 136) is configured to determine context information based on the data stream from the data stream processor 114. Advantageously, in various aspects, the context information for the system 100 can be generated or provided by any one of the presenter (computing device 110), the participant (computing device 120), or the host (computing device 130). The context information can then be provided to the appropriate context processor to provide notifications to the corresponding users. The context information can include one or more of an audio state for an audio component, such as a volume or signal level, the presence or absence of a signal that meets a predetermined threshold (e.g., exceeds a background noise level), a video state for a video component (e.g., the presence or absence of a video signal, whether a user is present in the video signal), and a shared content state (e.g., whether the content has been shared, whether the content is visible, etc.).

[0026] The context information may be related to the entire conference session, such as the conference title (e.g., "Financial Review 4Q2022"), or may be related only to segments within the conference session. For example, context information based on the content within a shared document may be relevant only when certain pages or slides are displayed. In some examples, the context information includes keywords, names, or topics associated with the data stream. For example, when the presenter speaks the name of a participant so that the name appears within the audio component, the name may be added to the context information together with the timestamp of when the name was spoken. As another example, when a document is shared so that its content appears within the shared content component, keywords within the document may be added to the context information. As yet another example, the context processor 116 may determine context information from a chat message within a conference session, for example, when a user types "BRB" or "AFK" to indicate that they have temporarily left their computer. In some examples, segments within the data stream are marked with relevant context information, thereby allowing comparison of context information.

[0027] Typically, the context processor 116 can determine or receive context information for one or more of the presenter and the participants. Since each participant can send or receive media during the conference session, each participant can have corresponding context information. In some scenarios, the context information for the participants can be different and generally independent of each other, for example, one participant (the presenter) can have an activated mute feature, so the audio component state can indicate a mute state. As another example, another participant may have a slow network connection and cannot view the video component, so the video component state can indicate that the video feed is interrupted. As yet another example, one participant may be viewing the first page of a shared document, while another participant may be viewing the fifth page of a shared document.

[0028] The context processor 116 is configured to identify a mismatch between the presenter context information (i.e., for the computing device 110) and the first participant context information (i.e., for the computing device 120). When a mismatch is identified, the context processor 116 can provide a mismatch notification for the identified mismatch to the presenter (via the conference module 112 to the computing device 110). In various aspects, the mismatch notification can be a visual display, such as a pop-up window, an icon or other element within a graphical user interface, an audio queue, tactile feedback, or other suitable notification. Accordingly, when the presenter is speaking and providing audio to the conference module 112 (i.e., the presenter context information indicates that there is audio), but the audio is not received by other participants due to a mute feature (i.e., the first participant context information indicates that there is no audio), the context processor 116 can provide a notification to the presenter, such as a user interface pop-up window for proposing an audio unmute action. In this way, the presenter does not need to ask other participants "Can you hear me?" and wait for a response before continuing the conference session. In a similar example, the context processor 116 can indicate when the presenter is visible to other participants, eliminating reliance on hardware-based solutions (e.g., webcam status LEDs). In yet another example, the context processor 116 can indicate when shared content, such as a document (e.g., a text document, a spreadsheet, a presentation or slide show, etc.), a shared screen, or a shared application, is visible to other participants, so the presenter does not need to ask other participants whether the shared content is visible.

[0029] In various aspects, media, components from a data stream, or contextual information for a conference session is provided to a machine learning model 138, for example, by data stream processor 114, data stream processor 124, or data stream processor 134, or by context processor 116, context processor 126, or context processor 136. Figure 1 Only one machine learning model 138 is shown in the figure, but the computing device 130 may include one, two, three or more machine learning models 138 trained for different tasks. In some aspects, the machine learning model 138 is integrated with the context processor 136. The machine learning model 138 can be implemented as a deep learning model, a transformer model, a species distribution model, or a combination thereof. In some examples, multiple instances of the machine learning model 138 are provided within the conference system 100, for example at the computing device 110 or the computing device 120.

[0030] Typically, the machine learning model 138 is configured to identify mismatches between media, components, or context information. In other words, the machine learning model 138 processes components (e.g., audio, video, or shared content components) or context information and marks inconsistencies between them. In various aspects, the machine learning model 138 is configured to mark inconsistencies between audio components and video components, between audio components and shared content components, or between video components and shared content components. In some aspects, inconsistencies occur when the comparison of context information for the participants is below a relevance threshold (e.g., the context indicates that the components are no longer relevant to each other). For example, when the keywords within the displayed page of a shared document (shared content component) no longer match the keywords being spoken (audio component), the machine learning model 138 can mark inconsistencies. When 50% of the keywords in the keywords match between the shared content component and the audio component, 30% of the keywords in the keywords match between the audio component and the video component, and so on, the relevance threshold can be met. In some aspects, specific keywords or phrases are given higher weights, such as keywords or phrases that process "pages" or "slides." In some examples, machine learning model 138 processes the video component to determine whether a party appears confused, which may increase the likelihood of a mismatch.

[0031] Network 140 may include one or more networks, such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc., and may include one or more of wired and / or wireless portions. Computing device 110, computing device 120, and computing device 140 may include at least one wired or wireless network interface that enables communication with each other (or with an intermediate device, such as a Web server or a database server) via network 140. Examples of such network interfaces include, but are not limited to, an IEEE 802.11 wireless LAN (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth® interface, and a cellular network interface. TM An interface, or a near field communication (NFC) interface. Examples of network 140 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, and / or any combination thereof.

[0032] Figure 2A block diagram of data flow for conference session 200 according to example aspects is shown. Conference session 200 is hosted by server 230, and participants of conference session 200 include presenter 210, participant 220, and participant 240. In various aspects, presenter 210, participant 220, and participant 240 may correspond to computing device 110 or computing device 120, and include an instance of context processor 216 (e.g., similar to context processor 116). In various aspects, server 230 may correspond to computing device 130, and include an instance of context processor 216.

[0033] Conference session 200 includes three data streams: audio stream (or component) 250, video stream (or component) 260, and shared content stream (or component) 270. Figure 2 In the example shown, presenter 210 uses a webcam to provide audio and video to audio stream 250 and video stream 260, respectively, and also provides a spreadsheet to shared content stream 270 (e.g., using the document sharing feature of conference module 112). The audio, video, and spreadsheet are sent to participant 220, but the spreadsheet cannot be viewed by participant 240 (shown by the dashed lines of the shared content), for example, due to excessive packet loss. In this example, presenter context information (generated by context processor 216 of presenter 210) indicates the presence of audio, video, and shared content streams 250, 260, and 270, but participant context information (generated by context processor 216 of participant 240) indicates the presence of only audio stream 250 and video stream 260. Context processor 216 of presenter 210 may provide a notification to the user of presenter 210 indicating that the spreadsheet cannot be viewed by the user of participant 240.

[0034] The context processor 216 can be configured to propose a remediation strategy for addressing the identified issue. For example, the context processor 216 can prompt the user to unmute the microphone, go to a specific page in a shared document, disable video to save bandwidth, rejoin the conference session, reshare the content, etc. In some examples, the machine learning model 138 provides the context processor 216 with a ranked list of remediation options for delivery to the user.

[0035] Figure 3 FIG. 2 shows a diagram of an example notification for a conference session according to an example embodiment. Figure 3As shown, display 302 of computing device 110 provides user interface 304 that may be generated by conference module 112. User interface 304 includes pane 360, pane 370, and pane 380 for three participants in the conference session. Each of panes 360, 370, and 380 includes icons A, V, and C that indicate the status of the audio component, video component, and shared content component for the conference session, respectively, and the video component received by computing device 110. Figure 3 In the example, pane 380 indicates that the video component of the third party is inactive by graying out the icon for the video component. In other examples, the notification can be provided as a colored icon, animation or other suitable visual notifications.

[0036] Figure 4 FIG. 2 shows another example notification for a conference session according to an example embodiment. Figure 4 As shown, the display 402 of the computing device 110 provides a user interface 404 that can be generated by the conference module 112. The user interface 404 includes a pane 470 for a shared document (shared content component), a pane 472 for a chat window (shared content component), and respective icon sets 482, 484, and 486 for the three participants in the conference session. Figure 4 , the icon set 482 "Abe" for the first user has been grayed out to indicate that the user may not be viewing the conference session. For example, the context processor 116 can determine that the first user has left the computing device 120 based on the text chat of "BRB" ("BRB" is usually written to mean that the user will be "back soon") and based on the absence of the first user within the captured image from the webcam of the computing device 120.

[0037] In some examples, the context processor 116 is configured to remove elements from the user interface 304 or the user interface 404 when no content mismatch is identified. For example, when each participant can see the video from the presenter, the context processor 116 can remove the presenter's selfie image from the presenter's user interface. Removing the user's image can improve the user's comfort during the conference session or increase space for other content within the user interface.

[0038] Figure 5 A flow chart of an example method 500 for monitoring the quality of a conference session between multiple participant devices according to an example embodiment is shown. Unless otherwise indicated, the technical processes shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, possibly using different parameters or data. The steps in the embodiment may also be performed in the same manner as in the example embodiment. Figure 5The steps of method 500 may be performed in a different order from the top to bottom order of the layout in FIG. The steps may be performed serially, in a partially overlapping manner, or completely in parallel. Therefore, the order in which the steps of method 500 are performed may vary from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise differ from the flow shown, so long as the performed process is operable and complies with at least one claim. Figure 5 The steps of may be performed by computing device 110 (eg, via conferencing module 112, data stream processor 114, or context processor 116) or other suitable computing devices.

[0039] Method 500 begins at step 502. At step 502, one or more data streams of a conference session are monitored. For example, the data streams may correspond to data stream 250, data stream 260, and data stream 270, and are monitored, for example, by data stream processor 114, data stream processor 124, or data stream processor 134. In some aspects, each of data stream processor 114, data stream processor 124, and data stream processor 134 monitors a data stream.

[0040] At step 504, presenter context information for media sent by a presenter device among the plurality of participant devices via one or more data streams is determined. In some examples, the context processor 116 of the computing device 110 determines the presenter context information. In some aspects, determining the presenter context information includes one or more of: determining a presenter audio state for an audio component of the conference session provided by the presenter device; determining a presenter video state for a video component of the conference session provided by the presenter device; and determining a presenter shared content state for a shared content component of the conference session provided by the presenter device.

[0041] At step 506 , a mismatch between the presenter context information and the first participant context information for a first participant device of the plurality of participant devices is identified. For example, the context processor 216 may identify a mismatch between the presenter context information and the participant context information for the shared content component 270 .

[0042] At step 508, for the identified mismatch, a mismatch notification is provided to the presenter device. In one example, the context processor 216 provides a gray icon as a mismatch notification to the pane 380. In another example, the context processor 216 provides a gray icon as a mismatch notification to the icon set 482. In other examples, the context processor 116, the context processor 126, the context processor 136, or the context processor 216 displays an element within a graphical user interface or provides an audio queue, tactile feedback, or other suitable notification.

[0043] In some examples, method 500 also includes receiving first party context information from a first party device, wherein the first party context information includes one or more of: a party audio state of an audio component, a party video state of a video component, and a party shared content state of a shared content component.

[0044] In some examples, identifying a mismatch includes generating a mismatch notification when: a presenter audio state indicates the presence of an audio component and a participant audio state indicates the absence of an audio component, a presenter video state indicates the presence of a video component and a participant video state indicates the absence of a video component, or a presenter shared content state indicates the presence of a shared content component and a participant shared content state indicates the absence of a shared content component.

[0045] In some examples, a conference session has multiple participants and at least one presenter sharing data, the at least one presenter sharing data including: a first audio component, and the participant audio state indicates the absence of the first audio component, a first video component, and the participant video state indicates the absence of the first video component, or a first shared content component, and the participant shared content state indicates the absence of the first shared content component. The shared content component includes one or more of a screen sharing session, an application sharing session, a collaboration tool sharing session, and a document sharing session. In some examples, the first shared content component includes a document, and identifying a mismatch includes: tagging pages of the document with keywords based on content within the document; and providing the tagged pages and the first audio component to a machine learning model. In some examples, the first audio component is a transcription of an audio signal captured during the conference session.

[0046] In some examples, identifying a mismatch includes generating a mismatch notification when the machine learning model flags an inconsistency between: the first audio component and the first video component, the first audio component and the first shared content component, or the first video component and the first shared content component.

[0047] In some aspects, method 500 further includes receiving one or more of a first audio component, a first video component, and a first shared content component from at least one presenter; and sending one or more of the first audio component, the first video component, and the first shared content component to multiple participants.

[0048] In some aspects, method 500 also includes generating a mismatch notification to identify one or more remediation options for the identified mismatch. Examples of remediation options may include one or more of: unmute the microphone (e.g., "unmute your headphones using the on / off slider"), go to a specific page in a shared document (e.g., "go to slide 7"), disable video to save bandwidth, rejoin the conference session, reshare content.

[0049] Figure 6 Flowcharts showing an example method 600 for training a conference system according to an example embodiment are shown. Unless otherwise indicated, the technical processes shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, possibly using different parameters or data. The steps in the embodiment may also be performed in the same manner as in the example embodiment. Figure 6 The steps of method 600 may be performed in a different order than the top-to-bottom order of the layout in FIG. The steps may be performed serially, in a partially overlapping manner, or completely in parallel. Thus, the order in which the steps of method 600 are performed may be different from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise differ from the flow shown, so long as the performed process is operable and complies with at least one claim. Figure 6 The steps of may be performed by computing device 110 (eg, via context processor 116) or other suitable computing devices.

[0050] Method 600 begins at step 602. At step 602, a data stream of a conference session is monitored between a plurality of participant devices, the data stream having one or more of an audio component, a video component, and a shared content component.

[0051] At step 604, presenter context information is determined for first media sent via a data stream by a presenter device among a plurality of participant devices.

[0052] At step 606, first party context information is determined for second media received by a first party device of the plurality of party devices.

[0053] At step 608 , a first segment of the data stream is labeled according to the presenter context information, and a second segment of the data stream is labeled according to the first participant context information.

[0054] At step 610, the machine learning model is trained to provide a mismatch notification based on the marked first segment and the marked second segment, wherein the overlapping portions of the marked first segment and the marked second segment have different tags. In various examples, the mismatch notification includes one or more of the following: a first notification to a presenter device, the first notification indicating a suggested page jump for a shared content component; a second notification to a presenter device, the second notification indicating a suggested alternative path for one or more of an audio component, a video component, and a shared content component; a third notification to a presenter device, the third notification indicating a suggested document to be shared via the shared content component; or a fourth notification to a first participant device, the fourth notification indicating a suggested audio unmute action.

[0055] In some examples, method 600 further includes: receiving presenter context information from a presenter device, wherein the presenter context information is generated by the presenter device based on the data stream; and receiving participant context information from a first participant device, wherein the participant context information is generated by the first participant device based on the data stream. In one example, the data stream of the conference session is stored in a video recording of the conference session, and method 600 further includes separating one or more of the audio component, the video component, and the shared content component into independent streams using a machine learning model.

[0056] Figure 7 and 8 and the associated descriptions provide a discussion of various operating environments in which aspects of the present disclosure may be practiced. Figure 7 and Figure 8 The devices and systems shown and discussed are for purposes of example and explanation, and are not intended to be limiting of the wide variety of computing device configurations that may be used to practice aspects of the present disclosure, as described herein.

[0057] Figure 7 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700 in which aspects of the present disclosure may be practiced. The computing device components described below may have computer executable instructions for implementing a conference system application 720 on a computing device (e.g., computing device 110, computing device 120, computing device 130), including computer executable instructions for the conference system application 720 that may be executed to implement the methods disclosed herein. In a basic configuration, the computing device 700 may include at least one processing unit 702 and a system memory 704. Depending on the configuration and type of the computing device, the system memory 704 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of these memories. The system memory 704 may include an operating system 705 and one or more program modules 706 suitable for running the conference system application 720, such as programs related to the computer system 702. Figure 1 and Figure 2 One or more components, and specifically, a conference module 721 (e.g., corresponding to conference module 112), a data stream processor 722 (e.g., corresponding to data stream processor 114) and a context processor 723 (e.g., corresponding to context processor 116).

[0058] The operating system 705 may be suitable for controlling the operation of the computing device 700, for example. In addition, embodiments of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program, and are not limited to any particular application or system. Figure 7 708. The computing device 700 may have additional features or functionality. For example, the computing device 700 may also include additional data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tapes. Such additional storage Figure 7 , which is illustrated by a removable storage device 709 and a non-removable storage device 710.

[0059] As described above, a plurality of program modules and data files may be stored in system memory 704. When executed on processing unit 702, program module 706 (e.g., conference system application 720) may perform processes including, but not limited to, the aspects described herein. Other program modules according to aspects of the present disclosure may be used, and in particular for monitoring the quality of conference sessions, may include conference module 112, data stream processor 722, and context processor 723.

[0060] In addition, embodiments of the present disclosure may be practiced on a single chip including discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or containing electronic components or a microprocessor. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC) wherein Figure 7 Each or more of the components shown may be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via a SOC, the functionality described herein regarding the client-to-switch protocol capabilities may be operated via dedicated logic integrated with other components of the computing device 700 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations such as (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present invention may be practiced within a general-purpose computer or in any other circuit or system.

[0061] The computing device 700 may also have one or more input devices 712, such as a keyboard, mouse, pen, sound or voice input device, touch or slide input device, etc. Output device(s) 714 may also be included, such as a display, speaker, printer, etc. The above devices are examples, and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of suitable communication connections 716 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0062] The term computer-readable media used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media for storage of information (such as computer-readable instructions, data structures or program modules) implemented in any method or technology. System memory 704, removable storage device 709 and non-removable storage device 710 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassette, magnetic tape, disk storage or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 700. Any such computer storage media may be part of computing device 700. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0063] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal having one or more characteristics that are set or changed in some manner to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0064] Figure 8 A mobile computing device 800 is shown on which embodiments of the present disclosure may be practiced, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc. In some aspects, a client may be a mobile computing device. Figure 8800 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, a mobile computing device 800 can incorporate a system (e.g., architecture) 802 to implement some aspects. In one embodiment, the system 802 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 802 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0065] One or more application programs 866 may be loaded into the memory 862 and run on or in association with the operating system 864. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile storage area 868 within the memory 862. The non-volatile storage area 868 may be used to store persistent information that should not be lost if the system 802 loses power. The application programs 866 may use information and store information in the non-volatile storage area 868, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 868 synchronized with the corresponding information stored at the host computer.

[0066] The system 802 has a power source 870, which can be implemented as one or more batteries. The power source 870 can also include an external power source, such as an AC adapter or a power docking station that replenishes or recharges the batteries.

[0067] The system 802 may also include a radio interface layer 872 that performs the functions of sending and receiving radio frequency communications. The radio interface layer 872 supports wireless connections between the system 802 and the "outside world" via a communications operator or service provider. Transmissions to and from the radio interface layer 872 are performed under the control of the operating system 864. In other words, communications received by the radio interface layer 872 can be propagated to the application 866 via the operating system 864, and vice versa.

[0068] Visual indicator 820 can be used to provide visual notification, and / or audio interface 874 can be used to communicate via audio transducer 825 (e.g., Figure 8The audio transducer 825 shown in the figure) produces an audible notification. In the illustrated embodiment, the visual indicator 820 is a light emitting diode (LED), and the audio transducer 825 can be a speaker. These devices can be directly coupled to the power supply 870 so that when activated, even if the processor 860 and other components may be turned off to save power, they are also turned on for a period of time indicated by the notification mechanism. The LED can be programmed to remain turned on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 874 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 825, the audio interface 874 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to support the control of the notification, as described below. The system 802 can also include a video interface 876, which enables the operation of the peripheral device 830 (e.g., an onboard camera) to record still images, video streams, etc.

[0069] The mobile computing device 800 implementing the system 802 may have additional features or functionality. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 8 , shown by non-volatile storage area 868.

[0070] As described above, data / information generated or captured by the mobile computing device 800 and stored via the system 802 may be stored locally on the mobile computing device 800, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 872 or via a wired connection between the mobile computing device 800 and a separate computing device associated with the mobile computing device 800 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 800 via the radio interface layer 872 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transfer and storage means, including email and collaborative data / information sharing systems.

[0071] It should be understood that the Figure 7 and Figure 8 The methods and systems are described for purposes of illustration and are not intended to limit the present disclosure to a specific sequence of steps or a specific combination of hardware or software components.

[0072] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey the disclosure that possesses and enables others to make and use the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether it is shown and described in combination or individually, various features (structures and methods) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. Description and explanation of the present application have been provided, and those skilled in the art can envision variations, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, and these variations, modifications and alternative aspects do not depart from the broader scope of the disclosure claimed.

Claims

1. A method for monitoring the quality of a conference session between multiple participant devices, the method comprising: monitoring one or more data streams of the conference session; Determining presenter context information for media sent by a presenter device among the plurality of participant devices through the one or more data streams; identifying a mismatch between the presenter context information and first participant context information for a first participant device of the plurality of participant devices; as well as A mismatch notification is provided to the presenter device for the identified mismatch.

2. The method according to claim 1, wherein determining the presenter context information comprises one or more of the following: Determining a presenter audio state for an audio component of the conference session provided by the presenter device; determining a presenter video status for a video component of the conference session provided by the presenter device; and A presenter shared content state is determined for a shared content component of the conference session provided by the presenter device.

3. The method according to claim 2 further includes receiving the first participant context information from the first participant device, wherein the first participant context information includes one or more of the following: the participant audio status of the audio component, the participant video status of the video component, and the participant shared content status of the shared content component.

4. The method of claim 3, wherein identifying the mismatch comprises generating the mismatch notification when: The presenter audio status indicates the presence of the audio component, and the participant audio status indicates the absence of the audio component, The presenter video status indicates the presence of the video component, and the participant video status indicates the absence of the video component, or The presenter shared content status indicates the presence of the shared content component, and the participant shared content status indicates the absence of the shared content component.

5. The method according to claim 3, wherein the conference session has a plurality of participants and at least one presenter sharing data, and the at least one presenter sharing data comprises: a first audio component, and the party audio status indicates the absence of the first audio component, a first video component, and the participant video status indicates the absence of the first video component, or A first shared content component is present, and the participant shared content status indicates the absence of the first shared content component. 6 . The method of claim 5 , wherein the first shared content component comprises one or more of a screen sharing session, an application sharing session, a collaboration tool sharing session, and a document sharing session.

7. The method of claim 5, wherein identifying the mismatch comprises generating the mismatch notification when a machine learning model flags an inconsistency between: the first audio component and the first video component, the first audio component and the first shared content component, or The first video component and the first shared content component.

8. The method of claim 7, wherein the first shared content component comprises a document, and identifying the mismatch comprises: tagging pages of the document with keywords based on content within the document; as well as The tagged page and the first audio component are provided to the machine learning model.

9. The method according to claim 3, further comprising: The mismatch notification is generated to identify one or more remediation options for the identified mismatch.

10. A system for monitoring the quality of a conference session between a plurality of participant devices, the system comprising: a data stream processor configured to monitor one or more data streams of the conference session; The first context processor is configured as follows: Determining presenter context information for media sent by a presenter device among the plurality of participant devices through the one or more data streams; identifying a mismatch between the presenter context information and first participant context information for a first participant device of the plurality of participant devices; as well as A mismatch notification is provided to the presenter device for the identified mismatch.

11. The system of claim 10, wherein the first context processor is configured to: Determining a presenter audio state for an audio component of the conference session provided by the presenter device; determining a presenter video status for a video component of the conference session provided by the presenter device; and A presenter shared content state is determined for a shared content component of the conference session provided by the presenter device.

12. A system according to claim 11, wherein the first context processor is configured to receive the first participant context information from the first participant device, wherein the first participant context information includes one or more of the following: the participant audio state of the audio component, the participant video state of the video component, and the participant shared content state of the shared content component.

13. The system of claim 12, wherein the first context processor is configured to generate the mismatch notification in the following circumstances: The presenter audio status indicates the presence of the audio component, and the participant audio status indicates the absence of the audio component, The presenter video status indicates the presence of the video component, and the participant video status indicates the absence of the video component, or The presenter shared content status indicates the presence of the shared content component, and the participant shared content status indicates the absence of the shared content component.

14. The system according to claim 12, wherein the conference session has a plurality of participants and at least one presenter sharing data, wherein the at least one presenter sharing data comprises: a first audio component, and the party audio status indicates the absence of the first audio component, a first video component, and the participant video status indicates the absence of the first video component, or A first shared content component is present, and the participant shared content status indicates the absence of the first shared content component.

15. The system of claim 14, wherein the first shared content component comprises one or more of a screen sharing session, an application sharing session, a collaboration tool sharing session, and a document sharing session.

Images