Automation for inserting meeting content into screen-sharing application files
Patent Information
- Application Number
- CN202580028673.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-31
- Filing Date
- 2025-03-11
- Publication Date
- 2026-11-17
AI Technical Summary
这些情况通常导致降低的生产力和计算资源的低效使用,从而突显对更高效的解决方案和用户界面布置的需要
Smart Images

Figure CN123125612A_ABST
Abstract
Description
Background Technology
[0001] There are many different types of collaborative systems that allow users to communicate. For example, some systems allow people to collaborate by sharing content such as video and audio streams, shared files, and chat messages. Some systems manage communication sessions, which are also referred to herein as online meetings, virtual meetings, broadcasts, etc. Such sessions can include events that occur on a specific date with defined start and end times. People can schedule these sessions on a calendar and schedule multiple events throughout the day. Users can pre-schedule meetings, invite other participants, and use various content-sharing features such as audio, video, chat, screen sharing, whiteboards, etc.
[0002] While some existing systems offer multiple features that allow people to collaborate during specific events, such systems have several drawbacks. For example, when users share content using different productivity applications (such as word processing, spreadsheet, or presentation applications), these applications often operate independently of the communication programs that manage the shared video and audio streams. Therefore, during a meeting, when users share a view of their desktop or a view of an application displaying files to share the content of a specific document, the person sharing the document is required to manually provide editing to the document while also attending to discussions about it. Users may have to switch back and forth between the communication application and the document application to perform all tasks. This is because productivity applications such as Word, Excel, or PowerPoint operate independently of the communication applications that manage the meeting (e.g., Microsoft Teams, Google Meet, etc.). During a meeting, a person may have to switch from a communication application to view text messages and then switch back to the productivity application to edit a document. This could involve minimizing the communication program's window, selecting the productivity application as its main window, and then providing input for editing the document. This could distract users who are editing and cause them to miss shared content and important information during the meeting.
[0003] If the document is stored on a server or cloud service, the aforementioned technical issues can also lead to additional problems. If a user wants to share a document with other participants, they must upload the document to a storage service, configure permissions to allow others to edit it, and then provide additional input to share the document via a screen-sharing session. Such tasks increase distractions from the meeting flow while still requiring the user to manually edit the shared document and participate in communications with other participants. These situations typically result in reduced productivity and inefficient use of computing resources, highlighting the need for more efficient solutions and user interface layouts. Summary of the Invention
[0004] The disclosed technology provides a system for automatically inserting content from meeting summaries and meeting transcripts into application files transmitted during meeting screen sharing. Generally, the disclosed technology manages and enriches meeting transcripts, meeting summaries, meeting records, and screen-shared files during online meetings. During an online meeting, when a presenter screen-shares application files such as Word documents, PowerPoint presentations, or Excel files, the system creates a meeting transcript or summary for the live discussion based on audio signals and / or chat messages related to the shared content of the application file. The system determines the location of the application file and inserts at least a portion of the transcript or summary into the application file. The system can perform these operations on application files residing locally on the presenter's machine or on remote storage services (such as OneDrive, Google Drive, etc.).
[0005] The system can also automatically control permissions for accessing and modifying application files, as well as transcripts or summaries. For example, the person sharing a document in a screen-sharing session can be the only one with permission settings to access and modify that file at the start of the meeting. However, during the screen-sharing session, when a meeting transcript or summary is generated, the system can insert at least a portion of the selected content into the application file. Furthermore, the system updates the file permission settings to allow other members of the meeting to access and modify the file. If the file is stored locally on the presenter's local device, a copy of the file is made to a storage server to allow meeting members to access it. Permissions for meeting communication data, such as voice audio communications and messaging communications, are also configured to allow each member of the meeting to access the meeting communication data using links also located in the application file. Links in the file can provide users with playback of audio clips associated with a portion of the file.
[0006] This paper addresses the technical challenge of simultaneously accessing shared files and meeting summaries or transcripts by providing an enhanced automated technical solution for inserting portions of communication data into application files displayed during screen-sharing sessions. The system offers improved user interaction by providing access to information that is typically difficult to view, as meeting data and file content are traditionally controlled by different applications. Permissions are also automatically set, so users do not need to interrupt the meeting to set access permissions. Therefore, in addition to improved system security, the techniques disclosed herein also offer numerous other computational resource efficiencies. By providing automatic link insertion for meeting content, meeting participants can keep track of the meeting and spend less time controlling permissions and views of meeting content, focusing on key points with minimal disruption.
[0007] When information and permission transitions are organized more precisely and involve less manual input, audience members are less likely to miss key information during an event, and this automation of finer-grained control over permissions also eliminates the possibility of human error when permissions are manually set. These benefits improve the efficiency and security of computing systems by reducing the number of times users need to interact with computing devices to obtain information or restrict access to it. For example, if users in a meeting miss shared content due to inefficient human interaction, they may have to resort to extended meetings, excessive use of meeting minutes, or requests for copies of previously shared content that might require an email system. Therefore, various computing resources, such as network resources, storage resources, and processing resources, can be reduced by mitigating scenarios where content is missed or unintentionally restricted.
[0008] As stated above, the features claimed for protection result in a reduction in manual data input required by the user. By reducing the need for manual input, unintentional input and human error can be reduced. This can ultimately lead to a reduction in undesirable license settings that could leave attack vectors in shared content, and more efficient use of computational resources such as memory usage, network usage, and processing resources.
[0009] The techniques disclosed herein also provide improved access to meeting summaries and transcripts. By automatically inserting meeting summaries and transcripts into application files, the disclosed system presents users with access to information, allowing for better and more efficient interaction on general computers, especially for users with small-screen devices. The disclosed features are particularly useful on small-screen devices and other devices because users do not have to switch between programs and communication programs to access meeting content and productivity file content.
[0010] Features and technical benefits beyond those explicitly described above will become apparent upon reading the following detailed description and the associated accompanying drawings. This summary is provided to present, in a simplified form, a selection of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technology" can refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic, and / or operations permitted in the context and throughout this document as described above. Attached Figure Description
[0011] Detailed embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numeral of the reference numeral indicates the drawing in which the reference numeral first appears. Identical reference numerals in different drawings indicate similar or identical items. References to individual items among a plurality of items may use reference numerals with letters in a letter sequence to refer to each individual item. Generic references to these items may use specific reference numerals without a letter sequence.
[0012] Figure 1A The first stage of the process of inserting a link to a file into the transcription is shown, illustrating the user interface displayed on the first computer of the system in normal meeting mode.
[0013] Figure 1B The user interface shown is displayed in normal meeting mode, which receives input for switching the computer from normal meeting mode to screen sharing mode.
[0014] Figure 1C The user interface displayed on a first computer in screen-sharing mode is shown, which identifies selected content from communication data to generate a summary or transcript.
[0015] Figure 1D The user interface on the first computer in screen-sharing mode is shown, where additional communication data for transcription or summarization is received from meeting participants.
[0016] Figure 1E The user interface on the first computer in screen-sharing mode is shown, where the first participant displays the shared content of a file in a productivity application, with links to the file inserted into the transcript or summary.
[0017] Figure 1F The user interface is shown, in which links to files are inserted into the meeting transcript or summary, and hovering the input invokes the display of the links to files.
[0018] Figure 2A The user interface displayed on the second computer shows a transcript or summary with links to files displayed in the conference's screen sharing.
[0019] Figure 2B The user interface displayed on the second computer in normal operating mode is shown. The second computer displays files in response to the selection of links to files in the meeting minutes or meeting summary.
[0020] Figure 3A The diagram shows a user interface displayed on a first computer in screen-sharing mode, where the first computer or system analyzes communication data to identify selected content for a meeting summary.
[0021] Figure 3B The user interface on the first computer in screen-sharing mode is shown, where additional communication data is received during the meeting but is not selected to be included in the meeting summary.
[0022] Figure 3C The user interface on the first computer in screen-sharing mode is shown, where additional communication data is received during the meeting and selected to be included in the meeting summary, which includes links to files displayed in the shared desktop environment.
[0023] Figure 4A A block diagram of a system for identifying the location of files stored on a client device is shown.
[0024] Figure 4B A block diagram of a system for identifying the location of files stored on a client device is shown, which performs operations for moving or copying files to a server.
[0025] Figure 4C A block diagram of a system for identifying the location of a file stored on a client device is shown, which performs operations to generate links to the file for inclusion in a transcription or summary.
[0026] Figure 5A A block diagram of a system for identifying the location of files stored on a server is shown.
[0027] Figure 5B A block diagram of a system for identifying the location of files stored on a server is shown, which performs operations to generate links to the files for inclusion in transcription or summaries.
[0028] Figure 6A The first stage of the process of inserting selected content from conference communications into a file is shown, illustrating the user interface displayed on the first computer in normal conference mode.
[0029] Figure 6B The user interface shown is for receiving input to switch from normal meeting mode to screen sharing mode.
[0030] Figure 6C The diagram shows a user interface displayed on a first computer in screen sharing mode for inserting selected content of communication data into a file. This stage shows communication data received during the meeting but not selected for inclusion in the file displayed in screen sharing.
[0031] Figure 6D The user interface displayed on the first computer in screen sharing mode is shown, where additional communication data is received and content is selected for insertion into a file.
[0032] Figure 6E The user interface displayed on the first computer in screen sharing mode is shown, where additional communication data is received and content describing the editing of the file is selected to be inserted into the file.
[0033] Figure 6F The user interface displayed on the first computer in screen sharing mode is shown, where the system automatically edits the file during screen sharing mode using a description of editing a file derived from communication data from a conference.
[0034] Figure 7A The user interface displayed on the second computer shows selected content from conference communication data inserted into the conference, as well as links to audio clips of the conference that include the selected content.
[0035] Figure 7B The user interface for displaying a file is shown on a second computer in normal operating mode, where the second computer plays an audio clip from a meeting that includes the selected content.
[0036] Figure 8A A block diagram of a system for identifying the location of files stored on a client device is shown.
[0037] Figure 8B A block diagram of a system for identifying the location of files stored on a client device is shown, which performs operations for moving or copying files to a server.
[0038] Figure 8C A block diagram of a system for identifying the location of a file stored on a client device is shown, which performs operations to insert selected content from conference communication data into a file that is copied or moved to a server.
[0039] Figure 9A A block diagram of a system for identifying the location of files stored on a server is shown.
[0040] Figure 9B A block diagram of a system for identifying the location of files stored on a server is shown, which performs operations to insert selected content from conference communication data into files stored on the server.
[0041] Figure 10A An example of a user interface that receives instructions during a screen-sharing session, resulting in the generation of new content data, is shown.
[0042] Figure 10B An example is shown of generating new content in response to instructions received during a screen-sharing session.
[0043] Figure 11 An example of a system that uses AI models to analyze communication data is shown.
[0044] Figure 12 This is a flowchart illustrating various aspects of routines used to perform the techniques disclosed herein.
[0045] Figure 13 This is a diagram illustrating a distributed computing environment capable of implementing the technologies and aspects of the technologies presented in this article.
[0046] Figure 14 This is a computer architecture diagram illustrating the computing device architecture capable of implementing the technologies and aspects of the technologies presented in this article. Detailed Implementation
[0047] Appendix Figure 1A-1F A system 100 is shown for automatically inserting links to application files into conference transcripts and conference summaries. For example... Figure 1A As shown, system 100 includes multiple computing devices 11, each associated with a user 10. The computers interconnect using communication sessions for sharing video signals, audio signals, and other shared content (e.g., documents). In this example, multiple users (10A-10H) are present in the meeting, where user A 10A (Serena Davis) is associated with the first computing device 11A, user B 10B (Miguel Silva) is associated with the second computing device 11B, user C 10C (Krystal McKinney) is associated with the third computing device 11C, user D 10D (Jazmine Simmons) is associated with the fourth computing device 11D, user E (Mahendra Sekaran) is associated with the fifth computing device 11E, user F 10F (Michael Wong) is associated with the sixth computing device 11F, user G 10G (Will Newman) is associated with the seventh computing device 11G, and user H 10H (Cassie Price) is associated with the eighth computing device 11H. Each user's device is configured to participate in a communication session, such as a conference, at a time configured according to a calendar. As described in more detail below, system 100 may also include a server for managing conferences using communication applications and storage services.
[0048] Figure 1A-1FA process for automatically inserting links for accessing application files into meeting transcripts and meeting summaries is also illustrated. In this example, during an online meeting, a presenter, such as user A using a first device 11A, initiates a screen-sharing application file, such as a Word document, PowerPoint presentation, Excel file, etc. This file resides locally on the presenter's machine or on a server (e.g., a OneDrive server). The system then creates a meeting transcript or summary for the live discussion based on audio signals and / or chat messages associated with the shared content (e.g., the shared file). The server also determines the location of the application file; a link to the application file is then inserted into the transcript / summary. As described in more detail below, the method shown in the following figures includes operations for generating links for accessing application files displayed in screen-sharing mode during the meeting, and operations for inserting the links into the meeting summary or meeting transcript. Participants viewing the meeting transcript or meeting summary can then access the application file that was screen-shared during their meeting by selecting a link in the meeting transcript or meeting summary.
[0049] A meeting transcript or summary can be generated by identifying selected content from the communication data exchanged between participants in an online meeting. The selected content in the communication data can include any content shared in audio streams or messages shared between the participants of the meeting. Figure 1A and Figure 1B An example is shown where a first device is initiated in a first operating mode (e.g., normal meeting mode), in which audio streams and messages are shared among users, but the device does not share the display of the desktop environment; this sharing could also be the sharing of an application's presentation of files to other meeting participants. These operations include invoking a communication session (e.g., a conference) in normal meeting mode to transmit audio streams and messages between multiple computing devices 11A-11H, wherein normal meeting mode is invoked and maintained when computing devices 11A-11H are not in a screen-sharing mode for transmitting application files.
[0050] Figure 1A The first stage of the process of displaying a user interface on the first device 11A in normal meeting mode is illustrated. In normal meeting mode, the first computer 11A can record transcripts and generate summaries, while allowing meeting participants to communicate using text, audio, and video data. In normal meeting mode, the user can select menu item 105 to display a shared menu.
[0051] Figure 1BThe user interface displayed in normal meeting mode is shown, where the first device 11A receives input at the sharing menu 201 to switch the device from normal meeting mode to screen sharing mode. The first menu item 201A can be selected to share files, the second menu item 201B to share a specific window, and the third menu item 201C to display a shared desktop on the screens of other users' other devices 11B-11H. In this example, the user input indicates the selection of the third menu item 201C, which invokes screen sharing mode, where the first device displays a rendering of its desktop to the other devices 1B-11H in the communication session.
[0052] Figure 1C The user interface displayed on the first device 11A in screen-sharing mode is shown. In screen-sharing mode, the first device displays a rendering of its desktop (referred to as shared desktop 131) on the screens of all participating devices 11A-11H. The shared desktop also includes individual user renderings 142. These renderings may display live video streams of individual meeting participants or still images of individual participants. These renderings may be statically located on the desktop and appear on other windows displayed on the desktop. Furthermore, control element 143 can be configured to terminate screen-sharing mode and return the device to normal meeting mode in response to a user selection. Although this embodiment relates to desktop sharing, content sharing or file sharing also includes any technical features where a computer uses an application to display a file, and a rendering of that file is displayed on the computers of other meeting participants.
[0053] In some embodiments, a screen sharing mode is activated in response to at least one computer invoking screen sharing of a desktop displaying files using a productivity application. For example, during a communication session, a first computing device 11A invokes a screen sharing mode that includes executing a productivity application that displays shared content 121 of application file 110. The communication application causes a rendering of a desktop environment 131 shared by the first computing device 11A to be displayed on the display screens of multiple computing devices 11A-11H in the communication session.
[0054] When in screen-sharing mode, the system analyzes communication data to identify selected content for a meeting summary or meeting transcript. In some embodiments, the system creates a meeting transcript or summary for a real-time discussion based on audio signals and / or chat messages related to the shared content. As shown, as participants converse in the meeting, the system captures each user's audio stream, transcribes the audio stream into text, and inserts the text into the meeting record or meeting summary. For illustrative purposes, the meeting summary or meeting transcript may be referred to herein as "data structure 124". This may include operations for processing audio streams and messages transmitted between multiple computing devices 11A-11H to identify selected content 123 from the audio streams and messages of a communication session. For illustrative purposes, the selected content may also be referred to herein as content 123 extracted from communication data.
[0055] Figure 1C Additional aspects of client devices (e.g., first device 11A) are also illustrated. Any client device 11 that invokes screen-sharing mode may include the execution of a communication application 120 that manages communication between participants. The communication application may also manage the shared desktop 131 displayed on each participant's client device during the meeting. The shared desktop 131 is displayed on the display screen using a corresponding desktop UI object 131'. The communication application may also generate, maintain, and store meeting minutes or meeting summaries that include selected content 123. The client device may also include a productivity application 122 configured to display and modify the content of document 110. The content of document 110 is referred to herein as shared content 121. The productivity application 122 is also referred to herein as an application 122 configured to edit and display the content of a document. This may include a word processing application configured to open and edit documents, a spreadsheet application configured to open and edit spreadsheets, a presentation application for opening and editing presentation files, etc.
[0056] Figure 1DThe user interface on the first computer in screen-sharing mode is shown, which is displayed as additional communication data is received from the meeting participants. As shown, the system captures each user's audio stream in real time and transcribes their speech into text as the meeting progresses. If the communication data meets one or more criteria, the text of the communication data is included in data structure 124. For example, if the data structure is a transcription, the system determines that the majority of the communication data is selected, except for some filtered words and phrases, such as profane language, controversial terms, etc. However, if the data structure is a meeting summary, the system selects only the portion of the communication data that meets one or more criteria. For example, if the statement relates to an application file, a specific context, a specific person, or a specific topic, the selected portion of the communication is determined as the selected content of the summary. In this particular example, Serena and Miguel only exchange general greetings and are therefore not included in the summary. This example is provided for illustrative purposes and should not be construed as restrictive. As described in more detail below, any suitable criteria can be used to derive the selected content from the communication data.
[0057] Figure 1E A user interface on a first computer in screen-sharing mode is shown, where a first device 11A of a first participant 10A displays the shared content of a file in a productivity application. This user interface is displayed in response to a file being selected for screen-sharing mode during a meeting. The system generates a link to the file and inserts the link into a transcript or summary, such as data structure 124. While the user is viewing the transcript or summary, the link may be displayed as a button 130 (graphical element 130) or hyperlink text within the transcript or summary to provide access to the link. Figure 1F The user interface is shown, in which user input (such as mouse hover input on button 130) invokes a link to the file.
[0058] In some embodiments, a link 111 to file 110 is generated based on one or more triggering events. For example, when a user shares a file using a productivity application via a shared desktop, the system may generate a link 111 to file 110. In another example of a triggering event, the link may be generated in response to a predetermined input gesture or predetermined voice command stated in an audio stream or message during a meeting. A person may state in the audio stream that they “want access to this file,” or someone may state that “the permissions for this file should be changed to grant access to everyone,” etc. These types of commands can cause the system to generate a link to the file, but also change the permissions for the file itself.
[0059] Figure 2A-2B An example is shown of how links in the transcription / summary can be used to make file 110 visible. Figure 2AA user interface displayed on a second computer is shown, which displays a transcript or summary with links 111 to a file displayed during screen sharing. Figure 2B The user interface displayed on the second computer in normal operating mode is shown, in which file 110 is displayed in a productivity application invoked in response to the selection of a link to a file.
[0060] In some embodiments, a link (or "reference") to a file can cause a portion of the application to be displayed. A portion of the application file may include a page of a document, a selected cell of a spreadsheet, a page of a slide group, etc. Each page of the document can be identified by an identifier, referred to herein as a section identifier. Thus, as Figure 2A As shown, the system enables the display of data structures (meeting summaries or meeting minutes) on a computer (such as the second computing device 11B). The display of the data structures includes content derived from communication data and references to application files. Then, as... Figure 2A As shown, the computer receives input indicating a selection of a section identifier having a reference for a section of an application file. This may include a specific page number. In response to the input indicating a selection of a section identifier having a reference for a section of an application file, the system causes the application to identify the section of the application associated with the section identifier and causes the section of the application file associated with the section identifier to be displayed.
[0061] Figures 3A-3B Another example is shown where transcripts of a meeting are analyzed to identify selected content from the meeting summary. In such an embodiment, the system can analyze communication data and identify selected content based on one or more criteria. Transcripts can include all words spoken during the meeting. Alternatively, the system can generate a meeting summary based on phases or actions related to a specific topic or person. For example, such as... Figure 3A As shown, transcripts generated from audio streams can begin with a general statement, such as "Welcome everyone. Thank you for coming to the meeting." If the system determines that the statement does not relate to a specific person, topic, or shared application file, it will not select the statement for inclusion in the summary.
[0062] If the system determines that a statement or action relates to a specific person, topic, or shared application files, it selects those statements or descriptions of those actions to include in the summary. For example, Figure 3B This shows a statement made by another person regarding a shared document. In response, the system includes such a statement or a description of such a statement in the summary.
[0063] The system can also identify selected content to be included in the summary based on the detection of pre-defined gesture inputs or specific activities in the meeting. For example... Figure 3CAs shown in the example, when someone shares a specific file on a shared desktop, the system can generate a description of the action, such as Serena sharing a desktop with an economics document. Furthermore, the system provides a link to the shared application file 130. Because Serena also provides statements related to the document in the audio stream, such as thank you for reviewing the document, such statements or descriptions of those statements are included in the summary.
[0064] Figure 3C The example also illustrates that the system can determine the location of link 130 within the summary. In some embodiments, the location of the link can be based on the timing of an input gesture or statement. For example, if a person is sharing a file at a specific time during a meeting, and the summary's outline is based on a timeline, a link to the file is inserted at a location associated with the location of the relevant statement. For instance, a link to the file could be placed next to the user's statement, "I am now sharing a file on my desktop." In another example, a link to the file could be placed next to a description of a predetermined input or action. When a user provides input to begin sharing a document, the system can generate a statement in the record or summary indicating that the user has begun sharing the file. The system can then position the link to the file close to the location associated with that computer-generated statement.
[0065] The system can identify the location of a file for the purpose of generating a link to it. Figures 4A-4C This illustrates an example of the process by which the system identifies the location of a file stored on a client device. For example... Figure 4A As shown, when sharing a file in screen sharing mode, the system identifies the location of the file by sending a query to productivity application and / or storage service 222 to retrieve a path indicating the storage location of the application file. In this example, the path indicates that application file 110 is stored on client device 11A. In response to determining that application file 110 is stored on the first computing device, as... Figure 4A and 4B As shown, the system copies or moves application file 110 from the first computing device to the storage service 222 of server 200. Then, as... Figure 4C As shown, the system generates links to provide access to application files stored on storage service 222. Also, Figure 4C As shown, the system configures permissions for links to allow attendees of a communication session to use those links to view and edit application files. These changes can also be applied to copies of files stored on client device 11A.
[0066] Figures 5A-5B This illustrates an example of the process by which the system identifies the location of a file stored on server 200. For example... Figure 5AAs shown, when sharing a file in screen sharing mode, the system identifies the file's location by sending a query to productivity application and storage service 222 to retrieve the path indicating the storage location of the application file. In this example, the path indicates that application file 110 is stored on storage service 222 of server 200. Figure 5A and 5B As shown, in response to determining that application file 110 is stored on storage service 222 of server 200, the system generates link 111 to provide access to application file 110 stored on storage service 222. Also... Figure 5B The system configuration allows access to links so that attendees of a communication session can use those links to view and edit application files.
[0067] Figures 6A-6F This illustrates a process for automatically inserting content from a meeting summary and / or meeting transcript into an application file transmitted during a meeting screen-sharing session. In this example, during an online meeting, a presenter, such as user A using a first device 11A, initiates a screen-sharing application file, such as a Word document, PowerPoint presentation, Excel file, etc. This file resides locally on the presenter's machine or on a server (e.g., a OneDrive server). The system then creates a meeting transcript or meeting summary for the live discussion based on audio signals and / or chat messages related to the shared content. The server also determines the location of the application file being shared on the computing device in screen-sharing mode; and then inserts at least a portion of the meeting transcript or meeting summary into the application file.
[0068] When a user is in screen-sharing mode, a meeting transcript or summary can be generated by identifying selected content from the communication data exchanged between participants in an online meeting. The selected content in the communication data can include anything shared in the audio stream between participants or in messages exchanged between participants. Figure 6A and 6B An example is shown where the first device begins in a first operating mode (e.g., a normal meeting mode), in which audio streams and messages are shared between users, and no device in the meeting shares the display of the desktop environment. These operations include invoking a communication session (e.g., a conference) in normal meeting mode for transmitting audio streams and messages between multiple computing devices 11A-11H, wherein normal meeting mode is invoked when computing devices 11A-11H are not in a screen-sharing mode for transmitting application files.
[0069] Figure 6AThe first stage of the process of displaying a user interface on the first device 11A in normal meeting mode is shown. In normal meeting mode, the first computer 11A can record transcripts and generate summaries, while allowing meeting participants to communicate using text, audio, and video data. In normal meeting mode, the user can select menu item 105 to invoke a shared menu.
[0070] Figure 6B The user interface displayed in normal meeting mode is shown, where the first device 11A receives input at the sharing menu 201 to switch the device from normal meeting mode to screen sharing mode. The first menu item 201A can be selected to share files, the second menu item 201B to share a specific window, and the third menu item 201C to display a shared desktop on the screens of other users' other devices 11B-11H. In this example, the user input indicates the selection of the third menu item 201C, which invokes screen sharing mode, where the first device displays a rendering of their desktop to the other devices 11B-11H in the communication session.
[0071] Figure 6C The user interface displayed on the first device 11A in screen-sharing mode is shown. In screen-sharing mode, the first device displays a rendering of a desktop, referred to as shared desktop 131, on the screens of the other devices 11A-11H participating in the conference. The shared desktop also includes individual user renderings 142. These renderings can display a live video stream for each user or a still image of each user. These renderings can be statically located on the desktop and appear on other windows displayed on the desktop. Furthermore, control element 143 can be configured to terminate the screen-sharing mode and return to normal conference mode in response to a user selection.
[0072] In some embodiments, a screen sharing mode is activated in response to at least one computer invoking screen sharing of a desktop displaying a file using a productivity application. For example, during a communication session, a first computing device 11A invokes a screen sharing mode that includes executing a productivity application displaying shared content 121 of application file 110. The communication application causes a rendering of a desktop environment 131 shared by the first computing device 11A to be displayed on the display screens of multiple computing devices 11A-11H in the communication session. Figure 6C As shown, the desktop environment 131 includes the rendering of the shared content 121 of the application file 110.
[0073] When in screen-sharing mode, the system is configured to identify selected content from communication data for inclusion in a file. In some embodiments, the system creates a meeting transcript or summary for a live discussion based on audio signals and / or chat messages related to the shared content. The selected content of the transcript or summary is then inserted into the screen-sharing file. As shown, as participants speak in the meeting, the system captures each user's audio stream, transcribes the audio stream into text, and inserts the text into the meeting transcript or meeting summary. For illustrative purposes, the meeting summary or meeting transcript may be referred to herein as data structure 124. In some embodiments, the system may analyze audio streams or messages exchanged between users to identify selected content 123 to be added to the transcript or summary. This may include operations for processing audio streams and messages transmitted between multiple computing devices 11A-11H to identify selected content 123 from the audio streams and messages.
[0074] Figure 6C Additional aspects of client devices (e.g., first device 11A) are also illustrated. Any client device 11 capable of invoking screen-sharing mode may include a communication application 120 that manages communication between participants. The communication application may also manage a shared desktop 131 displayed on each participant's client device during the meeting. The shared desktop 131 is managed by the communication application executing a corresponding desktop UI object 131'. The communication application may also generate, maintain, and store transcriptions or meeting summaries of selected content 123. The client device may also include a productivity application 122 configured to display and modify the content of document 110. The content of document 110 is referred to herein as shared content 121.
[0075] In some embodiments, selected portions of a meeting summary or meeting transcript are inserted into a file. If statements provided in communication data (e.g., shared messages or statements made in audio streams) are detected as originating from a specific person, relating to a specific topic, or relating to a shared application file, the system selects these statements or descriptions of these actions to include in the file. The system analyzes audio streams and messages transmitted between multiple computing devices 11A-1 IF to identify selected content that has a threshold relevance to the shared content 121 of application file 110, the person, or the topic. The system then adds the selected content with a threshold relevance level to the shared content 121 of application file 110 to a data structure and / or inserts the selected content into file 110. In some embodiments, the selected content is added to a comment area 146 of the file.
[0076] exist Figure 6C In the example, Serena's statement is not included in the document because it is irrelevant to the document or to a specific person or topic. However, as... Figure 6DAs shown, Miguel makes a statement related to the document, such as, "I like the first paragraph." Because this statement references the document, it is identified as selected content 123 and inserted into the file. However, another statement that does not reference the document, "Thank you, Serena," is not included.
[0077] Additionally, link 189 A is also inserted into the file. When selected, this link causes playback of an audio clip, which includes a portion of an audio stream containing the phrase "I like the first paragraph." Link 189 A inserted into the file is configured to play the meeting's audio stream starting at 9:02 AM, and only play 3 seconds containing the relevant statement. This link can be configured to play any scheduled time period containing selected content relevant to the shared file, a specific person, or a specific topic.
[0078] In some configurations, the system can determine the placement of selected content within a document. For example, if the selected content is a statement, such as "I like this first paragraph," the statement is placed near the first paragraph. Comments can be added as an overlay on the first paragraph or in the document's comment area. For spreadsheets and other graphics programs, selected content can be linked and activated for display via input such as mouse hover or other activation inputs.
[0079] In some configurations, the system can position selected sections of content closer to other parts of the document, such as different sections of a document. These sections can be pages, sentences, paragraphs, words, or any other part of the document. For other file formats, such as spreadsheets, sections can be cells or cells within a section.
[0080] The system can select specific parts of a document based on the level of relevance between the content of a specific part of the document and selected content derived from communication data. For example, if a comment is made about a specific number or cell in a spreadsheet, the comment can be placed as a link within the relevant cell.
[0081] Figure 6E The illustration shows the system's operation state in which it identifies additional selected content from a statement made by Serena. In this example, Serena makes a statement related to a file. In response to determining that the statement is related to a file, the statement is identified as selected content 123 and inserted into file 110. Another link 189B is also added to the file, wherein this second link 189B is configured to play a portion of the conference audio stream that includes Serena's statement.
[0082] Figure 6FThis illustrates the system's determination that Serena's statement includes editing of a file, and the system automatically performs those edits on the file's shared content. In this specific example, Serena's statement instructs that paragraphs should include indentation. In response to detecting these instructions, the system can interpret them and make those changes to the content, such as... Figure 6F As shown in the example, while this example illustrates that the system can change the format of a document, the system is also configured to identify content that must be inserted into the main content area of the document. For example, if someone suggests adding a second paragraph describing another topic, that content is selected from the audio stream or shared messages between meeting participants, and that content is inserted into the main content area. In some embodiments, edits to the document performed by the system based on editing instructions interpreted from the meeting audio stream or messages are only displayed to people with a predetermined role, such as presenters or people who provided the input that caused the edits to the document. For example, in response to Serena's statement describing the edits to the document, in some embodiments, the system makes those edits to the shared content of the document and only displays those edits to Serena's computer. Others see a version of the document without those edits. Then, once Serena approves those edits, those edits are displayed on other users' computers. This can be accomplished using the revision tracking feature of a productivity app with multiple viewing modes: View Original, which allows selected users to view the original version without redline formatting changes; View Markup, which allows selected users to view the edited version with redline edits; and No Markup, which allows selected users to view the edited version without redline formatting changes. The viewing mode can be selected for any specific user and can be chosen by any user providing input (such as UI input, input gestures, or voice commands).
[0083] Figures 7A-7B This example demonstrates how selected content exported from communications data captured during a meeting can be viewed within a file. Additionally, the example shows how a user can select a link inserted into the file to play a portion of the meeting that includes the selected content. Figure 7A The user interface displayed on the second computer shows selected content from conference communication data inserted into a file, and links to audio clips of the conference that include the selected content. Figure 7BThe diagram illustrates a user interface for displaying a file on a second computer in normal operating mode, where the second computer plays an audio segment of a meeting that includes the selected content. For example, if link 189 A inserted into the file is configured to play the audio stream of a meeting starting at 9:03:30 AM, and only play the 8 seconds containing the statement, the user's selection of that link will play only the 8 seconds of the meeting starting at the 9:03:30 AM timemark.
[0084] The system can identify the location of a file for the purpose of inserting selected content from communication data from a conference into the file. Figures 8A-8C An example of the process by which the system identifies the location of a file stored on a client device is shown. File 110 contains shared content 121, which is shown as the original content of file 110 in the example above. Figure 8A As shown, when sharing a file in screen sharing mode, the system identifies the location of the file by sending a query to productivity application and storage service 222 to retrieve a path indicating the storage location of the application file. In this example, the path indicates that application file 110 is stored on client device 11A. In response to determining that application file 110 is stored on the first computing device, as... Figure 8A and Figure 8B As shown, the system copies or moves application file 110 from the first computing device to the storage service 222 of server 200. For example... Figure 8C As shown, the system inserts the selected content 123 into file 110. The selected content 123 may also include a link to the conference audio 129. Also, Figure 8C As shown, the system configures permissions for the link to audio 129 and permissions for file 110 to allow attendees of the communication session to view and edit application files and access audio 129 segments containing selected content. These changes can also be applied to copies of files stored on client device 11A.
[0085] Figures 9A-9B This illustrates the process by which the system identifies the location of files stored on server 200. For example... Figure 9A As shown, when sharing a file in screen sharing mode, the system identifies the file's location by sending a query to productivity application and storage service 222 to retrieve the path indicating the storage location of the application file. In this example, the path indicates that application file 110 is stored on storage service 222 of server 200. Figure 9B As shown, in response to determining that application file 110 is stored on storage service 222 of server 200, the system inserts selected content 123 into file 110. Selected content 123 may also include a link to conference audio 129. Also as... Figure 9BAs shown, the system configures permissions for the link to audio 129 and for file 110 to allow attendees of the communication session to view and edit application files and access to receive audio 129 segments including selected content. The file can also be copied to a first device with the same permissions.
[0086] Figures 10A-10B An example is shown where the communication data from a meeting can be interpreted to generate content for a document. In this example, such as... Figure 10A As shown, if the system detects that the audio stream includes statements for generating new content, the system can interpret those statements and generate content based on them, automatically displaying the generated content in a file shown in desktop screen sharing. In this example, when the first device is in screen sharing mode, the system interprets statements made in the conference audio stream. In this example, Serena provides specific statements detailing a particular architectural project. In this example, the statement also includes preferences for specific applications (e.g., CAD applications). In response to the statement, the system identifies the productivity application described in the statement and causes that application to be executed. Furthermore, portions of the statements related to the content are sent to an AI model, such as a large language model. This input, along with other underlying data, also enables the AI model to generate content for the selected productivity application. In this example, as... Figure 10B As shown, the system generates content data based on voice input captured in the audio stream and displays that content using application 120 in application UI 141. When the first device is in screen-sharing mode, application UI 141 is also displayed to other meeting participants.
[0087] refer to Figure 11 The embodiments disclosed herein use an AI model to identify selected content from conference communication data. The AI model can also be used to generate new content from the conference communication data, and can also generate instructions for modifying the content of existing documents. The system can utilize an AI model such as a Large Language Model (LLM 160) to identify selected content from activity data 190, which may include messages 191 exchanged between conference participants, transcripts or summaries 193 derived from audio streams 192, or other content shared during a conference screen-sharing session.
[0088] For example, an AI model can be used to analyze activity data 190 to determine whether any portion of the activity data has a threshold level of relevance to shared content 121 of application file 110, a predetermined topic, and / or a predetermined person. A portion of the activity data may be a phrase, sentence, or word stored in audio, image, or text format. If a portion of the activity data is determined to have a threshold level of relevance to a file, topic, or person, such a portion is identified as selected content 123, such as... Figure 11As shown, selected content 123 is included in document 110 and / or is included as a summary or transcription of selected content.
[0089] The system can generate a query 153 that includes query content 152 and query parameters 154. The query content can include content filtered from the active data by selector 150. For example, the selector can sort and filter the active data to construct a query in a specific format. A complete transcription cannot be processed by the LLM, so a portion of the transcription is selected for a separate query. This is because the LLM has limitations regarding the amount of information that can be inserted into a query. The selector can analyze messages from a specific user and arrange the query content into a specific format suitable for the large language model. For example, the message and transcription can be parsed and reformatted into the string "Serena said 'Thank you for attending the meeting,' Miguel said 'I like the first paragraph.'" The query content can be arranged along with parameters 154 that provide instructions to the model, such as "Select statements related to or mentioning a file provided by the user during a screen-sharing session." The system can also include the file as part of the parameters, where the LLM will use the file as the underlying data. In response to such a query, the large language model can identify the selected content 123, such as Miguel's statement "I like the first paragraph." As shown in the figure, such content can be inserted into a document and / or into a summary.
[0090] In another example, the message and transcription data can be parsed and reformatted into a string targeting the query content, such as "Serena states that paragraphs should be indented". This query content can be arranged along with parameter 154, which provides instructions to the model, such as "Generate executable code to modify this file according to the instructions provided by Serena". This file can be included as part of the query parameters, and the query can be sent to the LLM. In response to such a query, the large language model can generate selected content as executable instructions to modify the file, such as indenting each paragraph of the file. In another example, the selected content can also be used to generate new content, and instructions can also be generated to identify the application to which the generated content should be displayed. Such examples are in... Figure 10B As shown in the diagram, the natural language description of the architecture file is sent to the large language model, resulting in the generation of a new file.
[0091] Now go to Figure 12Routine 800 is used to insert links in meeting minutes or meeting summaries containing selected content obtained from the meeting's communication data, wherein these links provide access to application files transmitted to meeting participants in desktop sharing mode. It should be understood that the operations of the methods disclosed herein are not necessarily presented in any particular order, and it is possible and contemplated to perform some or all of the operations in an alternative order. For ease of description and illustration, the operations have been presented in the order shown. Operations may be added, omitted, and / or performed simultaneously without departing from the scope of the appended claims.
[0092] It should also be understood that the illustrated method may end at any time and does not need to be fully performed. Some or all of the operations of the method and / or substantially equivalent operations may be performed by executing computer-readable instructions included on computer storage media and computer-readable media, as defined herein. The term "computer-readable instructions" and variations thereof, as used in the specification and claims, are used extensively herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions may be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc.
[0093] Therefore, it should be understood that the logical operations described herein are implemented as (1) a sequence of computer-implemented actions or program modules running on a computing system, and / or (2) interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations described herein are referred to differently as states, operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules can be implemented in software, firmware, dedicated digital logic, and any combination thereof.
[0094] For example, the operation of a routine is described herein as being implemented at least in part by an application, component, and / or circuitry (e.g., a device module that may be included in any memory component disclosed herein, including but not limited to RAM). In some configurations, a device module may be a dynamic link library (DLL), a static link library, a function enabled by an application programming interface (API), a compiler, an interpreter, a script, or any other set of executable instructions. Data received by the device module (e.g., input data or signals from sensors) may be stored in a data structure within one or more memory components. Data can be retrieved from a data structure by addressing a link or reference to that data structure.
[0095] Although the following description refers to the components depicted in this application, it will be understood that the operation of the routine can also be implemented in many other ways. For example, the routine can be implemented at least in part by a processor or circuitry of another remote computer (which may be a server), or by a local processor or circuitry of a local computer (which may be a client device receiving or sending messages). Any aspect of the routine can be executed on the device sending the message, the device receiving the message, or on a server managing the message communication of the thread, and the routine may include generating prompts, sending any message along with the prompts to an NLP algorithm, using an NLP algorithm, or displaying the results generated by an NLP algorithm. Additionally, one or more operations of the routine can be implemented, alternatively or additionally, at least in part, by a chipset that works alone or in combination with other software modules. Any service, circuitry, or application suitable for providing input data indicating the state of any device can be used in the operations described herein.
[0096] The routine begins at operation 802, where the system invokes a communication session in normal conference mode. In normal conference mode, client devices share audio streams, video streams, and messages, referred to herein as communication data. In normal conference mode, also referred to herein as normal operating mode, none of the client devices are sharing their desktop screen with other client devices.
[0097] At operation 804, at least one client device of the system invokes screen sharing mode for an application file. For example, a client device such as first device 11A invokes screen sharing mode during a communication session. In operation 804, first computing device 11A executes a productivity application that displays the shared content 121 of the selected application file 110. The system causes a rendering of a desktop environment 131 shared by first computing device 11A to be displayed on the display screens of multiple computing devices 11A-11H in the communication session. Desktop environment 131 includes rendering the shared content 121 of application file 110 on each of the multiple computing devices 11A-11H in the communication session.
[0098] At operation 806, when in screen-sharing mode, the system analyzes audio streams and / or messages exchanged between multiple computing devices 11A-11H in a communication session. The system identifies selected content to be added to a transcript or summary based on one or more factors. In one example, the system identifies selected content from the conference's communication data. The communication data includes voice discussions encoded from audio streams or text derived from messages exchanged between client devices in the communication session.
[0099] At operation 808, the system generates either meeting minutes or a meeting summary, either or both of which can be stored in data structure 124. Data structure 124 includes selected content derived from communication data. Data structure 124 also includes metadata, which includes the identity of the user associated with the selected content, as well as other data associated with the selected content, timestamps, etc.
[0100] In some embodiments, selected portions of a meeting summary or meeting transcript are inserted into a file, transcript, and / or summary. If statements provided in communication data (e.g., shared messages or statements made in audio streams) are detected as originating from a specific person, relating to a specific topic, or belonging to a shared application file, the system selects these statements or descriptions of these actions to include in the file, transcript, and / or summary. The system analyzes audio streams and messages transmitted between multiple computing devices 11A-11F to identify selected content that has a threshold level of relevance to the shared content 121 of the application file 110, the person, or the topic. The system then adds the selected content with a threshold level of relevance to the shared content 121 of the application file 110 to a data structure and / or inserts the selected content into the file 110. In one example, if a person in the meeting has a specific name or title, such as CEO or editor-in-chief, the system may be more inclined to select content received from such a participant. In another example, if a person says they want to edit or change the content of a file, the system may be more inclined to select content from such a statement.
[0101] At operation 810, the system determines the location of application file 110 (also referred to herein as file 110). File 110 may be located on the storage device of the client computer in screen-sharing mode, or it may be located on a server. Operation 810 may include sending a query to the productivity application being used to edit and display file 110, or sending a query to storage service 222, to retrieve a path indicating the storage location of the application file. In response to determining that the application file is stored on the first computing device, the system copies or moves the application file from the first computing device to the storage service.
[0102] At operation 812, the system generates a link to provide access to application files stored on the storage service. Also in operation 812, the system configures permissions for the link to allow attendees of the communication session to use the link to view and edit the application files. Permissions can also be restricted to read-only permissions for meeting participants.
[0103] At operation 814, the system inserts a link to a file into the transcription or summary, for example, into data structure 124. Operation 814 may include inserting a link 111 to application file 110 into data structure 124 storing selected content 123. When displayed in a productivity application, the application file causes the display of this link to occur simultaneously with the display of shared content stored within that file. Figure 1E An example of such an operation is shown in the figure.
[0104] At operation 814, the system inserts the selected content into the file. This can include inserting text from an audio stream of a user commenting on the file, such as... Figure 6D and 6E The example shown illustrates this. Selected content can be inserted into the comment section or the main content section of the file. Selections can also allow one or more programs to edit the shared content of the file by interpreting commands derived from the analysis of speech captured in the conference audio stream.
[0105] The routine may also include operations for displaying application file 110 via links stored in a data structure. For example, such as... Figure 2A-2B As shown, the server enables a computer (e.g., a second computer) to access a file by selecting links in the transcript or summary. This can include displaying the data structure on a second computing device 11B, for example, when a person is in a meeting program and is viewing a display including selected content (e.g., comments made during the meeting) and a link to the application file. In response to input at the client device, the client device enables the display of the application file using a productivity application.
[0106] As mentioned above, such as Figure 3A As shown in -3D, the system can select specific portions of audio communications or messages to generate a meeting summary. This summary can originate from real-time discussions based on audio signals and / or chat messages related to shared content. The system can process audio streams and messages transmitted between multiple computing devices 11A-11F to identify selected content 123 from the audio streams and messages. The system also analyzes the audio streams and messages transmitted between the multiple computing devices 11A-11F to identify selected content with a threshold level of relevance to the shared content 121 of application file 110. The system then generates a meeting summary that includes the selected content with a threshold level of relevance to the shared content of the application file. This meeting summary is stored in a data structure where links to the file are located within the meeting summary based on the position of the selected content within the meeting summary.
[0107] In the above operations, the system can also determine the file's location for the purpose of generating a link to the file or for the purpose of inserting selected content into the file. These operations may include sending a query to the productivity application and storage service 222 to retrieve a path indicating the storage location of the application file. The server then determines that the application file is stored on a first computing device 11A among a plurality of computing devices 11A-11H. In response to determining that the application file is stored on the first computing device: the system then copies or moves the application file from the first computing device to the storage service and configures the link to provide access to the application file stored on the storage service. The system then configures permissions for the link to allow attendees of the communication session to use the link to view and edit the application file.
[0108] like Figures 5A-5B As shown, even if the file is stored on a server, the operations described herein can determine the file's location. This may include sending a query to the productivity application and storage service 222 to retrieve a path indicating the storage location of the application file. The server can then determine, based on the path received from the storage service or by the productivity application displaying the file, that the application file is stored on the storage service 222 communicating with the first computing device 11A performing screen sharing. In response to determining that the application file is stored on the storage service, the system configures the link to provide access to the application file stored on the storage service, and configures a permission for the link to allow participants in the communication session to use the link to view and edit the application file. The permission for the link allows attendees of the communication session to view and edit the application file when accessing it using the link.
[0109] The data structures described herein may include meeting transcripts, meeting summaries, or meeting records. Data structures storing selected content include one of the following: a summary of a communication session, a transcript derived from an audio stream received from a self-computing device, or an audio file containing a record of an audio stream captured during a communication session. Links are embedded in the transcript or summary at positions within the transcript relative to the selected content associated with the application file. Metadata can be used to insert links into the audio file, causing the link to appear when the audio file is played using a media player application.
[0110] In some embodiments, during normal meeting mode, the computing device is restricted from adding selected content from communication data to the application file, and is also restricted from adding links to selected content from communication data to the application file.
[0111] In some embodiments, normal meeting mode is invoked only when the computing device is not in screen-sharing mode displaying application files. Normal meeting mode can be invoked when the computing device is in screen-sharing mode but not displaying any application files. Normal meeting mode can also be invoked when the computing device is displaying application files but is not in screen-sharing mode.
[0112] When screen sharing mode is turned off, the system no longer analyzes audio streams, transcripts, or summaries for inclusion in documents. Furthermore, when screen sharing mode is turned off, the system no longer includes links to files in summary or transcript data structures.
[0113] In some embodiments, content sharing modes may include screen sharing, desktop sharing, or application sharing. Screen sharing includes sharing a rendering of any content displayed on a user's display screen, where the rendering of any content is shown to meeting participants. Desktop sharing includes a rendering of an operating system environment displayed to a user, where the operating system environment is shown to meeting participants. Application sharing includes a rendering of an application that shares a file, where the rendering is shown to meeting participants. Screen sharing, application sharing, or desktop sharing modes are specific to sharing pixel data of an application file displayed on the presenter's system, rather than sharing the file itself. Sharing the file itself with others is also referred to as content sharing, such as a collaborative editing event where all users access a OneDrive file. For illustrative purposes, content sharing modes may also include sharing a rendering of a file with other meeting participants, without sharing the entire file with meeting participants.
[0114] Content sharing may include operations in which the computer determines that an event has occurred during a communication session, indicating that a screen-sharing mode has been entered. This event may include voice input from a user, such as User A, displaying a file in an application and sharing an image of the displayed content with other users in the conference. In response to determining that a screen-sharing mode has occurred during the communication session, a rendering of the file is displayed to other users in the conference, and the system identifies the location of the file, modifies the file based on an interpretation of the communication data, and causes the display of the application file with inserted content derived from the audio stream and messages of the communication session.
[0115] This disclosure is supplemented by the following provisions: Figure 1A-1F An example is shown where, during screen-sharing mode, the system inserts a link to an application file into a transcription or meeting summary.
[0116] Clause A: A method performed by a data processing system (100) for inserting a reference in a meeting transcript or meeting summary having content obtained from communication data of the meeting, the reference providing access to an application file transmitted to meeting participants in screen-sharing mode, the method comprising: appending Figure 1A and 1B The first operating mode is shown, which is audio and message communication without screen sharing. Then... Figure 1C In the middle, the system switches to screen sharing mode activated for rendering shared files. During the communication session, the first computing device (11A) for performing the application of displaying the shared content (121) of the application file (110) invokes screen sharing mode, thereby causing the rendering of the shared content (121) shared by the first computing device (11A) to be displayed on the display screens of multiple computing devices (11A-11H) in the communication session. Figure 1C-1E In screen sharing mode, the system creates a meeting transcript or summary for a real-time discussion based on audio signals and / or chat messages related to the shared content. In response to the invocation of screen sharing mode: the system analyzes audio streams or messages to identify “selected content” to be added to the transcript or summary; processes audio streams and messages transmitted between multiple computing devices (11A-11H) to extract the content of the communication session from the audio streams and messages (123); determines an event that screen sharing mode has occurred during the communication session; in response to determining an event that screen sharing mode has occurred during the communication session: generates a transcript or summary, referred to as a data structure, and generates a data structure (1) storing the extracted content (123). 24), the system determines the location of the application file, determines the location of the application file (110) for generating a reference (111) that provides access to the application file (110), generates a reference (111) to the application file (110), and inserts a link to the application file into the transcription / summary, inserts the reference (111) to the application file (110) into the data structure (124) of the stored content (123); based on the data structure (transcription / summary), causes the extracted content transcription / summary content and the reference (link) to the application file to be displayed; receives input indicating the selection of the reference; and in response to the input, causes the application file to be displayed according to the reference.
[0117] Clause B: The method according to the clause described herein, wherein processing audio streams and messages transmitted between multiple computing devices (11A-11F) to extract content from the audio streams and messages (123) includes: analyzing the audio streams and messages transmitted between the multiple computing devices (11A-11F) to identify content with a threshold level of relevance to a predetermined topic; and generating a meeting summary including content with a threshold level of relevance to the predetermined topic, the meeting summary being stored in a data structure wherein references to documents are located within the meeting summary based on the position of the content within the meeting summary.
[0118] Appendix Figure 3A-3D: Meeting Summary: Select specific portions of audio communications or messages for use as a meeting summary. For example, a summary for a live discussion is based on audio signals and / or chat messages related to the shared content.
[0119] Clause C: The method according to the clause described herein, wherein processing audio streams and messages transmitted between multiple computing devices (11A-11F) to extract content (123) from the audio streams and messages includes: analyzing audio streams and messages transmitted between multiple computing devices (11A-11F) to identify content that has a threshold level of relevance to shared content (121) of the application file (110); and generating a meeting summary including the relevance of the content to the shared content of the application file at the threshold level, the meeting summary being stored in a data structure wherein references to the file are located within the meeting summary based on the position of the content within the meeting summary.
[0120] Figures 4A-4C Determine the location of the file. Scenario A: The file is stored on the client.
[0121] Clause D: The method according to the clause described herein, wherein determining the location of the application file includes: sending a query to the application and storage service (222) to retrieve a path indicating the storage location of the application file; determining that the application file is stored on a first computing device (11A) among a plurality of computing devices (11A-11H); in response to determining that the application file is stored on the first computing device: copying or moving the application file from the first computing device to the storage service, and configuring a reference to provide access to the application file stored on the storage service, and configuring a permission of the reference to allow participants in the communication session to use the reference to view and edit the application file.
[0122] Figures 5A-5B : Determine the location of the file. Scenario B: The file is stored on the server.
[0123] Clause E: According to the method described herein, determining the location of the application file includes: sending a query to the application and storage service (222) to retrieve a path indicating the storage location of the application file; determining that the application file is stored on the storage service (222) communicating with the first computing device (11A) performing the screen sharing; and in response to determining that the application file is stored on the storage service: configuring the reference to provide access to the application file stored on the storage service, and configuring a permission for the reference to allow participants in the communication session to use the reference to view and edit the application file.
[0124] Figure 11 : Use AI models to interpret communication data to generate a summary or metadata of the transcription.
[0125] Clause D: The method according to the clauses described herein further includes generating a query, generating a query for a large language model that includes activity data, such as messages and transcripts derived from the audio stream of the conference and conference participant data, the query including instructions that cause the large language model to identify a portion of the activity data that is associated with at least one of: the application file, the person who made the reservation, the pre-selected topic, or instructions for modifying the shared content of the application file; transmitting the query to the large language model such that the large language model generates the content based on the portion of the activity data that is associated with at least one of the application file, the person who made the reservation, the pre-selected topic, or instructions for modifying the shared content of the application file;
[0126] Clause E: The method according to the clauses described herein further includes: configuring a license for the reference to allow attendees of the communication session to view and edit the application file when accessing the application file using the reference, wherein the data structure for storing the content includes at least one of the following: a summary of the communication session, a transcript derived from an audio stream received from the computing device, or an audio file having a record of the audio stream captured during the communication session, wherein the reference is embedded in the transcript or the summary at a position relative to the content of the application file, wherein the reference may be inserted into the audio file using metadata that causes the reference to be displayed when the audio file is played using a media player application.
[0127] Figures 6A-6D An example is shown in which, during screen-sharing mode, the server (a) creates a meeting transcript or summary, (b) determines the location of the shared file, and (c) inserts the transcript / summary into the file.
[0128] Clause F: A method performed by a data processing system (100) for inserting content obtained from communication data of a meeting during a meeting into an application file displayed in screen-sharing mode, the method comprising: appending Figure 6A and 6B The first operating mode features audio and messaging communication without screen sharing. In the first operating mode, the system does not capture communication data (voice and text) used for insertion into documents. Figure 6CDuring a communication session, a screen sharing mode for sharing files displayed in an application is activated. This is achieved by rendering shared content (121) shared by the first computing device (11A) on the display screens of multiple computing devices (11A-11H) in the communication session. The screen sharing mode is invoked for the first computing device (11A) to execute the application (122) and thus display the shared content (121) of the application file (110). Figures 6C-6D In the second mode, the system creates a conference transcript or summary for real-time discussion based on audio signals and / or chat messages related to the shared content, wherein the system analyzes the audio stream or messages to identify “selected content” to be added to a file. The method also includes processing audio streams and messages transmitted between multiple computing devices (11A-11H) to extract the content of the communication session from the audio streams and messages (123). The system determines the location of the application file by determining an event in which a content sharing mode has occurred during the communication session. In response to determining an event in which a content sharing mode has occurred during the communication session: the location of the application file (110) is determined. The application file is accessed via the determined location; the application file is modified by inserting the content (123) into the application file (HO), directly inserting the transcript / summary content into the file; and the application file with the inserted content derived from the audio streams and messages of the communication session is displayed.
[0129] Figure 6E Comments made in an audio stream or message can lead to edits of a document, such as based on transcription / summary.
[0130] Clause G adds the proposed changes to the application file: the method described in this document also includes: processing audio streams and messages transmitted between multiple computing devices (11A-11H) to identify editing instructions for shared content; and enabling the application to modify the shared content based on the editing instructions.
[0131] Figure 6E It is a more specific version of claim 2, adding the generation of a summary or transcript as an intermediate step for processing editing instructions, for example, adding the proposed changes to the application file based on the transcript / summary.
[0132] Clause H: The method according to the clauses described herein further includes: processing audio streams and messages transmitted between multiple computing devices (11A-11H) to generate a summary or transcript of a communication session; processing the summary or transcript of the communication session to identify editing instructions for shared content; and enabling an application to modify the shared content based on the editing instructions.
[0133] Figures 8A-8C : Determine the location of the file. Scenario A: The file is stored on the client.
[0134] Clause I: The method according to the clauses described herein, wherein determining the location of the application file includes: sending a query to the application and storage service (222) to retrieve a path indicating the storage location of the application file; determining that the application file is stored on a first computing device (11A) among a plurality of computing devices (11A-11H); in response to determining that the application file is stored on the first computing device: copying or moving the application file from the first computing device to the storage service, and inserting content into the application file stored on the storage service, and configuring a license for the application file to allow participants in the communication session to view and edit shared content within the application file and content within the application file.
[0135] Figures 9A-9B : Determine the location of the file. Scenario B: The file is stored on the server.
[0136] Clause J: The method of the clause described herein, wherein determining the location of an application file includes: sending a query to the application and storage service (222) to retrieve a path indicating the storage location of the application file; determining that the application file is stored on the storage service (222) communicating with the first computing device (11A) performing screen sharing; and in response to determining that the application file is stored on the storage service: inserting content into the application file stored on the storage service, and configuring permission for the application file stored on the storage service to allow participants in the communication session to view and edit the shared content within the application file and the content within the application file.
[0137] Figure 11 An example of how the system uses an AI model to interpret communication data is shown.
[0138] Clause K: The method according to the clauses described herein further includes: generating a query for a large language model that includes activity data, the activity data including messages and transcripts derived from the audio stream of the conference and conference participant data, the query including instructions that cause the large language model to identify a portion of the activity data that is associated with at least one of: the application file, the person making the reservation, the predetermined topic, or instructions for modifying the shared content of the application file; transmitting the query to the large language model such that the large language model generates the content based on the portion of the activity data that is associated with at least one of the application file, the person making the reservation, the predetermined topic, or instructions for modifying the shared content of the application file, wherein the content is inserted into the application file or executed to edit the shared content of the application file.
[0139] Clause L: The method according to the clauses described herein also includes: configuring a license to allow attendees of a communication session to view and edit an application file, wherein the content includes at least one of the following: a summary of the communication session, a transcript derived from an audio stream received from a computing device, or an audio file having a record of an audio stream captured during the communication session, wherein the content is located within the application file adjacent to the relevant portion of the shared content.
[0140] Now go to Figure 13 The diagram illustrates an example environment 600 in which system 602 can implement the disclosed techniques. It should be understood that the above-described subject matter can be implemented as a computer-controlled apparatus, computer process, computing system, or article of manufacture such as a computer-readable storage medium. Operations of exemplary methods are shown in separate boxes and summarized with reference to those boxes. The methods are described as a logical flow of blocks, each of which may represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, an operation represents computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable one or more processors to perform the described operations.
[0141] Typically, computer-executable instructions include routines, programs, objects, modules, components, data structures, etc., that perform specific functions or implement specific abstract data types. The order in which operations are described is not intended to be construed as a limitation, and any number of described operations can be executed in any order, combined in any order, subdivided into multiple sub-operations, and / or executed in parallel to implement the described process. The described process can be executed by resources associated with one or more devices, such as one or more internal or external CPUs or GPUs, and / or one or more hardware logic devices, such as field-programmable gate arrays (“FPGAs”), digital signal processors (“DSPs”), or other types of accelerators.
[0142] All the methods and processes described above can be embodied in software code modules executed by one or more general-purpose computers or processors, and fully automated via these software code modules. The code modules can be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all of the methods can alternatively be embodied in dedicated computer hardware, such as the computer hardware described below.
[0143] Any routine descriptions, elements, or blocks depicted in the flowcharts described herein and / or the accompanying drawings should be understood as potentially representing modules, segments, or portions of code comprising one or more executable instructions for implementing a particular logical function or element in a routine. Alternative implementations are included within the scope of the examples described herein, wherein elements or functions may be omitted or performed not in the order shown or discussed, including substantially synchronously or in reverse order, depending on the functionality involved, as will be understood by those skilled in the art. For illustrative purposes, an "application file" can be a file of any data structure used for word processing, spreadsheets, presentation files, emails, data objects, markup languages, or any format. Additionally, a content sharing mode can be an operational mode of a computer during a communication session (e.g., a conference, broadcast, etc.) in which the computer causes content to be displayed on other computers participating in the communication session. This can include desktop sharing sessions in which a user has an application that displays the content of a file and the content of the display, and the user's desktop is displayed on a remote computer. This also includes other content sharing sessions, including web-based content sharing sessions, application-based content sharing sessions, or any other communication paradigm that allows a user to share the content of a file or other data structure with a remote computer participating in the communication session. Furthermore, references or links can include data of any format, such that when a user selects a reference or link, that selection causes the computer to retrieve data from the address defined in the reference or link.
[0144] In some implementations, system 602 can be used to collect, analyze, and share data displayed to users in communication session 603. As shown, communication session 603 can be implemented among multiple client computing devices 606(1) to 606(N) (where N is a number with a value of 2 or greater) associated with or part of system 602. Client computing devices 606(1) to 606(N) enable users (also referred to as individuals) to participate in communication session 603.
[0145] In this example, communication session 603 is hosted by system 602 via one or more networks 608. That is, system 602 can provide services that enable users of client computing devices 606(1) to 606(N) to participate in communication session 603 (e.g., via live viewing and / or recorded viewing). Therefore, “participants” in communication session 603 can include users and / or client computing devices (e.g., multiple users in a room participating in a communication session via a single client computing device), each user communicating with other participants. Alternatively, communication session 603 can be hosted by one of the client computing devices 606(1) to 606(N) using peer-to-peer technology. System 602 can also host chat conversations and other team collaboration features (e.g., as part of an application suite).
[0146] In some implementations, such chat conversations and other team collaboration features are considered external communication sessions distinct from communication session 603. The computing system 602, which collects participant data from communication session 603, can be linked to such external communication sessions. Therefore, the system can receive information such as date, time, and session details, enabling connectivity to such external communication sessions. In one example, a chat conversation can be conducted based on communication session 603. Additionally, system 602 can host communication session 603, which includes at least multiple participants located at a shared meeting location (such as a conference room or auditorium) or in different locations.
[0147] In the example described herein, client computing devices 606(1) to 606(N) participating in communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of live and / or recorded content. The collection of various instances or streams of live and / or recorded content may be provided by one or more cameras, such as video cameras. For example, a single live or recorded content stream may include media data associated with a video feed provided by a camera (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). In some implementations, the video feed may be communicated along with messages.
[0148] Figure 13 System 602 includes device 610. Device 610 and / or other components of system 602 may include distributed computing resources that communicate with each other and / or with client computing devices 606(1) to 606(N) via one or more networks 608. In some examples, system 602 may be a separate system responsible for managing aspects of one or more communication sessions (such as communication session 603). As an example, system 602 may be managed by entities such as SLACK, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.
[0149] Network 608 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. Network 608 may also include any type of wired and / or wireless network, including but not limited to local area networks (“LANs”), wide area networks (“WANs”), satellite networks, wired networks, Wi-Fi networks, WiMax networks, mobile communication networks (e.g., 4G, etc.), or any combination thereof. Network 608 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as Internet Protocol (“IP”), Transmission Control Protocol (“TCP”), User Datagram Protocol (“UDP”), or other types of protocols. Furthermore, network 608 may include multiple devices that facilitate network communication and / or form the hardware infrastructure of the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone equipment, etc.
[0150] In some examples, network 608 may also include devices that enable connection to a wireless network, such as a wireless access point (“WAP”). Examples support connections to WAPs that transmit and receive data over various electromagnetic frequencies (e.g., radio frequency), including WAPs that support the Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standard (e.g., 802.11g, 802.11n, 802.11ac, etc.) and other standards.
[0151] In various examples, device 610 may include one or more computing devices that operate in a cluster or other grouped configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, device 610 may belong to various categories of devices, such as traditional server-type devices, desktop computer-type devices, and / or mobile-type devices. Therefore, although shown as a single type of device or a server-type device, device 610 may include a wide variety of device types and is not limited to a particular type of device. Device 610 may represent, but is not limited to, a server computer, desktop computer, web server computer, personal computer, mobile computer, laptop computer, tablet computer, or any other kind of computing device.
[0152] Client computing devices (e.g., one of client computing devices 606(1) to 606(N)) (each of which is also referred to herein as a “data processing system”) may belong to various device categories and may be the same as or different from device 610, such as conventional client-type devices, desktop computer-type devices, mobile devices, dedicated devices, embedded devices and / or wearable devices. Thus, client computing devices may include, but are not limited to, desktop computers, game consoles and / or gaming devices, tablet computers, personal data assistants (“PDAs”), mobile phone / tablet hybrid devices, laptop computers, telecommunications equipment, computer navigation client computing devices such as satellite-based navigation systems including Global Positioning System (“GPS”) devices, wearable devices, virtual reality (“VR”) devices, augmented reality (“AR”) devices, implantable computing devices, automotive computers, network-enabled televisions, thin clients, terminals, Internet of Things (“IoT”) devices, workstations, media players, personal video recorders (“PVR”), set-top boxes, cameras, integrated components (e.g., peripherals) for inclusion in computing devices, appliances or any other kind of computing device. In addition, client computing devices may include combinations of previously listed examples of client computing devices, such as desktop computer-type devices or mobile devices combined with wearable devices.
[0153] Client computing devices 606(1) to 606(N) of various categories and device types can represent any type of computing device having one or more data processing units 692 operatively connected to a computer-readable medium 694 via a bus 616. In some instances, the bus 616 may include one or more of a system bus, data bus, address bus, PCI bus, Mini-PCI bus, and any of a variety of local, peripheral, and / or stand-alone buses. Executable instructions stored on the computer-readable medium 694 may include, for example, an operating system 619, a client module 620, a profile module 622, and other modules, programs, or applications that can be loaded and executed by the data processing unit 692.
[0154] Client computing devices 606(1) to 606(N) may also include one or more interfaces 624 to enable communication between client computing devices 606(1) to 606(N) and other networked devices (such as device 610) via network 608. Such network interfaces 624 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data over the network. In addition, client computing devices 606(1) to 606(N) may include input / output (“I / O”) interfaces (devices) 626 that enable communication with input / output devices such as peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices such as microphones, cameras for acquiring and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices (e.g., displays, printers, audio speakers, haptic output devices, etc.). Figure 13 A client computing device 606 (1) is shown connected in some way to a display device (e.g., display screen 629 (N)) which can display a UI according to the techniques described herein.
[0155] exist Figure 13 In the example environment 600, client computing devices 606(1) to 606(N) can use their respective client modules 620 to connect to each other and / or to other external devices in order to participate in communication sessions 603 or to contribute activities to a collaborative environment. For example, a first user can use client computing device 606(1) to communicate with a second user on another client computing device 606(2). When client module 620 is executed, the user can share data, which allows client computing device 606(1) to connect to system 602 and / or other client computing devices 606(2) to 606(N) via network 608.
[0156] Client computing devices 606(1) to 606(N) can use their respective profile modules 622 to generate participant profiles. Figure 13 (Not shown in the diagram), and provides participant profiles to other client computing devices and / or systems 602 via device 610. The participant profile may include one or more of the following: the identity of a user or user group (e.g., name, unique identifier (“ID”), user data (such as personal data), machine data (such as location (e.g., IP address, room in a building), and technical capabilities). The participant profile can be used to register participants in a communication session.
[0157] like Figure 13As shown, device 610 of system 602 includes server module 630 and output module 632. In this example, server module 630 is configured to receive media streams 634(1) to 634(N) from various client computing devices such as client computing devices 606(1) to 606(N). As described above, media streams may include video feeds (e.g., audio and visual data associated with a user), audio data to be output along with a presentation of the user's avatar (e.g., an audio-only experience in which no video data of the user is transmitted), text data (e.g., text messages), file data, and / or screen-sharing data (e.g., documents, slideshows, images, videos displayed on a display screen, etc.). Therefore, server module 630 is configured to receive a set of various media streams 634(1) to 634(N) (this set is referred to herein as "media data 634") during live viewing of communication session 603. In some scenarios, not all client computing devices participating in communication session 603 provide media streams. For example, the client computing device may be a consumption or "listening" device, such that it only receives content associated with communication session 603, but does not provide any content to communication session 603.
[0158] In various examples, server module 630 may select aspects of media stream 634 to be shared with each of the participating client computing devices 606(1) to 606(N). Thus, server module 630 may be configured to generate session data 636 based on stream 634 and / or pass session data 636 to output module 632. Output module 632 may then transmit communication data 639 to client computing devices (e.g., client computing devices 606(1) to 606(3) participating in a live viewing of the communication session). Communication data 639 may include video, audio, and / or other content data provided by output module 632 based on content 650 associated with output module 632 and based on the received session data 636. Content 650 may include stream 634 or other shared data, such as image files, spreadsheet files, slide sets, documents, etc. Stream 634 may include a video component depicting images captured by I / O devices 626 on each client computer. Content 650 also includes input data from each user, which may be used to control the orientation and position of the representation. The content may also include instructions for sharing data and identifying the recipients of the shared data. Therefore, content 650 is also referred to herein as input data 650 or input 650.
[0159] As shown in the figure, the output module 632 transmits communication data 639(1) to the client computing device 606(1), and transmits communication data 639(2) to the client computing device 606(2), and transmits communication data 639(3) to the client computing device 606(3), etc. The communication data 639 transmitted to the client computing device can be the same or can be different (for example, the positioning of the content stream within the user interface can change from one device to the next).
[0160] In various embodiments, device 610 and / or client module 620 may include GUI rendering module 640. GUI rendering module 640 may be configured to analyze communication data 639 delivered to one or more client computing devices in client computing device 606. Specifically, GUI rendering module 640 at device 610 and / or client computing device 606 may analyze communication data 639 to determine an appropriate manner for displaying video, images, and / or content on display screen 629 of the associated client computing device 606. In some embodiments, GUI rendering module 640 may provide video, images, and / or content to a rendering GUI 646 presented on display screen 629 of the associated client computing device 606. The rendering GUI 646 may be presented on display screen 629 by GUI rendering module 640. Rendering GUI 646 may include video, images, and / or content analyzed by GUI rendering module 640.
[0161] In some implementations, the presentation GUI 646 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed from a presenter or individual, and a second section of the presentation GUI 646 may include a video feed from an individual consuming meeting information provided by the presenter or individual. The GUI presentation module 640 may populate the first and second sections of the presentation GUI 646 in a manner that appropriately mimics the environmental experience that the presenter and individual may be sharing.
[0162] In some implementations, the GUI presentation module 640 can zoom in or provide a scaled view of an individual represented by a video feed to highlight the individual's response to the presenter, such as facial features. In some implementations, the presentation GUI 646 can include video feeds of multiple participants associated with a meeting, such as a general communication session. In other implementations, the presentation GUI 646 can be associated with a channel, such as a chat channel, a corporate team channel, etc. Therefore, the presentation GUI 646 can be associated with an external communication session that is different from a general communication session.
[0163] Figure 14A diagram illustrates example components of an example device 700 (also referred to herein as a "computing device") configured to generate data for some user interfaces disclosed herein. Device 700 can generate data that may include one or more portions, which may render or include video, images, virtual objects, and / or content for display on display screen 629. Device 700 may represent one of the devices described herein. Additionally or alternatively, device 700 may represent a client computing device within client computing device 606.
[0164] As shown in the figure, device 700 includes one or more data processing units 702, computer-readable medium 704, and communication interface 706. Components of device 700 are operatively connected, for example, via bus 709, which may include one or more of a system bus, data bus, address bus, PCI bus, Mini-PCI bus, and any kind of local, peripheral, and / or independent bus.
[0165] As used herein, data processing units (such as data processing unit 702 and / or data processing unit 692) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field-programmable gate array (“FPGA”), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. Illustrative types of hardware logic components that may be utilized include, for example but not limited to, application-specific integrated circuits (“ASICs”), application-specific standard products (“ASSPs”), systems-on-a-chip (“SOCs”), complex programmable logic devices (“CPLDs”), etc.
[0166] As used herein, computer-readable media such as computer-readable media 704 and computer-readable media 694 may store instructions executable by a data processing unit. The computer-readable media may also store instructions executable by an external data processing unit (such as an external CPU, external GPU) and / or by an external accelerator (such as an FPGA-type accelerator, DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator are external to the computing device.
[0167] Computer-readable media (which may also be referred to herein as computer-readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of volatile memory, non-volatile memory, and / or other persistent and / or auxiliary computer storage media, removable and non-removable computer storage media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Therefore, computer storage media includes media in tangible and / or physical form that are part of a device or included in devices and / or hardware components external to a device, including but not limited to random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), phase-change memory (“PCM”), read-only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, optical disc read-only memory (“CD-ROM”), digital versatile disc (“DVD”), optical cards or other optical storage media, magnetic tape cassettes, magnetic tape, disk storage, magnetic cards or other magnetic storage devices or media, solid-state storage devices, storage arrays, network-attached storage, storage area networks, hosted computer storage or any other storage storage, storage device, and / or storage media that can be used to store and maintain information accessible by a computing device. Computer storage media may also be referred to herein as computer-readable storage media, non-transitory computer-readable storage media, non-transitory computer-readable medium, computer-readable storage medium, computer-readable storage device, or computer storage medium.
[0168] Compared to computer storage media, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals (such as carrier waves) or other transmission mechanisms. As defined herein, computer storage media does not include communication media. That is, computer storage media does not include communication media consisting solely of modulated data signals, carrier waves, or propagation signals themselves.
[0169] The communication interface 706 may represent, for example, a network interface controller (“NIC”) or other type of transceiver device for sending and receiving communications over a network. Furthermore, the communication interface 706 may include one or more camera and / or audio devices 722 to enable the generation of video feeds and / or still images, etc.
[0170] In the illustrated example, computer-readable medium 704 includes data storage 708. In some examples, data storage 708 includes data storage such as a database, data warehouse, or other type of structured or unstructured data storage. In some examples, data storage 708 includes a corpus and / or relational database having one or more tables, indexes, stored procedures, etc., to enable data access to one or more of the following tables: Hypertext Markup Language (“HTML”) tables, Resource Description Framework (“RDF”) tables, Web Ontology Language (“OWL”) tables, and / or Extensible Markup Language (“XML”) tables.
[0171] Data storage 708 may store data for operations of processes, applications, components, and / or modules stored in computer-readable medium 704 and / or executed by data processing unit 702 and / or accelerator. For example, in some examples, data storage 708 may store a main calendar and secondary calendars, as well as other session data indicating the status and activity level of each user. Session data may include the total number of participants (e.g., users and / or client computing devices) in a communication session, activities occurring in the communication session, a list of invitees to the communication session, and / or other data related to when and how the communication session is conducted or hosted. Data storage 708 may also include licensing data 714. Licensing data may indicate whether meeting participants can access or edit files linked in a transcript or summary. Furthermore, licensing may indicate whether meeting participants can access or write to files containing transcripts or summaries of the meeting.
[0172] Alternatively, some or all of the aforementioned data may be stored on a separate memory 716 on one or more data processing units 702, such as the memory on a CPU-type processor, GPU-type processor, FPGA-type accelerator, DSP-type accelerator, and / or another accelerator. In this example, the computer-readable medium 704 also includes an operating system 718 and application programming interfaces (APIs) 710(s) configured to expose the functionality and data of device 700 to other devices. Additionally, the computer-readable medium 704 includes one or more modules, such as a user module 730, an output module 732, and a GUI rendering module 740, although the number of modules shown is merely an example and can vary. That is, the functionality associated with the illustrated modules described herein may be performed by a fewer or more modules on one device, or distributed across multiple devices.
[0173] In conclusion, although various configurations have been described in a language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended representation is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claimed subject matter.
Claims
1. A method, executed by a data processing system (100), for inserting content obtained from communication data of a meeting during a meeting into an application file displayed in screen-sharing mode, the method comprising: During a communication session, by causing the rendering of shared content (121) shared by the first computing device (11A) to be displayed on the display screens of the multiple computing devices (11A-11H) in the communication session, the screen sharing mode is invoked for the first computing device (11A) to execute the application (122) thereby displaying the shared content (121) of the application file (110). The audio streams and messages transmitted between the plurality of computing devices (11A-11H) are processed to extract the content of the communication session from the audio streams and the messages (123). Determine an event that has occurred during the communication session that represents a content-sharing mode. In response to an event that determines that the content sharing mode has occurred during the communication session: Determine the location of the application file (110). Access the application file via the determined location; The application file is modified by inserting the content (123) into the application file (110); as well as This causes the application file to display inserted content derived from the audio stream and messages of the communication session.
2. The method according to claim 1, further comprising: The audio stream and the messages transmitted between the plurality of computing devices (11A-11H) are processed to identify editing instructions for the shared content; as well as This allows the application to modify the shared content based on the editing instructions.
3. The method according to claim 1, further comprising: The audio streams and messages transmitted between the plurality of computing devices (11A-11H) are processed to generate a summary or transcript of the communication session; Processing the summary or transcription of the communication session to identify editing instructions for the shared content; and This allows the application to modify the shared content based on the editing instructions.
4. The method according to claim 1, wherein, Determining the location of the application file includes: Send a query to the application and storage service (222) to retrieve the path indicating the storage location of the application file; It is determined that the application file is stored on the first computing device (11A) among the plurality of computing devices (11A-11H); In response to determining that the application file is stored on the first computing device: Copy or move the application file from the first computing device to the storage service, and insert the content into the application file stored on the storage service. Configuring permissions for the application file allows attendees of the communication session to view and edit the shared content within the application file and the content within the application file.
5. The method according to claim 1, wherein, Determining the location of the application file includes: Send a query to the application and storage service (222) to retrieve the path indicating the storage location of the application file; It is determined that the application file is stored on a storage service (222) communicating with the first computing device (11A) performing the screen sharing; and In response to determining that the application file is stored on the storage service: Insert the content into the application file stored on the storage service, and Configuring permissions for the application files stored on the storage service allows attendees of the communication session to view and edit the shared content within the application files and the content within the application files.
6. The method according to claim 1, further comprising: Generate a query for a large language model that includes activity data, including messages and transcripts derived from the audio stream of the conference and conference list data. The query includes instructions that cause the large language model to identify portions of the activity data that are associated with at least one of the following: the application file, the person who made the reservation, the topic of the reservation, or instructions for modifying the shared content of the application file. The query is transmitted to the large language model, which generates the content based on at least one of the activity data, the application file, the pre-registered person, the pre-registered topic, or instructions for modifying the shared content of the application file, wherein the content is inserted into the application file or executed to edit the shared content of the application file.
7. The method according to claim 1, further comprising: The configuration allows attendees of the communication session to view and edit the application file, wherein the content includes at least one of the following: a summary of the communication session, a transcription derived from an audio stream received from the computing device, or an audio file having a record of the audio stream captured during the communication session, wherein the content is located within the application file adjacent to the relevant portion of the shared content.
8. A system (700) for inserting content obtained from communication data of a meeting into an application file displayed in screen-sharing mode during a meeting, the system comprising: One or more processing units (702); as well as A computer-readable storage medium (704) having computer-executable instructions encoded thereon to cause the one or more processing units (702): During a communication session, a screen sharing mode is invoked for a first computing device (11A) to execute an application (122) thereby displaying the shared content (121) of the application file (110), wherein the system causes the rendering of the shared content (121) shared by the first computing device (11A) to be displayed on the display screens of the plurality of computing devices (11A-11H) in the communication session; The audio streams and messages transmitted between the plurality of computing devices (11A-11H) are processed to extract content (123) from the audio streams and messages. In response to invoking the content sharing mode: Determine the location of the application file (110). Access the application file via the determined location; The application file is modified by inserting the content (123) into the application file (110); as well as This causes the application file to display inserted content derived from the audio stream and messages of the communication session.
9. The system according to claim 8, wherein, The instruction also causes the one or more processing units to: Processing the audio streams and messages transmitted between the plurality of computing devices (11A-11H) to identify editing instructions for the shared content; and This allows the application to modify the shared content based on the editing instructions.
10. The system according to claim 8, wherein, The instruction also causes the one or more processing units to: The audio streams and messages transmitted between the plurality of computing devices (11A-11H) are processed to generate a summary or transcript of the communication session; Processing the summary or transcription of the communication session to identify editing instructions for the shared content; and This allows the application to modify the shared content based on the editing instructions.
11. The system according to claim 8, wherein, Determining the location of the application file includes: Send a query to the application and storage service (222) to retrieve the path indicating the storage location of the application file; It is determined that the application file is stored on the first computing device (11A) among the plurality of computing devices (11A-11H); In response to determining that the application file is stored on the first computing device: The application file is copied or moved from the first computing device to the storage service, and the content is inserted into the application file stored on the storage service. Permissions for the application file are configured to allow attendees of the communication session to view and edit the shared content within the application file and the content within the application file.
12. The system according to claim 8, wherein, Determining the location of the application file includes: Send a query to the application and storage service (222) to retrieve the path indicating the storage location of the application file; It is determined that the application file is stored on a storage service (222) communicating with the first computing device (11A) performing the screen sharing; and In response to determining that the application file is stored on the storage service: Insert the content into the application file stored on the storage service, and Configure permissions for the application files stored on the storage service to allow attendees of the communication session to view and edit the shared content within the application files and the content within the application files.
13. The system according to claim 8, wherein, The instruction also causes the one or more processing units to: Generate a query for a large language model that includes activity data, including messages and transcripts derived from the audio stream of the conference and conference list data. The query includes instructions that cause the large language model to identify portions of the activity data that are associated with at least one of the following: the application file, the person who made the reservation, the topic of the reservation, or instructions for modifying the shared content of the application file. The query is transmitted to the large language model, which generates the content based on at least one of the activity data, the application file, the pre-registered person, the pre-registered topic, or instructions for modifying the shared content of the application file, wherein the content is inserted into the application file or executed to edit the shared content of the application file.
14. The system according to claim 8, wherein, The instruction also causes the one or more processing units to: Configure permissions to allow attendees of the communication session to view and edit the application file, wherein the content includes at least one of the following: a summary of the communication session, a transcription derived from an audio stream received from the computing device, or an audio file having a record of the audio stream captured during the communication session, wherein the content is located within the application file adjacent to the relevant portion of the shared content.
15. A computer-readable storage medium (704) having computer-executable instructions encoded thereon, the computer-executable instructions causing a data processing system (700) to insert content obtained from communication data of a conference during a conference into an application file displayed in screen-sharing mode, the computer-executable instructions causing one or more processing units (702) of the data processing system (700): During the communication session, a screen sharing mode is invoked for the first computing device (11A) to execute an application (122) thereby displaying the shared content (121) of the application file (110), wherein, The system enables the display of a rendering of the shared content (121) shared by the first computing device (11A) on the display screens of multiple computing devices (11A-11H) in the communication session; The audio streams and messages transmitted between the plurality of computing devices (11A-11H) are processed to extract content (123) from the audio streams and messages. In response to invoking the content sharing mode: Determine the location of the application file (110). Access the application file via the determined location; The application file is modified by inserting the content (123) into the application file (110); as well as This causes the application file to display inserted content derived from the audio stream and messages of the communication session.
16. The computer-readable storage medium according to claim 15, wherein, The instruction also causes the one or more processing units to: Processing the audio streams and messages transmitted between the plurality of computing devices (11A-11H) to identify editing instructions for the shared content; and This allows the application to modify the shared content based on the editing instructions.
17. The computer-readable storage medium according to claim 15, wherein, The instruction also causes the one or more processing units to: The audio streams and messages transmitted between the plurality of computing devices (11A-11H) are processed to generate a summary or transcript of the communication session; Processing the summary or transcription of the communication session to identify editing instructions for the shared content; and This allows the application to modify the shared content based on the editing instructions.
18. The computer-readable storage medium according to claim 15, wherein, Determining the location of the application file includes: Send a query to the application and storage service (222) to retrieve the path indicating the storage location of the application file; It is determined that the application file is stored on the first computing device (11A) among the plurality of computing devices (11A-11H). In response to determining that the application file is stored on the first computing device: The application file is copied or moved from the first computing device to the storage service, and the content is inserted into the application file stored on the storage service. Permissions for the application file are configured to allow attendees of the communication session to view and edit the shared content within the application file and the content within the application file.
19. The computer-readable storage medium according to claim 15, wherein, Determining the location of the application file includes: Send a query to the application and storage service (222) to retrieve the path indicating the storage location of the application file; It is determined that the application file is stored on a storage service (222) communicating with the first computing device (11A) performing the screen sharing; and In response to determining that the application file is stored on the storage service: Insert the content into the application file stored on the storage service, and Configure permissions for the application files stored on the storage service to allow attendees of the communication session to view and edit the shared content within the application files and the content within the application files.
20. The computer-readable storage medium according to claim 15, wherein, The instruction also causes the one or more processing units to: Generate a query for a large language model that includes activity data, including messages and transcripts derived from the audio stream of the conference and conference list data. The query includes instructions that cause the large language model to identify portions of the activity data that are associated with at least one of the following: the application file, the person who made the reservation, the topic of the reservation, or instructions for modifying the shared content of the application file. The query is transmitted to the large language model, which generates the content based on at least one of the activity data, the application file, the pre-registered person, the pre-registered topic, or instructions for modifying the shared content of the application file, wherein the content is inserted into the application file or executed to edit the shared content of the application file.
