Computer implementation methods, computer systems, and computer programs (contextual real-time content highlights on a shared screen)

Abstract

To provide a way to highlight or otherwise indicate in real time which portion of content displayed during a presentation relates to the current discussion such that participants may quickly ascertain the relevant portion of the displayed content.SOLUTION: A method, a computer system and a computer program product for contextual digital content highlighting are provided. Discussion is monitored among a plurality of parties in conjunction with a digital presentation, and a context of the monitored discussion is then identified. Then, a most relevant portion of displayed digital content associated with the presentation is identified based on the identified context, and then highlighting is applied to the identified most relevant portion of the displayed content.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention generally relates to the field of computing, and more specifically to dynamic and intelligent interface operations.

Background Art

[0002] Electronic meetings and presentations are now being used more and more frequently. During these meetings, a common scenario occurs when a particular theme is discussed among two or more parties regarding the content displayed on the screen. In these situations, by identifying the context from the discussions related to the production content, it can be ensured that the focus of the presentation or discussion returns to the relevant content of the presentation.

Summary of the Invention

Problems to be Solved by the Invention

[0003] It may be advantageous to provide a way to highlight or otherwise indicate in real time which portions of the content displayed during a presentation are relevant to the current discussion so that participants can quickly grasp the relevant portions of the displayed content.

Means for Solving the Problems

[0004] According to one exemplary embodiment, a method for contextual digital content highlighting is provided. Discussions are monitored among multiple parties in conjunction with a digital presentation, and the context of the monitored discussions is then identified. Thereafter, the most relevant portions of the displayed digital content associated with the presentation are identified based on the identified context, and then a highlight is applied to the identified most relevant portions of the displayed content. A computer system and a computer program product corresponding to the above method are also disclosed herein.

Brief Description of the Drawings

[0005] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of its exemplary embodiments, which will be read in conjunction with the accompanying drawings. Various features of the drawings are not to scale, as they are provided for clarity to facilitate understanding of the invention by those skilled in the art in conjunction with the detailed description.

[0006] [Figure 1] This figure shows a networked computer environment according to at least one embodiment.

[0007] [Figure 2] This is an operational flowchart illustrating the process for contextual highlighting according to at least one embodiment.

[0008] [Figure 3A] This figure shows an exemplary contextual highlight applied to a presentation slide, according to at least one embodiment. [Figure 3B] This figure shows an exemplary contextual highlight applied to a presentation slide, according to at least one embodiment. [Figure 3C] This figure shows an exemplary contextual highlight applied to a presentation slide, according to at least one embodiment.

[0009] [Figure 4] This is a block diagram of the internal and external components of the computer and server shown in Figure 1, according to at least one embodiment.

[0010] [Figure 5] This is a block diagram of an exemplary cloud computing environment, including the computer system shown in Figure 1, according to one embodiment of the present disclosure.

[0011] [Figure 6]Figure 5 is a block diagram of the functional layer of an exemplary cloud computing environment according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0012] Detailed embodiments of the claimed structure and method are disclosed herein, but it should be understood that the disclosed embodiments are merely illustrative of the claimed structure and method, which may be embodied in various forms. However, the present invention may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided to make this disclosure thorough and complete and to fully convey the scope of the invention to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0013] As mentioned earlier, electronic meetings and presentations are now used more and more frequently. During these meetings, a common scenario arises when a specific topic is discussed between two or more parties related to the content displayed on the screen. In these situations, contextual identification from the discussion related to the presented content can ensure that the focus of the presentation or discussion returns to the relevant content of the presentation.

[0014] Therefore, it may be advantageous to provide a way to highlight, or otherwise indicate in real time, which parts of the content displayed during a presentation are relevant to the current discussion, so that participants can quickly grasp the relevant parts of the displayed content. In some embodiments, it may be even more advantageous to allow the meeting or presentation to proceed by validating the validity of the displayed information relevant to the current discussion in real time and communicating its validity to participants in order to resolve tangents more quickly.

[0015] The exemplary embodiments described below provide a system, method, and program product for real-time contextual cursor highlighting of displayed content. Thus, these embodiments have the ability to improve the technical field of user interfaces by intelligently highlighting relevant content in real time based on an ongoing dialogue. More specifically, the current topic of discussion between two or more parties is determined and compared with the currently displayed content. Displayed content with the highest similarity to the current discussion topic is then identified. The identified content within the displayed content is then highlighted. According to some embodiments, the validity of the identified content may also be determined, and the resulting validity may be indicated via an on-screen identifier.

[0016] This term is used in the context of a virtual meeting in relation to a specific embodiment. As used herein, a virtual meeting refers to an online meeting of two or more people (i.e., participants or attendees) conducted by digital means, as an alternative to a traditional physical meeting where participants meet face-to-face in the same room. A virtual meeting may include the digital transmission of data over a network between computers or other electronic devices, including audio, video, text, images, etc., so that participants can experience and interact with the presentation or meeting in the same way as in a traditional physical meeting. The term virtual meeting may be used synonymously with online meeting, electronic meeting, or video conference.

[0017] As used herein, presentation software refers to software, applications, or other executable code that causes a presentation to be displayed on the screen of an electronic device such as a computer. For example, a slide deck used in a presentation may be displayed on the presenter's screen using presentation software. For example, presentation software may include PowerPoint® (PowerPoint and all PowerPoint-based trademarks and logos are trademarks or registered trademarks of Microsoft Corporation or its affiliates, or a combination thereof).

[0018] As used herein, conferencing software refers to software, applications, or other executable code that enables an electronic device, such as a computer, to conduct a virtual conference by sharing the screens of the remaining conference attendees and conference participants (e.g., speakers), and to distribute audio, video, or other data, thereby facilitating the participation of attendees in a conference or presentation. An example of conferencing software may include Webex® (Webex and all Webex-based trademarks and logos are trademarks or registered trademarks of Cisco Systems, Inc. or its affiliates, or a combination thereof).

[0019] Referring to Figure 1, an exemplary networked computer environment 100 according to one embodiment is shown. The networked computer environment 100 may include a computer 102 having a processor 104 and a data storage device 106 capable of running a software program 108 and a contextual highlighting program 110a. The networked computer environment 100 may also include a server 112 capable of running a database 114 and a contextual highlighting program 110b that can interact with a communication network 116. The networked computer environment 100 may include multiple computers 102 and servers 112, of which only one is shown. The communication network 116 may include various types of communication networks, such as a wide area network (WAN), local area network (LAN), telecommunications network, wireless network, public switching network, or satellite network, or a combination thereof. It should be understood that Figure 1 provides only an example of one implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications to the shown environment may be made based on design and implementation requirements.

[0020] The client computer 102 can communicate with the server computer 112 via the communication network 116. The communication network 116 may include connections such as wired, wireless communication links, or optical fiber cables. As will be described with reference to FIG. 4, the server computer 112 may include internal components 902a and external components 904a respectively, and the client computer 102 may include internal components 902b and external components 904b respectively. The server computer 112 can also operate in a cloud computing service model such as software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS). The server 112 can also be deployed in a cloud computing deployment model such as a private cloud, community cloud, public cloud, or hybrid cloud. The client computer 102 can be, for example, a mobile device, phone, personal digital assistant, netbook, notebook computer, tablet computer, desktop computer, or any type of computing device capable of executing programs, accessing the network, and accessing the database 114. According to various implementations of this embodiment, the contextual highlight programs 110a, 110b can interact with a database 114 that can be incorporated into various storage devices such as, but not limited to, the computer / mobile device 102, the networked server 112, or a cloud storage service.

[0021] According to this embodiment, a user using the client computer 102 or the server computer 112 may use the contextual highlight programs 110a, 110b (respectively) to identify and highlight in real time relevant content displayed on a shared screen based on the theme of the discussion. The contextual highlight method will be described in more detail below in relation to FIGS. 2, and 3A - 3C.

[0022] Referring now to FIG. 2, there is shown an operational flowchart depicting an exemplary contextual highlighting process 200 used by contextual highlight programs 110a and 110b, according to at least one embodiment.

[0023] At 202, a presentation or meeting begins. The presentation, virtual meeting, etc. may include content presented to participants or audience members. The presented content may be presented on a device screen connected to computer 102, projected onto a screen, displayed on an augmented reality device, etc. The presented content may include text, tables, charts, images, video clips, audio clips, etc. related to the presentation or discussion (e.g., a work meeting). In other words, the subject matter of the presented content is related to the theme of the discussion and may be used in conjunction with the discussion.

[0024] Presenters, meeting leaders, etc., may utilize presentation software to display content to presentation participants (e.g., software program 108). For example, a set of ordered presentation slides may be displayed by presentation software running on computer 102, which prepares, loads, and then outputs one slide at a time to a screen that can be shared with participants. The shared screen may include, for example, a projection screen for an audience physically together in one location, or each participant may receive the content of the current slide via a communication network 116 on their individual computer 102, which displays the slide content on their individual screen in the virtual meeting, or a combination of these. Furthermore, microphones, cameras, and other relevant sensors may be used to monitor discussions occurring simultaneously during the presentation or meeting. The contextual highlight process 200 may determine that the meeting has started as soon as a trigger indicating that the presentation has begun is identified. For example, the trigger may include when the first slide is displayed to participants, or by using natural language processing (NLP) to identify a predetermined phrase (e.g., "Let's get started") spoken at the start of the meeting and captured via microphone.

[0025] Next, in 204, the discussion during the presentation is analyzed. As previously mentioned, microphones and other sensors may be used to monitor the participants' voices. In some embodiments, the voiced voice may not be used by some or all participants; instead, text messages or other nonverbal communication may be used by participants during the presentation to interact with the speaker or presenter. Thus, the contextual highlighting process 200 may monitor a combination of verbal and nonverbal communication during the discussion. Verbal communication may be monitored, for example, using a microphone connected to computer 102. Voice data may be acquired from the microphone by conferencing software for transmission to participants. In embodiments, the contextual highlighting process 200 may interface with conferencing software (e.g., via an application programming interface (API)) to acquire voice data and process the voice data using NLP techniques. In examples where text-based discussions may occur, the contextual highlighting process 200 may interface with conferencing software or a messaging application (e.g., via an API) to acquire questions, comments, or other communications from other participants or presenters for analysis.

[0026] Subsequently, in step 206, the contextual highlighting process 200 determines whether two or more parties are involved in the discussion. Parties involved in the discussion may include one or more presenters or participants. In a conventional physical meeting, determining whether two or more parties are involved in the discussion may involve analyzing audio data captured via microphones. This analysis of audio data may use NLP techniques to identify words and phrases indicating a conversation between the two parties. In some embodiments, audio characteristics may be identified and then used by a machine learning model to determine whether two or more different people are speaking to each other. In a virtual meeting setting, since the meeting software identifies the parties communicating based on the signed-in user and the device transmitting the signed-in user's communications, the contextual highlighting process 200 may interface with the meeting software to identify each person who is speaking or communicating nonverbally.

[0027] For example, in a virtual meeting discussing financial market trends, the speaker discusses the Nifty Index. A slide suggesting that the Nifty Index will rise over the next 10 days is displayed on the shared screen. While the speaker is giving their opinion in relation to the Nifty Index, participant P verbally asks a question about a recent investment in company R from social media company X and the impact on company R's future market capitalization. The contextual highlighting process 200 connects with the meeting software via an interface to determine that one party is the speaker by detecting voice from the speaker's microphone, and also determines that participant P is a party to the discussion by detecting voice from participant P's microphone.

[0028] If the contextual highlighting process 200 determines in 206 that two or more parties are not involved in the discussion, the contextual highlighting process 200 returns to 204 to analyze the discussion.

[0029] However, if the contextual highlighting process 200 determines in 206 that two or more parties are involved in the discussion, the context of the discussion is identified in 208. In embodiments, semantic analysis may be performed on the discussion data (e.g., audio data from a microphone) to grasp the meaning conveyed in the current discussion. As mentioned above, in some examples, the context of the discussion may be established by semantically analyzing a combination of audio and nonverbal data to determine the subject or theme of the discussion. In some embodiments, machine learning algorithms may be used to classify the discussion data based on the theme. Other semantic analysis techniques may identify emotions, intentions, etc. Furthermore, semantic analysis may include keyword extraction or entity extraction. After semantic analysis is performed, additional data on the identified themes and keywords may be collected by searching knowledge bases, the internet, etc.

[0030] Continuing from the above example, audio data from speaker and participant P is analyzed using a supervised machine learning algorithm, and the contextual highlighting process 200 receives keywords and phrases such as "Company R," "Investment by Company X," and "Future Market Capitalization" that the supervised machine learning algorithm extracts. After the keywords and phrases are extracted, the contextual highlighting process 200 searches for information related to the keywords via the internet and determines that Company R is part of the Nifty Index. Furthermore, the contextual highlighting process 200 determines that Company X is a profitable company, and that investment by a profitable company typically increases the market capitalization of the investee company. In addition, the contextual highlighting process 200 determines that if Company R's market capitalization increases, the Nifty Index will also be positively affected because Company R is a component of Nifty.

[0031] Next, in step 210, the most relevant displayed content is determined. In some embodiments, the contextual highlighting process 200 may interface with presentation software via an API to retrieve the displayed content. The displayed content may include text, images, etc. The displayed content may also be content contained in the currently displayed slide from the presentation. In some embodiments, the displayed content may include previously displayed content. For example, a speaker may not give participants an opportunity to ask questions about the content until the speaker moves to the next slide. Therefore, some embodiments may retrieve previously displayed content. In some embodiments, previously displayed content may be limited by a time threshold since the slide was displayed (e.g., 2 minutes), a threshold number of previously displayed slides (e.g., 3 slides ago), or a percentage of the previously displayed presentation (e.g., 10% of 100 slides, and therefore 10 slides ago).

[0032] To determine the most relevant content from the displayed content, semantic analysis may be performed on the displayed content, similar to the semantic analysis discussed in relation to 208 above. As a result, a machine learning model may be used to analyze the displayed content to identify keywords and phrases and other semantic analysis cues. Since the displayed content may include images, tables, charts, videos, and other visual data, semantic analysis may be accompanied by prior image analysis to generate textual representations of the visual data contained within the displayed content. The textual representations of the visual data may then be grouped with textual data from the displayed content and input into a machine learning model to identify keywords or phrases. The textual data representing the displayed content may be broken down or tokenized into parts such as sentence by sentence, phrase by phrase within a sentence, word by word, list item by list, or individual image by image.

[0033] Once the displayed content is semantically analyzed, a similarity algorithm may be used to compare the context of the discussion identified in 208 with the meaning of the displayed content to find the portion of the displayed content that most closely matches the discussion context. The semantic similarity algorithm may output a score (for example, normalized to 0-1, where 1 is the closest similarity) indicating how similar each portion of the displayed content is to the discussion context. According to at least one embodiment, once a similarity score has been calculated for each portion of the displayed content, the portions may be sorted and ranked by their similarity scores. The portion with the highest similarity score may then be selected as the most relevant portion of the displayed content.

[0034] Continuing the above example, as shown in Figure 3A, the content displayed on presentation slide 300 includes a graph 302 showing historical Nifty index data accompanied by text 304 stating, "Looking at the trend line, the Nifty index will rise 10% in the next 10 sessions. Keep an eye on Nifty stock for the next 10 days. This is a great investment opportunity." The contextual highlighting process 200 may retrieve the displayed content (i.e., graph 302 and text 304) from the presentation software. Thereafter, the contextual highlighting process 200 may identify the visual data (i.e., graph 302) and perform image analysis to generate a textual representation of graph 302. Text 304 may be broken down into parts based on phrases. The textual representation of graph 302, along with the phrases from text 304, is then semantically analyzed and compared to the previously identified argumentative context. After calculating a similarity score for each part of the displayed content, they are ranked from highest to lowest score. The portion of the displayed content with the highest similarity score is the phrase, "The Nifty index will increase by 10% in the next 10 sessions." Therefore, the phrase, "The Nifty index will increase by 10% in the next 10 sessions," is determined to be the most relevant portion of the displayed content.

[0035] Subsequently, in 212, the most relevant content within the displayed content is highlighted on the screen. According to at least one embodiment, the portion of the displayed content identified above in 210 is highlighted by communicating with presentation software (e.g., via an appropriate API call), and the presentation may thereafter respond to the communication from the contextual highlighting process 200 by highlighting the portion of the displayed content. The highlighting may include any combination of visual modifications or other modifications to the displayed content in order to distinguish the portion of the displayed content. For example, the highlighting may include applying a colored background to the text, changing the font style (such as from Times New Roman to Calibri), changing the font size, changing the font color, making it bold, italicizing, underlining, a static or animated arrow pointing to the relevant content, etc.

[0036] In an embodiment where a previous slide is searched for relevant content as described above in 210, highlighting the relevant content may include communicating with the presentation software to return to the slide deck, display the correct previous slide containing the relevant content, and apply a highlight to the relevant content. Thus, during the discussion, the relevant content is both displayed on the screen and highlighted in real time, even if the relevant content is on a different slide than the one initially displayed when the discussion began.

[0037] Continuing from the above example, as shown in Figure 3B, the phrase "The Nifty index will increase by 10% in the next 10 sessions." is determined to be the most relevant portion 306 of the displayed content. Subsequently, the contextual highlighting process 200 requests, via an API call, that the presentation software highlight the relevant portion 306, the phrase "The Nifty index will increase by 10% in the next 10 sessions.". As shown in Figure 3B, according to this example, the identified relevant portion 306 is highlighted by applying bold and underline to the text of the relevant portion 306.

[0038] In 214, the contextual highlighting process 200 determines whether the discussion on a topic is continuing. In embodiments, a discussion between two or more parties in a meeting or presentation may continue to be analyzed after highlighting has been applied to the relevant portion 306 of the displayed content as described above in 212. The contextual highlighting process 200 may determine whether the discussion between two or more parties is continuing by monitoring and analyzing the discussion as described above in 206. In embodiments, the topic of the discussion may be identified in the manner described above in 208, and the topic may be compared to the topic determined above in 208 to determine whether the same topic is continuing to be discussed. Additionally or alternatively, the current discussion topic may be compared to the meaning of the relevant portion 306 to determine whether the discussion is still closely related to the highlighted relevant portion 306.

[0039] In some embodiments, the contextual highlighting process 200 may also determine whether the discussion contains indications that one or more parties are expressing doubts or questions about the validity of the relevant portion 306. Various NLP techniques used in supervised machine learning models may be used to determine whether doubts or questions are raised in relation to the relevant portion 306, for example, by identifying predetermined words or phrases, analyzing the tone of voice, or monitoring participants' facial expressions or body language via a camera.

[0040] Continuing with the above example, and after highlighting the relevant section 306, another participant P2 states, "The value of company R will not change enough to cause the Nifty index to rise by 10% in 10 days." Based on the analysis of the discussion, the topic of the discussion remains related to the rise of the Nifty index over the next 10 days. Furthermore, participant P2 expresses doubts about the opinions raised in the discussion and the corresponding content highlighted in the relevant section 306, identified by the NLP analysis of what P2 said.

[0041] If the contextual highlighting process 200 determines that the discussion has moved to a different topic, or, according to some embodiments, if the discussion on the validity of a topic is not identified in 214, the contextual highlighting process 200 returns to 204 to analyze the discussion.

[0042] However, if the contextual highlighting process 200 determines that the discussion continues on the same topic, or, according to some embodiments, if the contextual highlighting process 200 also determines that a disagreement regarding the topic's validity has been identified in 214, the validity of the relevant content 306 is verified in 216. According to at least one embodiment, the contextual highlighting process 200 may use a separate service or program to verify the validity of the content. For example, in some implementations, the contextual highlighting process 200 may transmit the content to be verified (e.g., relevant content 306) to a validity verification service (e.g., via a communication network 116). The validity verification service may search the Internet, other knowledge bases, or some other information sources to determine the validity of the relevant content 306 transmitted by the contextual highlighting process 200. In response to a verification request from the contextual highlighting process 200, the validity verification service may return a validity score indicating the determined validity of the relevant content 306. Thereafter, the contextual highlighting process 200 may determine whether the relevant content 306 is valid by comparing the validity score to a predetermined threshold. In another embodiment, a validity service may make a validity determination and then send a response to the contextual highlighting process 200 indicating whether the relevant content 306 is valid.

[0043] Continuing the above example, the contextual highlighting process 200 sends the relevant content 306, a sentence stating, "The Nifty index will increase by 10% in the next 10 sessions," to the validation service. The validation service, based on current trends, determines that there is sufficient confidence that the relevant content 306 is valid and returns the result "valid" to the contextual highlighting process 200.

[0044] Subsequently, in step 218, a validity indicator is added adjacent to the relevant content 306. In an embodiment, the contextual highlighting process 200 may communicate with presentation software (e.g., via an API call) to request that a validity indicator be added to the displayed content (e.g., a presentation slide 300). The contextual highlighting process 200 may provide an appropriate indicator that the presentation software then places on the slide. In another embodiment, the presentation software may place its own indicator in response to the request of the contextual highlighting process 200, along with the validity result (i.e., valid or not). The validity indicator may take various forms of static, animated, audio, or other displays. For example, the indicator may be a thumbs-up (valid) or thumbs-down (not valid), an "X" (not valid), or a checkmark (valid), etc. In another embodiment, the highlighted relevant content 306 may be further modified to indicate validity, either alternatively or additionally to the aforementioned indicator. For example, when the relevant content is verified as valid, the text of the relevant content 306 may change color from black to green. From that point onward, the contextual highlighting process 200 may return to step 204 and continue the analysis of the argument.

[0045] Continuing the above example, referring to Figure 3C, the relevant content 306 was verified as valid. Subsequently, the contextual highlighting process 200 communicates with the presentation software to request that a validity indicator be added to the presentation slide 300 to show that the relevant content 306 is valid. The presentation software then adds a thumbs-up validity indicator 308 adjacent to the relevant content 306 to indicate that the relevant content 306 is valid.

[0046] Figures 2 and 3A–3C provide only illustrative examples of one embodiment and should be understood as not to imply any limitation on how different embodiments may be implemented. Many modifications to the embodiments shown may be made on a case-by-case basis for design and implementation requirements. For example, according to an alternative embodiment, the contextual highlighting process 200 may be integrated into presentation software or conferencing software so that communication is incorporated into the software. In some embodiments, this integration may be achieved by implementing the contextual highlighting process 200 as a plug-in to the presentation software or conferencing software. In some embodiments, the presentation software and conferencing software may be combined into a single application that implements the contextual highlighting process 200.

[0047] As described in the embodiments above, contextual highlighting programs 110a and 110b can improve the capabilities of computers or other technologies by providing real-time content highlighting of content displayed on a shared screen relating to the current discussion, enabling participants to quickly grasp the relevant portions of the displayed content while considering the current discussion. Furthermore, the effectiveness of the content is determined and shown in real time during the discussion, informing participants of the effectiveness of the highlighted content and enabling the presentation or meeting to proceed without undue delays by quickly resolving tangents in the discussion.

[0048] Figure 4 is a block diagram of the internal and external components of the computer shown in Figure 1, according to an exemplary embodiment of the present invention. It should be understood that Figure 4 provides only an example of one implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications to the shown environment may be made based on design and implementation requirements.

[0049] Data processing systems 902 and 904 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 902 and 904 may represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, or configurations, or combinations thereof, that may be represented by data processing systems 902 and 904 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the systems or devices described above.

[0050] The user client computer 102 and the network server 112 may include sets of internal components 902a, 902b and external components 904a, 904b, as shown in Figure 4. Each of the multiple sets of internal components 902a, 902b includes one or more processors 906, one or more computer-readable RAMs 908, and one or more computer-readable ROMs 910 on one or more buses 912, one or more operating systems 914, and one or more computer-readable tangible storage devices 916. One or more operating systems 914, software programs 108, and contextual highlighting programs 110a in the client computer 102, and contextual highlighting programs 110b in the network server 112 may be stored in one or more computer-readable tangible storage devices 916 for execution by one or more processors 906 via one or more RAMs 908 (typically including cache memory). In the embodiment shown in Figure 4, each of the computer-readable tangible storage devices 916 is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 916 is a semiconductor storage device such as a ROM 910, EPROM, flash memory, or other computer-readable tangible storage device capable of storing computer programs and digital information.

[0051] Each set of internal components 902a and 902b also includes an R / W drive or interface 918 for reading from or writing to one or more portable computer-readable tangible storage devices 920, such as CD-ROMs, DVDs, memory sticks, magnetic tapes, magnetic disks, optical disks, or semiconductor storage devices. Software programs such as software program 108 and contextual highlight programs 110a and 110b are stored in one or more of the respective portable computer-readable tangible storage devices 920, can be read via their respective R / W drives or interfaces 918, and loaded into their respective hard drives 916.

[0052] Each set of internal components 902a and 902b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi® interface card, or a 3G or 4G wireless interface card, or other wired or wireless link. The software program 108 and contextual highlighting program 110a in the client computer 102 and the contextual highlighting program 110b in the network server computer 112 can be downloaded from an external computer (e.g., a server) via the network (e.g., the Internet, a local area network, or another wide area network) and their respective network adapters or interfaces 922. The software program 108 and contextual highlighting program 110a in the client computer 102 and the contextual highlighting program 110b in the network server computer 112 are loaded from the network adapter (or switch port adapter) or interface 922 into their respective hard drives 916. The network may comprise copper, optical fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof.

[0053] Each of the multiple sets of external components 904a and 904b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. External components 904a and 904b may also include a touchscreen, a virtual keyboard, a touchpad, a pointing device, and other human interface devices. Each of the sets of internal components 902a and 902b also includes a device driver 930 that interfaces with the computer display monitor 924, the keyboard 926, and the computer mouse 928. The device driver 930, the R / W drive or interface 918, and the network adapter or interface 922 comprise hardware and software (stored in the storage device 916 or ROM 910, or both).

[0054] The present invention may be a system, method, or computer program product, or a combination thereof, in an integration of any possible level of technical detail. The computer program product may include a computer-readable storage medium (or multiple mediums) having computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0055] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the aforementioned storage devices. A non-exclusive list of more specific examples of computer-readable storage media may include portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures with recorded instructions, and any suitable combination of the aforementioned. When used herein, computer-readable storage media should not be interpreted as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or transient signals such as electrical signals transmitted through wires.

[0056] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or both. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0057] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially executed on the user's computer as a standalone software package, partially executed on the user's computer and partially executed on a remote computer, or fully executed on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions using state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0058] Aspects of the present invention will be described herein with reference to flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions are provided to a computer processor or other programmable data processing device, which can generate a machine, and as a result, instructions executed via the processor of the computer or other programmable data processing device create means for implementing functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, or other device, or a combination thereof, to function in a particular manner, thereby the computer-readable storage medium storing the instructions contains a product containing instructions that implement modes of functions / operations specified in one or more blocks of a flowchart or block diagram, or both.

[0060] Computer-readable program instructions can also be loaded into a computer, other programmable data processing device, or other device, causing a series of operational steps to be executed on the computer, other programmable device, or other device, thereby generating a computer implementation process, the instructions executed on the computer, other programmable device, or other device, which implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0061] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in the blocks may be performed in an order different from the order shown in the drawings. For example, two consecutively shown blocks may actually be implemented as a single step, may be executed simultaneously in a way that partially or entirely overlaps in time, may be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by an application-specific hardware-based system that performs a specified function or operation, or a combination of application-specific hardware and computer instructions.

[0062] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment currently known or to be developed in the future.

[0063] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0064] The characteristics are as follows: On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically when needed, without requiring human interaction from the service provider. Broad network access: Capabilities are available over a network and accessed through standard mechanisms that facilitate use by heterogeneous thin client platforms or thick client platforms (e.g., mobile phones, laptops, PDAs). Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model with different physical and virtual resources that are dynamically allocated and reallocated as needed. Consumers generally have no control or knowledge of the exact location of the resources provided, but they may be able to identify the location at a higher level of abstraction (e.g., country, state, or data center), thus demonstrating location independence. Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, to scale out rapidly, and released quickly to scale in rapidly. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time. Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). They can monitor, control, and report on resource usage, providing transparency to both service providers and consumers.

[0065] The service model is as follows: Software as a Service (SaaS): The ability offered to consumers is the use of a provider's applications running on a cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The ability offered to consumers is the ability to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, on a cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but they do control the deployed applications and, in some cases, the configuration of the environment hosting those applications. Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and, in some cases, limited control over selected networking components (e.g., host firewalls).

[0066] The deployment model is as follows: Private Cloud: A cloud infrastructure operates solely for a specific organization. This cloud infrastructure may be managed by that organization or a third party and may reside on-premises or off-premises. Community Cloud: A cloud infrastructure is shared by several organizations and supports a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or a third party and may reside on-premises or off-premises. Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by the organization that sells the cloud services. Hybrid Cloud: Cloud infrastructure is a configuration of two or more clouds (private, community, or public) that remain unique entities but are coupled together by standardized or proprietary technologies (e.g., cloudburst for load balancing between clouds) that enable data and application portability.

[0067] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0068] Referring now to Figure 5, an exemplary cloud computing environment 1000 is shown. As shown in the figure, the cloud computing environment 1000 comprises one or more cloud computing nodes 100, and local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1000A, a desktop computer 1000B, a notebook computer 1000C, or an automotive computer system 1000N, or a combination thereof, may communicate using the cloud computing nodes 100. The nodes 100 may communicate with each other. The nodes may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 1000 to provide infrastructure, a platform, or software, or a combination thereof, as a service that does not require cloud consumers to maintain resources on their local computing devices. The types of computing devices 1000A to 1000N shown in Figure 5 are for illustrative purposes only, and it should be understood that the computing node 100 and the cloud computing environment 1000 can communicate with any type of computerized device via any type of network, or a network addressable connection or a combination thereof (for example, using a web browser).

[0069] Referring now to Figure 6, we see a set of functional abstraction layers 1100 provided by the cloud computing environment 1000. It should be understood that the components, layers, and functionalities shown in Figure 6 are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionalities are provided:

[0070] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include a mainframe 1104, RISC (Reduced Instruction Set Computer) architecture-based servers 1106, 1108, blade servers 1110, storage devices 1112, and network and networking components 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0071] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities may be provided: a virtual server 1122, virtual storage 1124, a virtual network 1126 including a virtual private network, a virtual application and operating system 1128, and a virtual client 1130.

[0072] For example, the management layer 1132 may provide the functions described below. Resource provisioning 1134 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 1136 tracks costs as resources are used within the cloud computing environment and charges or invoices for the consumption of these resources. For example, these resources may include licenses for application software. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. User portal 1138 provides consumers and system administrators with access to the cloud computing environment. Service level management 1140 provides allocation and management of cloud computing resources to ensure that the required service levels are met. Service level agreement (SLA) planning and execution 1142 provides pre-configuration and procurement of cloud computing resources for which future requirements are anticipated in accordance with the SLA.

[0073] The workload layer 1144 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 1146, software development and lifecycle management 1148, virtual classroom education delivery 1150, data analysis processing 1152, transaction processing 1154, and contextual highlighting 1156. The contextual highlighting programs 110a and 110b provide a way to identify relevant content displayed on a shared screen in real time based on contextual queues and to highlight the relevant content.

[0074] The terms used herein are for the purpose of describing specific embodiments and are not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form as well, unless the context clearly indicates otherwise. It will be further understood that, when used herein, terms such as “comprises,” “comprising,” “includes,” “including,” “has,” “have,” “having,” and “with” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0075] The descriptions of various embodiments of the present invention are illustrative and not intended to be comprehensive or limitful to the embodiments disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the embodiments described. The terms used herein have been selected to best describe the principles of the embodiments, their practical applications, or the technical improvements to the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer implementation method for contextual digital content highlighting, The stage involves monitoring discussions among multiple parties in conjunction with the presentation, The steps include identifying the context of the monitored discussion, Based on the identified context, the step of identifying the most relevant portion of the displayed content associated with the presentation, The step of applying highlights to the most relevant identified portion. A step of determining whether the monitored discussion of the identified context among the multiple parties is continuing, If the monitored discussions among the aforementioned parties are ongoing, the stage of determining the validity of the relevant content, Based on the determined effectiveness, the steps include displaying the effectiveness indicator adjacent to the identified most relevant portion in the relevant content, A computer implementation method comprising the above.

2. The step of determining whether the monitored discussions by the aforementioned multiple parties are continuing is: The step of determining whether the monitored discussion contains any indication that one or more of the parties express doubts or questions about the validity of the relevant content, The computer implementation method according to claim 1.

3. The step of identifying the aforementioned context is, The computer implementation method according to claim 1, further comprising the step of semantically analyzing the combination of linguistic and non-linguistic data in the monitored discussion to identify the context.

4. A computer implementation method for contextual digital content presentation, The stage involves monitoring real-time discussions among multiple parties participating in a discussion that are linked to a displayed presentation, and the monitoring is performed by a machine learning device. A step of identifying the context of the monitored discussion among one or more participants and the reactions of the one or more participants to the monitored discussion, wherein the reactions of the one or more participants are expressed verbally or nonverbally, The machine learning device semantically analyzes the displayed presentation and verbal or nonverbal reactions in context, and the machine learning device classifies the discussion according to the topics of the discussion. The machine learning device determines whether or not it is necessary to display new content, and acquires the new content. A step of identifying the most relevant portion of the presentation and the new content, and applying highlights to the identified most relevant portion, wherein the identified new content includes tokenized text. The steps include generating similarity scores for each of the multiple parts of the new content, and selecting the part with the highest similarity score among the multiple parts as the most relevant part of the new content, The step of determining that the monitored discussion relating to the identified context is continuing among two or more participants, If the monitored discussion relating to the identified context is continuing among two or more participants, the step is to determine the validity of the most relevant identified portion, Based on the determined effectiveness, the steps include displaying the effectiveness indicator adjacent to the identified most relevant portion in the relevant content, A computer implementation method comprising the above.

5. The computer implementation method according to any one of claims 1 to 4, wherein the applied highlight is selected from the group consisting of font size, font color, text background color, font style, static arrows, and dynamic arrows.

6. The computer implementation method according to any one of claims 1 to 4, wherein the step of identifying the most relevant portion of the displayed content associated with the presentation based on the identified context comprises the step of tokenizing the text within the displayed content to generate a plurality of content sections.

7. The computer implementation method according to claim 6, wherein the step of identifying the most relevant portion of the displayed content associated with the presentation based on the identified context includes the step of generating a similarity score for each portion of the plurality of content portions by comparing the identified context with each content portion and selecting the content portion having the highest similarity score compared with the remaining content portions of the plurality of content portions.

8. The computer implementation method according to claim 6, wherein the displayed content includes one or more images, image analysis is performed on the one or more images to generate a textual image representation that describes each of the one or more images in text format, and the generated textual image representation is added to the multiple content sections.

9. A computer system for providing a presentation having contextual digital content highlights, comprising one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is The stage involves monitoring discussions among multiple parties in conjunction with the presentation, The steps include identifying the context of the monitored discussion, Based on the identified context, the step of identifying the most relevant portion of the displayed content associated with the presentation, The step of applying highlighting to the most relevant part identified above, A step of determining whether the monitored discussion of the identified context among the multiple parties is continuing, If the monitored discussions among the aforementioned parties are ongoing, the stage of determining the validity of the relevant content, Based on the determined effectiveness, the step is to display the effectiveness indicator adjacent to the identified most relevant portion in the relevant content. A computer system capable of performing a method that includes the following features.

10. A computer system for providing a presentation having contextual digital content highlights, comprising one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is The stage involves monitoring real-time discussions among multiple parties participating in a discussion that are linked to a displayed presentation, and the monitoring is performed by a machine learning device. A step of identifying the context of the monitored discussion among one or more participants and the reactions of the one or more participants to the monitored discussion, wherein the reactions of the one or more participants are expressed verbally or nonverbally, The machine learning device semantically analyzes the displayed presentation and verbal or nonverbal reactions in context, and the machine learning device classifies the discussion according to the topics of the discussion. The process includes determining whether new content needs to be displayed by the machine learning device and acquiring the new content, A step of identifying the most relevant parts of the presentation and the new content, and applying highlights to the identified most relevant parts, wherein the step of identifying the new content includes a step of tokenizing text, The steps include generating similarity scores for each of the multiple parts of the new content, and selecting the part with the highest similarity score among the multiple parts as the most relevant part of the new content, The step of determining that the monitored discussion relating to the identified context is continuing among two or more participants, If the monitored discussion relating to the identified context is continuing among two or more participants, the step is to determine the validity of the most relevant identified portion, Based on the determined effectiveness, the steps include displaying the effectiveness indicator adjacent to the identified most relevant portion in the relevant content, A computer system capable of performing a method that includes the following features.

11. The computer system according to claim 9 or 10, wherein the applied highlight is selected from the group consisting of font size, font color, text background color, font style, static arrows, and dynamic arrows.

12. The computer system according to claim 9 or 10, wherein the step of identifying the most relevant portion of the displayed content associated with the presentation based on the identified context comprises the step of tokenizing the text within the displayed content to generate a plurality of content sections.

13. The computer system according to claim 12, wherein the step of identifying the most relevant portion of the displayed content associated with the presentation based on the identified context includes the step of generating a similarity score for each portion of the plurality of content portions by comparing the identified context with each content portion and selecting the content portion having the highest similarity score compared with the remaining content portions of the plurality of content portions.

14. The computer system according to claim 12, wherein the displayed content includes one or more images, image analysis is performed on the one or more images to generate a textual image representation that describes each of the one or more images in text format, and the generated textual image representation is added to the plurality of content sections.

15. In the processor, Procedures for monitoring discussions among multiple parties in conjunction with presentations, A procedure for identifying the context of the monitored discussion, A procedure for identifying the most relevant portion of the displayed content associated with the presentation based on the identified context, The procedure for applying highlighting to the most relevant identified portion. A procedure for determining whether the monitored discussion of the identified context among the multiple parties is continuing, A procedure for determining the validity of relevant content when the monitored discussions among the aforementioned parties are ongoing, A procedure for displaying the effectiveness indicator adjacent to the identified most relevant portion in the relevant content, based on the determined effectiveness, A computer program that executes something.

16. In the processor, A procedure for monitoring real-time discussions among multiple parties participating in a discussion that is linked to a displayed presentation, wherein the monitoring is performed by a machine learning device. A procedure for identifying the context of the monitored discussion between one or more participants and the reactions of the one or more participants to the monitored discussion, wherein the reactions of the one or more participants are expressed verbally or nonverbally, A procedure for semantically analyzing the displayed presentation and verbal or nonverbal reactions in context, wherein the machine learning device classifies the discussion according to the topics of a plurality of discussions; The procedure for determining whether new content needs to be displayed by the machine learning device and for acquiring the new content, A procedure for identifying the most relevant portion of the presentation and the new content, and applying highlights to the identified most relevant portion, wherein the procedure for identifying the new content includes a procedure for tokenizing text, and A procedure for generating similarity scores for each of the multiple parts of the new content, and selecting the part with the highest similarity score among the multiple parts as the most relevant part of the new content, A procedure for determining whether the monitored discussion relating to the identified context is continuing among two or more participants, If the monitored discussion relating to the identified context is continuing among two or more participants, the step is to determine the validity of the most relevant identified portion, A procedure for displaying the effectiveness indicator adjacent to the identified most relevant portion in the relevant content, based on the determined effectiveness, A computer program that executes something.

17. The computer program according to claim 15 or 16, wherein the applied highlight is selected from the group consisting of font size, font color, text background color, font style, static arrows, and dynamic arrows.

18. A computer program according to claim 15 or 16, wherein the step of identifying the most relevant portion of the displayed content associated with the presentation based on the identified context includes the step of tokenizing the text within the displayed content to generate a plurality of content sections.

19. The computer program according to claim 18, wherein the step of identifying the most relevant portion of the displayed content associated with the presentation based on the identified context comprises the step of generating a similarity score for each portion of the plurality of content portions by comparing the identified context with each content portion and selecting the content portion having the highest similarity score compared with the remaining content portions of the plurality of content portions.

Images