Dynamically changing capabilities of a real-time communication session

Abstract

Presented herein is an approach for dynamically modifying the functionality of an RTC session that includes one or more data streams between participants in a real-time communication session 100. Accordingly, embodiments of the present invention are directed to enabling an RTC system to modify, extend, and / or improve the functionality of the RTC system while the RTC session is in progress. An exemplary embodiment includes identifying a request for a first algorithm A1 of a set of algorithms (A1, A2, A3) and dynamically modifying the functionality of the real-time communication session by adding the first algorithm A1 to the real-time communication session 100 as a virtual participant using at least one data stream from the one or more data streams.

Description

[Technical Field]

[0001] The present invention relates to the field of real-time communications (RTC), and more particularly to the functioning of RTC sessions. [Background technology]

[0002] US Patent Application Publication No. 2015 / 002619 discloses a scalable Web Real-Time Communications (WebRTC) media engine, as well as related methods, systems, and computer-readable media.

[0003] WO 2019 / 070822 discloses collaboration via chat in a healthcare system.

[0004] Real-time communication (RTC) is the near-simultaneous exchange of information from a sender to a receiver via any type of telecommunications service, associated with negligible latency. Examples of RTC typically employ real-time data streams (e.g., video, audio, and / or data streams) and include voice over landlines and mobile phones, VoIP, video and audio conferencing, chat, and robotic remote presence. Thus, RTC does not include email, bulletin boards, messaging applications, blogs, or other forms of Internet communication, where users read recorded content, which may incur significant delays between data transmission and reception. RTC is typically not stored in an interim state anywhere between the sender and receiver. Within that context, "real-time communication" can be considered a synonym for "live communication." Summary of the Invention [Problem to be solved by the invention]

[0005] Many products, systems, and applications are known for facilitating RTC sessions. The functionality provided by these can vary widely. However, for each product / system / application, the functionality provided is static (i.e., cannot be changed) and defined (e.g., hard-coded) as part of its configuration. For example, available functionality is available from the start of an RTC session and remains available (used or not) until the end of the RTC session. [Means for solving the problem]

[0006] Among other things, it is an object of the present invention to provide an improved system and method for dynamically changing the functionality of a real-time communication session. The invention is defined by the independent claims. The dependent claims define advantageous embodiments.

[0007] In general, in one aspect, the present invention focuses on a method for dynamic modification of functionality of a real-time communication session, including establishing a real-time communication session, identifying requirements for a first algorithm of a set of algorithms, each algorithm of the set of algorithms being configured to provide an associated functionality when executed during the real-time communication session, and adding the first algorithm to the real-time communication session as a virtual participant.

[0008] The proposed embodiments provide a concept for modifying the functionality of an RTC session. For example, the embodiments may be used to integrate new or additional algorithms into a (running) RTC session. Such algorithms may include, for example, newly developed algorithms for analyzing audio / video / text / data-related data streams, which may be useful additions in the RTC process. Thus, the embodiments help to provide improved or extended functionality to existing RTC implementations.

[0009] In particular, in one approach disclosed herein, an algorithm is added to an RTC session as a new (virtual) participant in the RTC session. By being added as a (virtual) participant, such an algorithm can be exposed to all data streams of the RTC session. As a participant in the RTC session, the algorithm may then be configured to employ one or more real-time data streams.

[0010] As an example, the proposed embodiments may enable dynamic selection of a larger set of algorithms, which may then be applied to one or more data streams of an RTC session. Unlike traditional RTC approaches that require algorithms / functions to be included in the system / application before establishing an RTC session, the proposed concepts facilitate modification (e.g., addition, removal, change) of algorithms / functions of an RTC session during the RTC session. Thus, simple integration and dynamic use of existing and / or new algorithms during an RTC session (e.g., a conference call) may be achieved by the proposed embodiments.

[0011] Typically, the functionality available during a conventional RTC session is determined by the system configuration and is therefore fixed. Therefore, such conventional RTC sessions do not allow for the addition of new functionality (e.g., new or improved algorithms for processing audio / video / data) during the RTC session as needed. In contrast, the proposed embodiments may provide the ability to easily extend an RTC session with new or different algorithms for processing any or all of the data streams (e.g., audio / video / data streams) exchanged between participants in the RTC session. The proposed concept may also allow for the definition and / or control of which data streams from which participants can be processed by which algorithms. Furthermore, the proposed concept may allow for the definition and / or control of which participants are provided with results / output from new or different algorithms.

[0012] By way of example, proposed embodiments may implement a wrapper (e.g., that may use a public application programming interface, API) that facilitates the incorporation of independently developed algorithms into an RTC session. In this way, newly developed and / or newly available algorithms can be integrated into an RTC session via the API. Furthermore, embodiments may also provide components (e.g., a container) for hosting such a set of algorithms, components (e.g., a REST-API or messaging service) for selecting from these algorithms, and components (e.g., a headless client) for incorporating these algorithms into an RTC session.

[0013] Embodiments can leverage the idea of ​​using wrappers that allow independent algorithm configuration and deployment. Also, embodiments provide a high degree of flexibility by facilitating the ability to add / remove algorithms to / from an RTC session at runtime (i.e., during an RTC session, rather than before). Thus, the proposed concept can provide for dynamic application of algorithms to a data stream during an established RTC session.

[0014] By way of further example, the dynamic nature of functionality changes provided by proposed embodiments is facilitated through: (i) Triggering an algorithm at a specific time period defined by the context (workflow type) of the data stream. In this case, the trigger can be a manual trigger provided by an administrator (physician), an automatic trigger (e.g., time-scheduled depending on a process step), and / or a context-sensitive trigger (e.g., depending on the results of a previous measurement made by another algorithm). (ii) Triggering algorithms configured to operate on specific data types (that are now part of the data stream). This allows you to specify which streams available in the RTC session should be processed by the algorithm being added to the conference. For example, when an algorithm is added to an RTC session as a participant, it can select which available streams of which participants should be processed; for example, algorithm 1 can be configured to process audio from participant 2, and algorithm 2 can be configured to process audio and video from participant 3.

[0015] Embodiments can increase the value of RTC by enabling modification of RTC session functionality in a dynamic manner, which can, for example, take into account a particular (e.g., selected or current) clinical context. Thus, embodiments can increase the clinical value of an RTC session by using context information for the modified and / or enhanced functionality provided.

[0016] In one embodiment, the process of adding a first algorithm as a virtual participant to a real-time communication session can include invoking the first algorithm through a container and configuring the first algorithm as a virtual participant in the real-time communication session by the container. This configuration can be performed in real time, thus enabling dynamic and real-time adjustment of the functionality of the RTC session. This can enable definition and control of what the first algorithm is applied to (which stream or which participant) and where the results of the algorithm should be communicated (which stream and which participant). Thus, the container can be configured to process commands and control the addition of the algorithm as a participant. The container can, for example, be connected to a real-time system and receive commands to dynamically add / remove algorithms during a multi-party RTC session. Thus, the container can function as a transport layer (transporting participant streams to the algorithm and vice versa).

[0017] In one embodiment, the request for the first algorithm may include an indication of the requirements of the first algorithm. Adding the first algorithm to the real-time communication session as a virtual participant may then include configuring at least one of the inputs and outputs of the first algorithm based on the indication of the requirements of the first algorithm. This may provide a way to specify which streams available in the RTC session should be processed by the first algorithm. By way of example, a wrapper may be associated with independently developed algorithm(s) to enable them for use with the RTC session. Data compatibility can be ensured by the proposed embodiment.

[0018] For example, in one embodiment, each algorithm in the set of algorithms may be associated with a respective wrapper that defines at least one of the inputs, outputs, and requirements of the algorithm, and adding a first algorithm to a real-time communication session as a virtual participant includes configuring at least one of the inputs, outputs, and requirements of the first algorithm based on the wrapper associated with the first algorithm.

[0019] Identifying the request may include determining the request to the first algorithm based on at least one of a user input signal provided by a participant in the real-time communication session (e.g., input in a chat function), a context of the data stream of the real-time communication session, the occurrence of a predetermined trigger event, and a non-predetermined trigger (e.g., a button press on an application UI).

[0020] Embodiments may further include removing the first algorithm as a virtual participant in the real-time communication session.

[0021] In some embodiments, a first algorithm may be configured to generate a request for a second algorithm of the set of algorithms when executed during a real-time communication session. In this manner, a first algorithm added to an RTC session may be used to trigger the addition of a further second algorithm to the RTC session.

[0022] Participants in an RTC session may include medical professionals. The real-time communication session may be configured for the exchange of clinical information for clinical decision support.

[0023] As a further example, at least one algorithm of the set of algorithms, when executed, may be configured to store and / or retrieve data to / from an electronic medical record database.

[0024] It is also proposed that the concept(s) for modifying the functionality of an RTC session can be utilized to complement a face-to-face interaction where participants are co-located. In this way, additional functionality can be added to, for example, a live interaction between two participants.

[0025] For example, if two participants are engaged in an interaction in the same location (e.g., in a room, at a conference location, in the same geographic location, etc.), the interaction can be considered a face-to-face (or direct) interaction because the participants are physically present with each other and each participant can communicate with the other without intermediary technology. It will be understood that such face-to-face interactions may be supplemented by creating an RTC session (e.g., a virtual RTC session) to accompany the face-to-face interaction, and that the RTC session may then use the proposed concepts to integrate new or additional algorithms into the RTC session, and that these new or additional algorithms may be used to improve or extend functionality for the face-to-face interaction.

[0026] As an example, a face-to-face interaction may occur in a medical examination room, involving a medical professional (as a first participant) and a subject (as a second participant). A system according to a proposed embodiment may be provided in a medical examination room to conduct an RTC session involving face-to-face interactions. In response to identifying an algorithmic request with an associated function, a first algorithm may be added to the RTC session as a virtual participant, thereby providing the associated function. For example, the system may include a first device associated with a medical professional for processing audio / video / text / data from and / or to the medical professional, and a second device associated with a patient for processing audio / video / text / data from and / or to the patient. The first and second devices may connect to (i.e., become participants in) the RTC session and be used to provide the algorithmic request. In this manner, the medical professional may request the provision of a function via the first device (e.g., by speech, via gestures, via input / selection, etc.), and the subject may provide data for processing by the function via the second device (e.g., via a sensor, microphone, input device, etc.). For example, functionality to assist in-person medical consultations by performing automated tests and / or providing diagnostic assistance can be provided by proposed embodiments.

[0027] Embodiments may be used to integrate new or additional algorithms into RTC sessions involving face-to-face interactions (e.g., meetings or doctor-subject consultations). Such algorithms may include, for example, newly developed algorithms for analyzing audio / video / text / data that may be useful additions to face-to-face interactions / meetings. Thus, embodiments may be useful for providing improved or expanded functionality to face-to-face interactions / meetings via accompanying RTC implementations.

[0028] As an example, the proposed embodiments may enable dynamic selection of an algorithm, and then the selected algorithm may be applied to one or more data streams provided by participants in a face-to-face meeting. Such proposed embodiments may facilitate modification (e.g., addition, removal, modification) of algorithms / functionality available to participants in a face-to-face meeting. Thus, simple integration and dynamic use of existing and / or new algorithms during face-to-face interactions may be achieved by the proposed embodiments.

[0029] The proposed embodiments may provide the ability to easily extend face-to-face interactions with new or different algorithms that process any or all of the data streams (e.g., audio / video / data streams) exchanged between participants in a face-to-face interaction. The proposed concept may also allow for the definition and / or control of which data streams from which participants can be processed by which algorithms. Furthermore, the proposed concept may allow for the definition and / or control of which participants are provided with results / output from new or different algorithms.

[0030] Embodiments can leverage the idea of ​​implementing an RTC session to accompany a face-to-face interaction, and then using the proposed embodiments to add / remove algorithms to / from the RTC session at runtime. Thus, the proposed concept may provide for dynamic application of algorithms to data streams during face-to-face interaction via accompanying RTC sessions.

[0031] As a further example, the dynamic nature of the functionality changes provided by proposed embodiments may be facilitated by triggering algorithms. The triggers may be manual triggers provided by an administrator (e.g., a physician), automatic triggers (e.g., dependent on processing steps, detected instructions, sensed values, etc.), and / or context-sensitive triggers (e.g., dependent on previous measurements made by another algorithm).

[0032] Embodiments can increase the value of a face-to-face interaction by enabling modification of the functionality of an accompanying RTC session in a dynamic manner, which can, for example, take into account a particular (e.g., selected or current) context. Where a face-to-face interaction has a clinical context (e.g., a physician-subject consultation), embodiments can increase the clinical value of the interaction by establishing an RTC session to accompany the face-to-face interaction and then using the context information to provide modified and / or extended functionality via the RTC session.

[0033] Thus, a method for dynamic modification of functionality of a face-to-face interaction between a first participant and a second participant may be provided, including modifying functionality of a real-time communication session according to proposed embodiments, wherein the real-time communication session is established to provide at least one functionality to the participants of the face-to-face interaction. For example, such a proposed method may include establishing a real-time communication session for providing at least one functionality to the participants of the face-to-face interaction, identifying a request for a first algorithm of a set of algorithms, each algorithm of the set of algorithms configured to provide an associated functionality when executed during the real-time communication session, and adding the first algorithm to the real-time communication session as a virtual participant.

[0034] According to another aspect, there is provided a computer program product for dynamically changing functionality of a real-time communication session, the computer program product including a computer readable storage medium having computer readable program code configured to perform all of the steps of the proposed embodiments.

[0035] A computer program product for dynamically changing functionality of a face-to-face interaction between a first participant and a second participant is also provided, the computer program product including a computer readable storage medium having computer readable program code configured to perform all of the steps of the proposed embodiments.

[0036] Therefore, there is also provided a computer system comprising a computer program product according to the proposed embodiments and one or more processors adapted to perform methods according to the proposed concepts by execution of computer readable program code of said computer program product.

[0037] According to yet another aspect of the present invention, a system for dynamic modification of functionality of a real-time communication session is provided, the system having a communications component configured to establish the real-time communication session, a processing component configured to identify a requirement for a first algorithm of a set of algorithms configured to provide an associated functionality when executed during the real-time communication session, and a modification component configured to add the first algorithm to the real-time communication session as a virtual participant.

[0038] The system may be located remotely from a user device for participating in an RTC session. In this manner, a user (e.g., a medical professional) may have a suitably configured system capable of receiving content / information at a location remote from the system for dynamic modification of the functionality of the RTC session. Thus, embodiments enable a user to dynamically modify the functionality of an RTC session using a local system (which may include, for example, a laptop, tablet computer, mobile phone, PDA, or other portable display device). By way of example, embodiments may provide an application for a mobile computing device, where the application may be executed and / or controlled by a user of the mobile computing device.

[0039] The system may further include a server device including a system for dynamically changing the functionality of the RTC session, and a client device including a user interface. Thus, dedicated data processing means may be used for the purpose of modifying the functionality of the RTC session and thus reducing the processing requirements or capabilities of other components or devices of the system.

[0040] The system may further include a client device, which comprises a processing component, a modification component, and a display unit. In other words, a user (such as a doctor or medical professional) may have a suitably configured client device (such as a laptop, tablet computer, mobile phone, PDA, etc.) that processes the received data to modify the functionality of the RTC session and generate display control signals. By way of example only, embodiments may thus provide a monitoring or observation system that allows monitoring of one or more subjects (e.g., patients) from a single location, where real-time communication between the subjects and a monitoring user (e.g., a nurse or doctor) is provided, and which may have its functionality extended or modified in accordance with the proposed concepts.

[0041] Also provided is a system for dynamically modifying capabilities of a face-to-face interaction between a first participant and a second participant, the system comprising a system for dynamically modifying capabilities of a real-time communication session according to the proposed embodiments.

[0042] Thus, it will be appreciated that processing power may be distributed throughout the system in different manners according to predetermined constraints and / or availability of processing resources.

[0043] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0044] Examples according to aspects of the present invention will now be described in detail with reference to the accompanying schematic drawings. [Figure 1] 1 is a simplified diagram of an RTC session according to an example embodiment. [Figure 2] 1 is a simplified diagram of an example embodiment in which a first algorithm is added to a multi-party RTC session. [Figure 3] 1 is a simplified diagram of an exemplary embodiment using algorithm chaining. [Figure 4] 1 is a simplified diagram of an exemplary embodiment in which a first algorithm is added as a (virtual) participant to an RTC session. [Figure 5] 1 is a simplified representation of adding first and second algorithms to an RTC session according to one embodiment. [Figure 6A] FIG. 1 illustrates an exemplary approach for including an algorithm as a virtual participant in an RTC session according to a proposed embodiment. [Figure 6B] FIG. 1 illustrates an exemplary approach for including an algorithm as a virtual participant in an RTC session according to a proposed embodiment. [Figure 7] FIG. 6C is a sequence diagram illustrating exemplary interactions between the components of FIG. 6B in accordance with the proposed embodiment. [Figure 8] 1 is a simplified diagram of an exemplary embodiment that facilitates adding, removing, or modifying features to a face-to-face meeting between a physician and a subject. [Figure 9] 10 is a simplified diagram of an RTC session according to another example embodiment. [Figure 10] A simplified diagram of an exemplary embodiment for a face-to-face meeting between a doctor and a subject, where the doctor and the subject are the first and second participants in the RTC session, respectively, and a first algorithm is added to the RTC session as a (virtual) participant. [Figure 11] FIG. 1 is a simplified block diagram of a computer that can use one or more portions of the embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0045] Proposed is an approach to allow dynamic modification of the functionality of an RTC session: Embodiments can add algorithms as virtual participants to an RTC session.

[0046] Such proposals may therefore facilitate simple and intuitive addition, removal or modification of functionality of an RTC session, which may be done, for example, taking into account the context of the RTC session.

[0047] Accordingly, embodiments of the present invention aim to enable the functionality of an RTC system to be modified, extended, and / or improved while the RTC system is conducting an RTC session. Furthermore, embodiments may aim to enable the provision of relevant or important functionality at appropriate times during an RTC session. Thus, embodiments may support, for example, clinical support assessment, medical diagnosis, and / or support planning.

[0048] Embodiments may also be utilized in combination with face-to-face interactions to provide complementary functionality to one or more participants in the face-to-face interaction.

[0049] By way of example only, the exemplary embodiments may be utilized in many different types of clinical, medical or subject-related environments, such as hospitals, clinics, wards, care homes, people's homes, etc.

[0050] To provide a context for the description of the elements and functionality of the exemplary embodiments, drawings are provided below as examples of how aspects of the exemplary embodiments may be implemented. It should therefore be understood that the drawings are examples only and are not intended to assert or imply any limitation with regard to the environments, systems, or methods in which aspects or embodiments of the invention may be implemented.

[0051] Note that many algorithms for audio / video / data / text processing already exist, but this is not facilitated (or provided) in existing RTC systems. Because such RTC systems are already defined, their functionality can only be modified (e.g., to incorporate such algorithms) by being redefined (e.g., reprogrammed) and redistributed before establishing a new RTC session.

[0052] It is proposed to provide one or more concepts that allow for modification (e.g., addition, removal, reconfiguration, etc.) of functionality of an RTC system during an RTC session (i.e., while the system is conducting an RTC session). Specifically, embodiments may facilitate adding / removing / modifying algorithms to / from an RTC session by adding or removing algorithms as participants in the RTC session. Embodiments may also allow for definition and / or control of which participants are provided with outputs from algorithms.

[0053] Current algorithms are specifically implemented "services" of the RTC system and cannot be added or removed by the RTC system during an RTC session. Therefore, specific algorithms must be included when the RTC system is programmed.

[0054] On the other hand, proposed embodiments may implement a wrapper (e.g., using a public API) that facilitates adding independently developed algorithms to an RTC session, so that algorithms found on the Internet and / or newly developed algorithms can be easily integrated into an RTC session by being invoked and added as participants in the RTC session.

[0055] The dynamic nature of the embodiments can be realized in various ways, such as (a) triggering algorithms at specific time periods, (b) triggering algorithms configured to operate on specific data types, etc.

[0056] (a) Triggering of algorithms at specific time periods defined by the data stream context (workflow type).

[0057] In this case, it can be either manually triggered by an administrator (e.g., a physician), automatically triggered (e.g., time-scheduled depending on the process step), or context-dependent. For example, in telemedicine, a physician typically knows in advance the type of conference call he or she will be conducting (e.g., patient admission, discussion of test results, evaluation of therapy progress, etc.). In each of these tests, the administrator typically follows a predefined series of steps that may be linked to specific algorithms that the system may suggest to execute. Embodiments can provide a "context-sensitive" menu to the administrator (e.g., a physician) based on the test selection. Embodiments can track the progress of the test over time and change the menu selection depending on the amount of time that has passed.

[0058] As an example, Figure 1 is a simplified diagram of an RTC session having a first participant A and a second participant B. More specifically, the RTC session is a doctor-subject consultation in which the first participant A is a doctor and the second participant B is a subject. A full-duplex communication link 100 between the first participant A and the second participant B communicates audio, video, and data streams (as indicated by the symbols in Figure 1).

[0059] A first algorithm A1, a second algorithm A2, and a third algorithm A3 may be used. Specifically, the first algorithm A1 is configured, when executed, to authenticate patient B. The second algorithm A2 is configured, when executed, to measure the respiratory rate of patient B. The third algorithm A3 is configured, when executed, to capture an image (i.e., take a picture) of patient B.

[0060] Doctor A starts a video consultation with Patient B and selects the consultation type to be "discussion of test results." The RTC application (automatically) triggers a first algorithm A1 to authenticate Patient B at the start of the consultation. After successful authentication, Doctor A starts discussing the test results and a respiratory rate measurement is needed to check the results. Therefore, the doctor (explicitly) requests the execution of a second algorithm A2 to measure Patient B's respiratory rate, and the measured respiratory rate is provided to Doctor A. Because the measured respiratory rate exceeds a predetermined threshold, a third algorithm A3 (automatically) executes to capture an image of Patient B (i.e., take a photo). In this way, the execution of the third algorithm A3 can be considered context-sensitive.

[0061] (b) Triggering an algorithm configured to operate on a particular data type (which has become part of the data stream).

[0062] Embodiments may provide the ability to specify which data streams available in an RTC session should be processed by algorithms added to the RTC session. Wrappers may be associated with independently developed algorithm(s) to enable their use with the RTC solution.

[0063] An embodiment that uses such a wrapper can ensure data compatibility. For example, existing RTC solutions may use standards such as VP8 for video, OPUS for audio, and JSON-formatted text for data by default. These standards are used by most meeting systems and existing algorithms. There are exceptions (such as IOS-based solutions) that use H.262 / H.264 for video streams only. H.264 is a widely known and supported video format. Therefore, data compatibility can be guaranteed by the proposed embodiment.

[0064] When an algorithm is added to an RTC session, it can select which available streams of which participants should be processed.

[0065] The system can also detect the connection quality / bandwidth of the RTC session. In this way, embodiments can determine whether the minimum requirements of a new or third-party algorithm for reliable processing results are met. The user who invoked the algorithm can then be notified so that appropriate action can be taken. For example, a camera may not be of the right quality for a certain algorithm. If it is detected that the resolution is too low, appropriate feedback can be provided. If the bandwidth is limited, the video / audio quality may be reduced. For example, a message may be provided to notify the user. The minimum requirement specifications may be included in a "wrapper" to enable embodiments to detect limitations.

[0066] - Innovation Rack Manager (IRM)

[0067] The Innovation Rack Manager (IRM) is configured to host information about available algorithms and their wrappers, as well as their (minimal) requirements. The IRM can therefore be thought of as a host for the set of available algorithms. Algorithms can be requested from the IRM and added as participants to an RTC session. The IRM can be hosted in the cloud or on-premise.

[0068] - Addition of algorithms as virtual participants

[0069] Embodiments may be configured to utilize the functionality of existing RTC systems to add algorithms as (virtual) participants to an RTC session. Such a proposed approach may allow application logic to define and control which streams an algorithm can access and where its results should be presented. For example, an API may be used that allows this to be specified when triggering or requesting an algorithm. Algorithms can be dynamically added / removed from an RTC session as needed by treating the algorithm as a virtual participant in the RTC session. For this purpose, basic / traditional functionality of the RTC system can be used.

[0070] 2 is a simplified diagram of an exemplary embodiment in which a first algorithm A1 is added to a multi-party RTC session, where the RTC session has a first participant A, a second participant B, and a third participant C. A full-duplex communication link 100 between the first participant A, the second participant B, and the third participant C communicates audio, video, and data streams (as indicated by symbols in FIG. 2).

[0071] The first algorithm A1 is added to the RTC session as a (virtual) participant.

[0072] A first algorithm A1 is configured to receive input from a first participant A, a second participant B, and a third participant C, as indicated by the dotted arrows.

[0073] As indicated by the solid thick arrow, the first algorithm A1 is configured to communicate its output only to the third participant C.

[0074] As a further example, FIG. 3 is a simplified diagram of an exemplary embodiment using a chain of algorithms, where an algorithm uses the output of a previous algorithm before feeding back the results to the participants in the call.

[0075] Here, the RTC session has a first participant A and a second participant B. A full-duplex communication link 100 between the first participant A and the second participant B communicates audio, video, and data streams (as indicated by the symbols in FIG. 3).

[0076] A first algorithm A1, a second algorithm A2, and a third algorithm A3 are added to the RTC session as (virtual) participants.

[0077] As shown by the arrows in Figure 3, the first algorithm A1 communicates its output to the second algorithm A2, which communicates its output to the third algorithm A3, which returns its output to the participants in the RTC session.

[0078] When added to an RTC session, an algorithm is represented as an additional (virtual) participant. An example of this proposed concept is shown in Figure 4. Figure 4 shows a simplified diagram of a first algorithm A1 added as a (virtual) participant to an RTC session.

[0079] In the example of FIG. 4, an RTC session has a first participant A and a second participant B, and a first algorithm A1 is added as a third participant.

[0080] As indicated by the dotted arrows, the first algorithm A1 is configured to receive input from a first participant A and a second participant B. As a result, the first algorithm A1 receives all data streams as potential inputs.

[0081] The first algorithm A1 is configured to communicate its output to a first participant A and a second participant B, as indicated by the solid thick arrows.

[0082] -Added algorithms

[0083] Participants in an RTC session can add algorithms. For example, a participant can trigger or request an algorithm by sending a request message in the RTC session. The request message is detected by a component of the IRM, which identifies the requested algorithm and adds the requested algorithm as a virtual participant in the RTC session.

[0084] An example of this process is shown in FIG. 5, which includes a simplified representation of adding first and second algorithms to an RTC session according to one embodiment.

[0085] A first user U1 and a second user U2 are participants in an RTC session 150. In this example, the RTC session includes a live discussion in which the first user U1 and the second user U2 participate in the real-time discussion by typing content into a discussion / chat application 200. In other embodiments, an audio / video / data stream may be provided between the first user U1 and the second user U2.

[0086] The monitoring component 220 of the IRM 250 monitors the content of the discussion / chat application 200 to identify algorithmic requirements.

[0087] 5, the monitoring component 220 identifies that a first user U1 requests a first algorithm A1. In response, the IRM 250 downloads an instance of the first algorithm A1 for the first user U1. U1 Generate a first algorithm instance A1 for the first user U1 As a participant in the RTC session 150, the monitoring component 220 also identifies that a second user U2 requests a second algorithm A2. In response, the IRM 250 creates an instance of the second algorithm A2 for the second user. U2 and generate a second instance of the algorithm A2 for the second user U2 Add as a participant in RTC session 150.

[0088] After joining an RTC session, the new participant can receive all streams currently available in the RTC session. To start processing, a second message can be sent to specify which streams should be processed by the algorithm and where the results should be displayed.

[0089] As mentioned above, to enable the use of and compatibility with existing third-party algorithms (e.g., that may be cloud-provided), algorithm wrappers can be employed. The wrappers can specify the input and output types of the algorithm, as well as any requirements of the algorithm (e.g., connection quality, speed requirements, memory space, etc.).

[0090] As an example, the wrappers can be represented in a tabular format as shown in Table 1 below. [Table 1]

[0091] From the above description, it will be appreciated that the proposed embodiments may be used in conjunction with conventional RTC systems to facilitate dynamic enhancement and / or flexible use of algorithms. For example, a third-party developer of an IRM algorithm may create a DLL (Dynamic Link Library) file that, when added to an IRM deployment, provides the ability to include the algorithm as a participant in an RTC session.

[0092] Embodiments may be used, for example, in a wide range of real-time communication systems and methods. Exemplary applications already identified as potentially benefiting from the proposed concepts for dynamically changing RTC session functionality include customer service support, doctor-patient real-time consultation, environmental monitoring (e.g., patient monitoring and elderly care), health self-assessment applications, and video surveillance applications (e.g., baby monitoring, remote video access, etc.).

[0093] From the above discussion, it can be seen that a system is proposed for creating a flexible and adaptive RTC system that can dynamically adjust its capabilities. Such a system can automatically add, remove, and / or modify algorithms as participants in an RTC session to adjust the available capabilities.

[0094] 6A and 6B, exemplary approaches for including an algorithm as a virtual participant in an RTC session are shown. More specifically, FIG. 6A shows how an algorithm is included as a (virtual) participant in an RTC session with one human participant. FIG. 6B shows how an algorithm is included as a (virtual) participant in an RTC session with two human participants.

[0095] 6A , a human participant 605 connects to a headless client 610 of an IRM 615. The headless client 610 connects to an algorithm 620 through a wrapper 625. The wrapper 625 feeds data from the RTC session to the algorithm 620. The wrapper 625 also feeds data from the algorithm 620 to the RTC session. For this purpose, the wrapper 625 may use an API. However, in some embodiments, the wrapper 625 may not use an API.

[0096] 6B, a first human participant 605A and a second human participant 605B of an RTC session connect to a headless client 610 of an IRM 615. The headless client 610 connects to a first algorithm 620A and a second algorithm 620B through a wrapper 625, where the first algorithm 620A provides a cloud-based service and the second algorithm 620B provides a C++ motion detection algorithm. The wrapper 625 uses various components (including components for APIs, C++, Matlab, etc.) to send and receive data to and from the algorithms 620A and 620B.

[0097] Referring now to FIG. 7, a sequence chart illustrating exemplary interactions between the components of FIG. 6B is shown.

[0098] In the exemplary sequence of FIG. 7, the following series of steps take place:

[0099] 705 - A first user / participant 605A invites a second user 605B to join the RTC session.

[0100] 710 - The second user 605B then responds and joins the RTC session as a second participant.

[0101] 715 and 720 - Audio, video and data streams are transmitted between the first participant 605A and the second participant 605B.

[0102] 725—The first user / participant 605A requests / invite the motion detection algorithm 620B to join the RTC session by sending an invitation / request to the headless client 610.

[0103] 730 - Next, the headless client 610 creates an algorithm instance by calling the algorithm wrapper 625.

[0104] 735—The algorithm wrapper 625 creates an API connection to the motion detection algorithm 620B.

[0105] 740—An instance of the motion detection algorithm 620B is created and provided to the headless client 610.

[0106] 745 - The headless client 610 adds the motion detection algorithm 620B as a participant in the RTC session.

[0107] 750 and 755 - Audio, video and data streams are sent from the first participant 605A and the second participant 605B to the headless client 610.

[0108] 760—A first user / participant 605A requests the use of a motion detection algorithm 620B to analyze the movements of a second user / participant 605B.

[0109] 765 - The headless client 610 passes the captured video frames of the second user / participant 605B to the motion detection algorithm 620B via the wrapper 625.

[0110] 770 - The captured video frames of the second user / participant 605B are passed to the motion detection algorithm 620B for analysis.

[0111] 775 - The results of the motion analysis are provided to the wrapper 625 by the motion detection algorithm 620B.

[0112] 780 - The results of the motion analysis are provided as a data stream from the wrapper 625 to the headless client 610.

[0113] 785—The results of the motion analysis are provided as a data stream from the headless client 610 to the first user / participant 605A.

[0114] It is also proposed that the concept of modifying the functionality of an RTC session can be leveraged to complement a face-to-face interaction (e.g., a co-located, in-person meeting). In this manner, additional functionality can be provided to one or more participants in the face-to-face meeting, and the addition of such functionality can be achieved dynamically during the meeting. In particular, an approach is provided for enabling the dynamic provision of functionality to a face-to-face interaction via an RTC session. For example, an RTC session may be established to accompany a face-to-face interaction, and the RTC session can then be used to dynamically provide functionality to participants in the face-to-face interaction.

[0115] Such suggestions may therefore facilitate simple and intuitive addition, removal, or modification of functionality to individual face-to-face meetings, where the functionality is provided via an RTC session, where the addition, removal, or modification of the functionality may be done taking into account, for example, the context and / or events of the face-to-face meeting.

[0116] Accordingly, embodiments of the present invention aim to enable functionality of individual face-to-face interactions to be modified, extended, and / or improved in real time (i.e., during the face-to-face interaction). Furthermore, embodiments may aim to enable relevant or important functionality to be provided at the appropriate time during individual face-to-face interactions. Thus, embodiments may support, for example, clinical support assessments, medical diagnoses, and / or support plans. By way of example only, exemplary embodiments may be utilized in many different types of clinical, medical, or subject-related environments, such as hospitals, clinics, wards, care homes, people's homes, etc.

[0117] In particular, the inventors have realized that face-to-face interactions can utilize the proposed concepts for dynamic modification of RTC session capabilities to provide improved or expanded functionality for the face-to-face interactions. In particular, by establishing an RTC session for providing functionality to the face-to-face interactions and then using the proposed embodiments for modifying the functionality of the RTC session, the functionality provided to the face-to-face interactions can be dynamically controlled and modified to improve the face-to-face interactions.

[0118] Such a proposal can therefore facilitate the use of additional functionality during face-to-face communication situations. Such functionality may be particularly interesting in clinical situations, e.g., doctor-subject consultations, where various activities may be performed routinely (e.g., vital sign monitoring, visual inspection of the subject, audio transcription, translation, audio analysis, emotion detection, recording of consultation summaries, etc.). This additional functionality can be added to live RTC sessions, for example, by voice commands, gesture commands, typing inputs via an input interface, or automatically, in a controlled, and / or dynamic manner upon detection of an event.

[0119] By way of example, a device may be provided at a location of a face-to-face interaction, where the device implements a proposed embodiment of a system for dynamic modification of RTC session functionality. Thus, the device can act as a "gateway" to additional functionality, where the functionality is provided by algorithms hosted locally or somewhere in the cloud. The algorithms can perform processing on audio / video / data streams obtained from the face-to-face interaction, and the results can be stored and / or presented to one or more participants, for example, on their computing devices and / or on a display device provided at the location of the face-to-face interaction.

[0120] It will therefore be appreciated that the proposed embodiments may provide a concept for supplementing face-to-face interactions (e.g., interpersonal meetings) with additional functionality / algorithms that process any or all of the available "audio / video / data" from the participants of the face-to-face interaction. Such face-to-face interactions may include multiple participants.

[0121] Referring now to Figure 8, an exemplary approach for supplementing face-to-face interactions (e.g., in-person meetings) with additional features / algorithms is shown. More specifically, Figure 8 is a simplified diagram of an exemplary embodiment that uses an RTC session to facilitate adding, removing, or modifying features to a face-to-face meeting between a physician and a subject.

[0122] 8, doctor 800 and subject 805 are positioned on either side of a table 810 in an examination room 815. Doctor 800 and subject 805 then engage in a face-to-face interaction (i.e., a face-to-face meeting), and in doing so, a verbal interaction between doctor 800 and subject 805 is created.

[0123] Provided on the table are a video camera 825 and a microphone 830. The video camera 825 is configured to capture video footage of the subject 805, and the microphone 830 is configured to capture audio of the conversation between the doctor 800 and the subject 805. In this way, audio and video data of the face-to-face interaction is obtained.

[0124] The captured audio and video data of the face-to-face interaction is provided to IRM 830 via one or more appropriately arranged communication links (e.g., wireless, wired, or short-range communication links). IRM 835 is also communicatively coupled to various devices provided within examination room 815 (e.g., via one or more wireless, wired, or short-range communication links). By way of example only, the devices in this example include a smartphone 840, a display device 845, a printer 850, a speaker 855, and a data storage device 860. Additionally, IRM 835 is communicatively coupled to cloud-based communication and / or processing resources (e.g., the Internet) 865.

[0125] Thus, it will be appreciated that a wide range of devices and / or computing resources may be used to input data into and / or output data from the IRM 835 in a variety of forms. For example, the IRM 835 may use algorithms / functions that reside in the cloud 865 and / or output results via a display device 845 for viewing by both the physician 800 and the subject 805.

[0126] The system establishes an RTC session to provide at least one feature to the face-to-face interaction. A monitoring component (not shown) of the IRM 835 then monitors the content of the discussion 820 and input signals from the video camera 825 and smartphone 840 to identify a request for an algorithm. For example, the physician 300 can request the addition of a feature (e.g., a respiratory analysis algorithm) via a spoken command (e.g., by speaking the “KEYWORD” name of the algorithm followed by “start”), via performing a predetermined gesture, and / or by typing / selecting the desired feature via the smartphone 840.

[0127] In response to identifying the algorithm request, the IRM 835 creates an instance of the algorithm and adds the algorithm instance as a participant in the RTC session. The IRM 835 also provides feedback via the smartphone 840 and / or speaker 855 to indicate that the algorithm has started.

[0128] Similarly, removal of a function can be accomplished via a spoken command (e.g., by speaking the "KEYWORD" name of the algorithm followed by "stop"), and / or via performing a predetermined gesture, and / or via typing / selecting the desired function via smartphone 840. In response to identifying a request for algorithm removal, IRM 835 removes the instance of the algorithm as a participant in the RTC session. IRM 835 also provides feedback via smartphone 840 and / or speaker 855 to indicate that the algorithm has been stopped.

[0129] In this way, additional functionality for face-to-face interactions is provided by the IRM 835 present in the room 815. The IRM 835 may be entirely local (i.e., on-premise) or may leverage remote (i.e., off-premise) resources.

[0130] In some embodiments, it may be preferable to implement some system configurations to take into account particular features or characteristics of face-to-face interactions. For example, in the doctor-subject consultation example above, it may be useful to define seating positions for the doctor and subject so that the appropriate video streams are processed by the algorithm. Similarly, the microphone device may be configured to distinguish between audio from the subject and audio from the doctor.

[0131] Instead of manually defining seating positions, embodiments may be trained to recognize the physician's voice and face. Such a system can then discover who is located where in the room and what audio / video should be processed by the algorithm.

[0132] Additionally, some embodiments may be configured to allow only physicians to request additional functionality during an in-person visit, which may help avoid misuse of the system.

[0133] It may also be desirable to facilitate context-based triggering of functionality. For example, algorithm selection may be defined by the context of the consultation. In this manner, algorithms may be triggered by manual triggers (e.g., by an administrator), automatic triggers (e.g., process step dependent), or context-sensitive triggers.

[0134] As an example, a physician may know in advance the type of consultation he or she will be performing. In each of the various types of consultation, the physician may follow a predefined set of steps that may be linked to specific algorithms that the system may provide for execution. The system may provide "context-sensitive" menus or some other form of hints / prompts to the physician based on consultation selections provided by the physician prior to the consultation. Additionally, embodiments may track the progress of the consultation over time and automatically change menu selections or prompts depending on the amount of time that has passed.

[0135] Embodiments may be configured to leverage the proposed concept of adding algorithms as (virtual) participants to an RTC session. In particular, it is proposed to use an RTC session to enhance face-to-face interactions. For example, an RTC session may be established involving face-to-face interactions, and the RTC session can then use the proposed concept to dynamically add and / or remove algorithms that may enhance the face-to-face interactions.

[0136] As a further example, an exemplary approach for supplementing face-to-face interactions (e.g., face-to-face meetings) with additional features / algorithms is shown, where the room hosting the face-to-face meeting of the physician and subject is considered a participant in the RTC session.

[0137] More specifically, FIG. 9 is a simplified diagram of an RTC session according to an exemplary embodiment, in which the RTC session has first and second participants (i.e., PARTICIPANT-1 and PARTICIPANT-2, respectively). Here, the RTC session is established to involve a doctor-subject face-to-face consultation in a private examination room. According to the proposed concept, the RTC session is established to provide at least one function to the participants of the face-to-face interaction. A full-duplex communication link 870 between the first and second participants communicates audio, video, and data streams (as indicated by the symbols in FIG. 9).

[0138] The people in the examination room (i.e., the doctor and the subject) are represented as a single first participant, PARTICIPANT-1, in the RTC session. In this way, all participants in the face-to-face consultation are represented as a single participant in the RTC session. Algorithms can then be dynamically added / removed during the face-to-face interaction as desired (in accordance with the proposed concept detailed above). In this way, algorithms are treated as virtual participants in the RTC session; that is, when an algorithm is required to enhance the face-to-face interaction, the algorithm is added to the RTC session as a virtual participant. For example, Figure 9 shows an algorithm being added to an RTC session as a second participant, PARTICIPANT-2.

[0139] In this way, algorithms / functions can be added during the face-to-face interaction to process the audio / video or data from the face-to-face interaction. Some algorithms / functions can be used during different periods of the face-to-face interaction or simultaneously.

[0140] Instead of having all people in the examination room as a single participant in the RTC session, each participant in the face-to-face interaction can be a separate (virtual) participant in the RTC session. Such an approach is shown in Figure 10.

[0141] 10 is a simplified diagram of an exemplary embodiment in which each participant in a face-to-face interaction is a separate participant in an RTC session established to provide at least one function to the face-to-face interaction, where the physician and subject are the first and second participants (i.e., PARTICIPANT-1 and PARTICIPANT-2, respectively).

[0142] Algorithms can then be dynamically added / removed during a face-to-face interaction as desired (in accordance with the proposed concepts detailed above). For example, Figure 10 shows an algorithm being added to an RTC session as a third participant, PARTICIPANT-3.

[0143] From the above description, it can be seen that a concept has been proposed to facilitate adding / removing functionality to a face-to-face interaction. Such a concept can be based on the idea that once a face-to-face interaction is established, an accompanying RTC session can be established to provide functionality to the face-to-face interaction. Depending on which algorithms are activated, they are added as participants to the RTC session. This approach to adding / removing processing algorithms to an RTC session has the same dynamics as adding / removing "normal" participants.

[0144] It is also possible to specify the available or applicable data types for each available participant in an RTC session, as well as which outputs should be presented to which participants and / or what should happen as a result.

[0145] While examples have been described above with reference to interactions between a physician and a subject (e.g., a patient), it should be understood that the proposed concepts may be applicable to other interactions. For example, embodiments may relate to a home monitoring application in which family members request functionality provided by one or more algorithms to measure vital signs. Other embodiments may relate to work / office-related meetings in which participants may request functionality provided by one or more algorithms to automatically create documents, create meeting minutes, and establish voting results. 11 illustrates an example of a computer 900 on which one or more portions of an embodiment can be used. The various operations described above can utilize the capabilities of the computer 900. For example, one or more portions of the system for providing a subject-specific user interface may be incorporated into any of the elements, modules, applications, and / or components described herein. In this regard, it should be understood that the system functional blocks can be executed on a single computer or distributed over several computers and locations (e.g., connected via the Internet).

[0146] The computer 900 may include, but is not limited to, a PC, a workstation, a laptop, a PDA, a palm device, a server, storage, etc. Generally, with respect to hardware architecture, the computer 900 may include one or more processors 910, memory 920, and one or more I / O devices 970 communicatively coupled via a local interface (not shown). The local interface may be, for example, but not limited to, one or more buses or other wired or wireless connections, as known in the art. The local interface may have additional elements, such as controllers, buffers (caches), drivers, repeaters, receivers, etc., to enable communication. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0147] The processor 910 is a hardware device for executing software that may be stored in the memory 920. The processor 910 may be virtually any custom or commercially available processor, central processing unit (CPU), digital signal processor (DSP), or coprocessor of several types associated with the computer 900, and the processor 910 may be a semiconductor-based microprocessor (in the form of a microchip) or microprocessor.

[0148] Memory 920 can include any one or combination of non-volatile memory elements (e.g., random access memory such as random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), tape, compact hard disk read-only memory (CD-ROM), hard disk, diskette, cartridge, cassette, etc.). Furthermore, memory 920 can incorporate electronic, magnetic, optical, and / or other types of storage media. It should be noted that memory 920 can have a distributed architecture, where various components are located remotely from each other but can be accessed by processor 910.

[0149] The software in memory 920 may include one or more separate programs, each including an ordered list of executable instructions for implementing logical functions. The software in memory 920 includes, according to an exemplary embodiment, a suitable operating system (O / S) 950, a compiler 940, source code 930, and one or more applications 960. As shown, application 960 comprises numerous functional components for performing the features and operations of the exemplary embodiments. The application 960 of computer 900 may represent various applications, computational units, logic, functional units, processes, operations, virtual entities, and / or modules according to an exemplary embodiment, although application 960 is not meant to be limiting.

[0150] Operating system 950 controls the execution of other computer programs and provides scheduling, input / output control, file and data management, memory management, and communication control and related services. It is contemplated by the inventors that application 960 for implementing exemplary embodiments may be applicable to all commercially available operating systems.

[0151] The application 960 may be a source program, an executable program (object code), a script, or any other entity comprising a set of instructions to be executed. In the case of a source program, the program is typically translated via a compiler (such as compiler 940), assembler, interpreter, etc., which may or may not be contained within memory 920 so that it operates properly in conjunction with the O / S 950. Furthermore, the application 960 may be written in an object-oriented programming language, which has classes of data and methods, or a procedural programming language, such as, but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP script, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc.

[0152] The I / O devices 970 may include input devices such as, but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. Additionally, the I / O devices 970 may include output devices such as, but not limited to, a printer, display, etc. Finally, the I / O devices 970 may further include devices that communicate both input and output, such as, but not limited to, network interface cards or modulators / demodulators (for accessing remote devices, other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. The I / O devices 970 also include components for communicating over various networks, such as the Internet or an intranet.

[0153] If computer 900 is a PC, workstation, or other intelligent device, the software in memory 920 may also include a basic input / output system (BIOS) (omitted for simplicity). The BIOS is a set of essential software routines that initializes and tests hardware at startup, starts O / S 950, and supports data transfers between hardware devices. The BIOS is stored in some type of read-only memory, such as ROM, PROM, EPROM, EEPROM, etc., so that the BIOS can be executed when computer 900 is started.

[0154] When computer 900 is operating, processor 910 is configured to execute software stored in memory 920, communicate results to and from memory 920, and generally control the operation of computer 900 in accordance with the software. Applications 960 and O / S 950 are read in whole or in part by processor 910, possibly buffered within processor 910, and then executed.

[0155] It should be noted that when application 960 is implemented in software, application 960 can be stored on virtually any computer-readable medium for use by or in connection with any computer-related system or method. In the context of this specification, a computer-readable medium can be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer-related system or method.

[0156] The application 960 may be embodied in any computer-readable medium for use by or associated with an instruction execution system, apparatus, or device, such as a computer-based system, a processor-containing system, or other system that can fetch instructions from and execute instructions from the instruction execution system, apparatus, or device. In the context of this specification, a "computer-readable medium" may be any means that can store, communicate, propagate, or transport a program for use by or associated with an instruction execution system, apparatus, or device. The computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0157] The present invention may be a system, a method and / or a computer program product. The computer program may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform the method of the present invention.

[0158] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories, read-only memories, erasable programmable read-only memories, static random access memories, portable compact disk read-only memories, digital versatile disks, memory sticks, floppy disks, mechanically encoded devices such as punch cards, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as being a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0159] The computer-readable program instructions described herein can be downloaded to each computing / processing device from a computer-readable storage medium or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0160] The computer-readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter situation, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0161] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0162] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored therein comprises a product containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0163] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0164] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0165] The description has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the invention in the form disclosed. Numerous modifications and variations will be apparent to those skilled in the art. The embodiments have been chosen and described in order to best explain the principles of the proposed embodiments and to enable others skilled in the art to understand various embodiments with various modifications. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim, and the word "a" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements and / or by means of a suitably programmed processor. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. Measures recited in mutually different dependent claims may be advantageously used in combination.

Claims

1. A method for dynamically changing a feature of a face-to-face interaction between a first participant and a second participant, comprising: establishing a real-time communication session including one or more data streams between participants of the real-time communication session; identifying a request for a first algorithm of a set of algorithms, each algorithm of the set of algorithms configured to provide an associated function when executed during a real-time communication session; dynamically modifying a capability of the real-time communication session by adding the first algorithm to the real-time communication session as a virtual participant using at least one data stream of the one or more data streams; allows dynamic changes to the capabilities of a real-time communication session, the type of data processed by the algorithm is at least one of audio data, video data, and measurement data for at least one of clinical support assessment, medical diagnosis, and support planning; The method, wherein the real-time communication session is established to provide at least one feature to participants in the face-to-face interaction.

2. adding the first algorithm to the real-time communication session as a virtual participant; Invoking the first algorithm via a container; configuring, by the container, the first algorithm as a virtual participant in the real-time communication session; 2. The method of claim 1, comprising:

3. The method of claim 2 , wherein invoking the first algorithm comprises communicating with the container via an interface.

4. the request for a first algorithm comprises an indication of a requirement of the first algorithm; adding the first algorithm to the real-time communication session as a virtual participant; configuring at least one of an input and an output of the first algorithm based on the indication of requirements of the first algorithm.

4. The method according to any one of claims 1 to 3.

5. each algorithm of the set of algorithms is associated with a respective wrapper that defines at least one of the inputs, outputs, and requirements of the algorithm; adding the first algorithm to the real-time communication session as a virtual participant; configuring at least one of an input and an output of the first algorithm based on the wrapper associated with the first algorithm.

5. The method according to any one of claims 1 to 4.

6. the step of identifying the request comprises: a user input signal provided by a participant of said real-time communication session; a data stream context for said real-time communication session; the occurrence of a predetermined trigger event; and the occurrence of a non-predetermined trigger event; determining the requirements of the first algorithm based on at least one of:

6. The method according to any one of claims 1 to 5.

7. The method of claim 1 , wherein establishing a real-time communication session comprises adding the first participant and the second participant of the face-to-face interaction as virtual participants in the real-time communication session.

8. Establishing a real-time communication session adding the first participant of the face-to-face interaction as a first virtual participant of the real-time communication session; and adding the second participant of the face-to-face interaction as a second virtual participant of the real-time communication session; 10. The method of claim 1, comprising:

9. The identification of the request a user input signal provided by a participant in the face-to-face interaction; a context of the data stream of the face-to-face interaction; the occurrence of a predetermined trigger event in the face-to-face interaction; and the occurrence of a non-predetermined trigger event in the face-to-face interaction; 9. The method of claim 1, wherein the first algorithm requirement is determined based on at least one of:

10. 10. The method of claim 1, wherein participants are medical professionals and the real-time communication session is configured for the exchange of clinical information for clinical decision support.

11. 11. The method of any one of claims 1 to 10, wherein at least one algorithm of the set of algorithms, when executed, is configured to store data in and / or retrieve data from an electronic medical record database.

12. A computer program product for dynamically modifying the capabilities of a real-time communication session, comprising computer readable program code configured to perform all the steps of the method according to any one of claims 1 to 11.

13. A system for dynamically modifying functionality of a face-to-face interaction between a first participant and a second participant, the system comprising one or more processors configured to perform all of the steps of the method of any one of claims 1 to 11, the one or more processors comprising: a communication component configured to establish a real-time communication session including one or more data streams between participants of the real-time communication session; a processing component configured to identify a request for a first algorithm of a set of algorithms, each algorithm of the set of algorithms configured to provide an associated function when executed during a real-time communication session; and a modification component configured to dynamically modify functionality of the real-time communication session by adding the first algorithm to the real-time communication session as a virtual participant using at least one data stream of the one or more data streams; configured to form the type of data processed by the algorithm is at least one of audio data, video data, and measurement data for at least one of clinical support assessment, medical diagnosis, and support planning; The system, wherein the communication component is configured to establish a real-time communication session to provide at least one feature to participants of the face-to-face interaction.

Images