A system for recording interaction between at least two parties in a session

Abstract

A system for recording interaction between at least two parties in a session comprises a communication module configured to send and receive information representing the interaction between the parties during the session, - at least two user interfaces configured to present the information representing the interaction between the parties, an information storage, and a processing module connected to the communication module, user interfaces, and the information storage. The processing module is configured to store the information representing the interaction between the parties together with time information to create a seamless, synchronized time line.

Description

[0001] A SYSTEM FOR RECORDING INTERACTION BETWEEN AT LEAST TWO PARTIES IN A SESSION

[0002] The invention regards a system and method for interacting between at least two parties located remote to each other. More specifically, the invention regards a system and method where the parties are in a teaching situation, for example one being a teacher or a representation of teaching material, and at least one being a student.

[0003] Today there are many platforms for interacting remotely between two parties, where several of these have been utilized for teaching situations, even though most of them are designed for meetings.

[0004] A problem with such prior art systems is that they do not ensure that students physically present and those who sit at home (participate digitally) receive the same follow-up from the teacher. This makes the lessons much less inspiring and contributes to students being distracted, and in the worst case dropping out.

[0005] Another problem is that most education and health software don't talk to each other.

[0006] For an optimal user experience, there is a need to make it possible for teachers and students to interact across faculties and departments.

[0007] US11128483 describes a system and method for providing a meeting record to a participant of the meeting. Events that occur during the meeting are stored in a timeline so that the participants have access to the events both during and after the meeting. The record is the same for all participants. Events in this context are for example shared screen during the meeting, a specific presentation starts, or which participants are talking and discussing. Features that are described provide an overview of who is in the meeting, relevant summary and highlights driven by transcription, as well as controls to go back and see earlier detail from the meeting.

[0008] US2022327309 describes a system and method for translating sign language content into word content and vice versa.

[0009] There is thus a need for software that makes it possible to talk to other software in a synchronous timeline, which is much more user-friendly for the education sector and the health service.

[0010] The object of the invention is to provide a system for recording interaction between at least two parties in a session that eliminates the problems of prior art.

[0011] The object of the invention is achieved by means of the patent claims.

[0012] In one embodiment, there is provided a system for recording interaction between at least two parties in a session, where the system comprises: - a communication module configured to send and receive information representing the interaction between the parties during the session,

[0013] - at least two user interfaces configured to present the information representing the interaction between the parties,

[0014] - an information storage, and

[0015] - a processing module connected to the communication module, user interfaces, and the information storage, and wherein the processing module is configured to store the information representing the interaction between the parties together with time information.

[0016] The information representing interaction between parties can comprise at least one of voice communication, written text, video, images, sound, symbols.

[0017] The processing module may also be configured to store a note that is made by one party during the session together with the information representing the interaction between the parties. Student id, the associated session id, and note id may be stored together with the text of the note.

[0018] The user interface can be configured to display separate information in separate display windows so that the display windows are represented on the user interface without overlap.

[0019] The processing module can be configured to associate information received from one party with a point in time. The point in time represents the real time point in time when the information was executed.

[0020] The system may further comprise a sign language module, configured to translate voice text to sign language during a session and display a representation of the sign language to at least one of the parties on the at least one user interface. This may for example be done by means of an Al module and / or machine learning. The representation of sign language can for example be in the form of an avatar or other figure mimicking or otherwise representing the individual signs that constitutes the sign language translation.

[0021] The communication module can be configured to fetch information from the information storage, dependent on input from one of the user interfaces.

[0022] The processing module is configured to store information representing the start of the session and the end of a session together with time information, and to associate a session ID with the information stored in the time period between the start of the session and the end of the session.

[0023] The processing module may also be configured to create timeline and also a visualization of the timeline of the session, comprising a visualization of the start of the session, the end of the session and all information stored during the session. By using the system in a teaching context, the user experience will be more optimal compared to prior art solutions. By using the archive / storage functionality, users will more easily be able to find back to important points in the teaching and notes they have made at the current times. Users can receive targeted guidance based on their activity and presence through the system.

[0024] Examples of possible functions and interaction with the archive / storage:

[0025] If a user has a question during a lesson, this can be flagged through the question function. If the archive module is enabled, the time the question was asked will be saved and appear as a flag on the timeline in the archive. If the question was in text form, you will also be able to see the question and any answer that was given in the archive.

[0026] When storing notes, these will also be flagged on the timeline in the archive module, as well as stored associated with the teaching so that the user can easily retrieve notes based on teaching afterwards.

[0027] The user can choose to manually add a flag to the timeline and link, for example, a search query in Google Translate function to the toolbox of this flag.

[0028] Another possibility is logging of answers and results through a quiz functionality, ie, integrating a quiz in the session to check if the user has learned the required lesson and the user will be able to see if they need further follow-up or repetition.

[0029] If consent has been given for the storage of data, users can also look at the individual's activity in the teaching based on relevant questions asked and quiz results. One can look at trends based on this data to draw conclusions about whether the user is at risk of dropping out.

[0030] The invention will now be described by means of an example and by reference to the accompanying figures.

[0031] Figure 1 illustrates an overview of a system according to the invention.

[0032] Figure 2 illustrates a workflow for sharing content.

[0033] Figure 3 illustrates a workflow for creating or updating notes.

[0034] Figure 4 illustrates a workflow for creating a time line.

[0035] Figure 1 illustrates schematically a system 1 for recording interaction between at least two parties A, B. The system comprises a communication module 12 configured to send and receive information representing the interaction between the parties. The information representing interaction between parties can for example be one or more of voice communication, written text, video, images, sound, symbols. The two parties in the illustrated embodiment, have each a user interface 14, 15 configured to present the information representing the interaction between the parties. The user interface may comprise a display device, speakers, etc., for ie. showing video images, still images, play sound, etc.

[0036] The user interfaces can be configured to display separate information in separate display windows in such a way that the display windows are represented on the user interface without overlap. For example, can the user interface show a video stream of the interacting parties, such as a teacher and a student, and simultaneously show pictures illustrating the topic the teacher is talking about. The user interface may then show the video streams side by side and the pictures beside, above or below the video streams, thereby giving the parties A, B access to all the information simultaneously.

[0037] The system further comprises an information storage 10, illustrated here as a cloud, ie. the information storage can be a remote storage / server that can communicate with the user interfaces 14, 15 of the two parties.

[0038] A processing module 11 is connected to the communication module 12, user interfaces 14, 15, and the information storage 10, and is configured to store the information representing the interaction between the parties together with time information representing the time where the interaction occurred.

[0039] The processing module can also be configured to store information representing the start of the session and the end of a session together with time information, and to associate a session ID with the information stored in the time period between the start of the session and the end of the session.

[0040] Further, each party can submit information, for example a text message, a symbol, etc. and the processing module can thus be configured to associate information received from the relevant party with a point in time where the information was received.

[0041] In one embodiment, the system can comprise a sign language module 13, configured to translate voice text to sign language during a session and display a representation of the sign language to at least one of the parties on the at least one user interface. This can be implemented as an artificial intelligence unit (Al), with machine learning abilities.

[0042] The following describes an example of the progression of a session.

[0043] In the description of the example, the term time stamp shall mean time information representing a point in time when an interaction happens.

[0044] 1. The session will be scheduled by an organizer. The organizer may or may not be one of the parties to interact during the session. The processing module 11 will then create a session ID and an activity table with scheduled start time and store the activity table in the information storage.

[0045] 2. The scheduled time has come, and the session is started in step at a specific point of time.

[0046] The start session is associated with session id and stored in the activity table in the information storage 10 with a start timestamp, ie. time information representing the actual starting time.

[0047] 3. The Organizer then starts webcam sharing, ie. a video stream.

[0048] The start webcam activity is saved together with a timestamp in the activity table.

[0049] 4. The Organizer will share their screen.

[0050] Start screen sharing activity is created and saved in the activity table together with a timestamp.

[0051] 5. Organizer stops sharing of screen.

[0052] The activity table will be updated with stop time for the screen sharing feature and information about the video file saved. A stop activity is added to the activity table.

[0053] Figure 2 illustrates a workflow for detecting and storing information of a shared video stream / di splay by one of the parties attending a session.

[0054] 6. Session is finished.

[0055] The Organizer stops webcam sharing. The end webcam activity is saved in the activity table together with information representing the webcam stream. A stop activity is created and stored in the activity table.

[0056] 7. Session closes.

[0057] A stop / end time is stored in the information storage. If there are any remaining activities these will also be ended and will be updated with stop times.

[0058] If a question, note, quiz, etc. is recorded during the session, they will, like the camera and screen, be stored in the information storage, in the activity table, together with a timestamp.

[0059] Figure 3 illustrates the workflow for storing notes that are made during a session. In this example, one user / party presses a button to activate storage of the event using the user interface, for example a keyboard key, touching a touch screen, etc. In other embodiment, the system may be set up to automatically store all or a selection of events. The information that is stored in the information storage in this example is student id, the associated session id, institute id (the teaching institute), text of the note, and note id.

[0060] As the session continues, the student may add or amend the notes, and the system continuously updates the note until the session is ended. The note may be stored separately and / or sent to the student as e-mail, but will always be part of the session for the student to read later.

[0061] When the active session is ended it will be made available with all activities linked in the archive, as a timeline. The timeline is built based on the session start and stop timestamps and all linked activities. This is achieved by means of the communication module that is configured to fetch information from the information storage, dependent on input from one of the user interfaces. For example, a user wants to review a specific session, this session with its session ID can be searched for and brought up on the user interface. Figure 4 illustrates an example of this process.

[0062] To this effect, the processing module is configured to create a visualization of the timeline of the session, comprising a visualization of the start of the session, the end of the session and all information stored during the session.

[0063] Activities are placed on the timeline based on their timestamp relative to the start / stop of the session. Such activities include but are not limited to: recording of shared tCam / screen, question, note saved, and quiz. When "searching" the timeline (dragging current time) the correct shared screen / tCam video clip will be selected based on relative timestamp.

Claims

CLAIMS1. A system for recording interaction between at least two parties in a session, where the system comprises:- a communication module configured to send and receive information representing the interaction between the parties during the session,- at least two user interfaces configured to present the information representing the interaction between the parties,- an information storage, and- a processing module connected to the communication module, user interfaces, and the information storage, and wherein the processing module is configured to store the information representing the interaction between the parties together with time information.

2. The system according to claim 1, where the processing module is configured to store a note that is made by one party during the session, together with the information representing the interaction between the parties.

3. The system according to claim 1 or 2, where the information representing interaction between parties comprises at least one of voice communication, written text, video, images, sound, symbols.

4. The system according to one of the previous claims, where the user interface is configured to display separate information in separate display windows, where the display windows are represented on the user interface without overlap.

5. The system according to one of the previous claims, where the processing module is configured to associate information received from one party with a point in time, where the point in time represents the real time point in time when the information was executed.

6. The system according to one of the previous claims, where the system comprises a sign language module, configured to translate voice text to sign language during a session and display a representation of the sign language to at least one of the parties on the at least one user interface.

7. The system according to one of the previous claims, where the communication module is configured to fetch information from the information storage, dependent on input from one of the user interfaces.

8. The system according to one of the previous claims, where the processing module is configured to store information representing the start of the session and the end of a session together with time information, and to associate a session ID with theinformation stored in the time period between the start of the session and the end of the session.

9. The system according to one of the previous claims, where the processing module is configured to create a visualization of the timeline of the session, comprising a visualization of the start of the session, the end of the session and all information stored during the session.

10. The system according to claim 2, where the processing module is configured to store student id, the associated session id, and note id together with the text of the note.

Images