System and method for generating a video representation of a file

Abstract

A computer implementation for managing files is disclosed. This method includes rendering at least a portion of a file as an image by opening the file (E110) using a rendering function (610), and obtaining one or more event messages (E120). For each of the one or more event messages, the method includes: obtaining an image corresponding to each event message from the rendering function (610) (E130), where the image represents the rendered portion of the file and is associated with a sequence instruction, generating metadata based on the event message and sequence instruction (E140), providing the metadata (E150), and providing a video containing a set of images (E160), where the set of images contains an image corresponding to each event message, and the video is a representation of the file.

Description

Technical Field

[0001] At least one embodiment of the present specification relates to a video communication system such as a video conferencing system, a video collaboration system, a video communication service, a collaboration service, etc. The video communication system can facilitate collaboration among a number of participants through a collaboration session. Further, at least some embodiments relate to the generation of videos such as video streams, video files, etc.

[0002] In a plurality of aspects, a system for managing content and / or files, related methods, and one or more related software functions are provided. Also provided are systems and methods for generating videos.

[0003] In particular, according to some aspects of the present specification, a method for managing the sharing of videos in a collaboration session with at least two client devices is provided. Further, according to a further aspect of the present specification, a method for managing files, for example, a method for converting files into videos, is provided.

Background Art

[0004] Many digital video conferencing systems are known, such as Microsoft (registered trademark) Teams (registered trademark), Zoom (registered trademark), Google (registered trademark) Meet (registered trademark), etc., where two or more participants can virtually hold a meeting using locally recorded digital videos and audio and broadcast it to all participants to emulate a physical meeting.

[0005] Regarding such digital video conferencing solutions, especially with respect to the quality of the videos shared by participants within a meeting, there is a general need for improvement. Quality may refer to resolution, frame rate, stuttering, time lag, etc.

[0006] It may be desirable to generate one or more output digital video streams by an automated generation process based on multiple input digital video streams, and then provide these generated digital video streams to one or more consumption entities. [Disclosure of the Invention] [Problems that the invention aims to solve]

[0007] A common scenario involves two participants connecting to a publicly known video communication service and collaborating within a session. In such sessions, participants can relatively easily share content displayed on their respective devices, such as computers. However, when sharing video content, lag, interruptions, and other degradations of the video / audio experience can occur. These types of problems can be frustrating for participants, and the video communication service may be perceived as lacking user-friendliness and having performance issues.

[0008] Another issue is that in an ongoing collaboration session, there may be a delay between the time a user requests to share content such as video files, presentation files, document files, or spreadsheet files, and the time when that content is actually shared among participants, such as their devices.

[0009] Furthermore, there is a general need for improvement in such digital video conferencing systems, particularly regarding the creation of viewable content—when, to whom, what, and through which distribution channel it is displayed.

[0010] For example, some systems automatically detect the participant currently speaking and display their corresponding video feed to other participants. Many systems allow sharing of graphics such as the currently displayed screen, reading pane, and digital presentations. However, as virtual meetings become more complex, it is becoming increasingly difficult for services to determine which information should be displayed to which participant at a given time from among all available information.

[0011] In another example, a participant presenting might move around the stage while explaining slides from their digital presentation. In this case, the system needs to decide whether to display only the presentation, only the presenter, both, or switch between the two. In many other situations, the system may make a variety of automated generation decisions about how to structure a shared video stream consumed by one or more participants and / or external entities.

[0012] Therefore, it is desirable that an automated creation process automatically generates one or more output digital video streams based on multiple input digital video streams and provides the generated digital video streams to one or more consuming entities.

[0013] However, in many cases, dynamic meeting screen layout and other automated creation features make it difficult to choose what information to display due to the many technical challenges faced by such digital video conferencing systems.

[0014] The object of the present invention is to eliminate, or at least reduce, one or more of the above-mentioned drawbacks and / or problems. [Means for solving the problem]

[0015] This objective, and other objectives, are achieved by one or more embodiments described herein.

[0016] According to the first aspect, a method is provided which is performed by the system in accordance with the first item of the attached exemplary list according to the first aspect.

[0017] For example, because the system retrieves video frames iteratively, for instance, frame by frame, from a storage function, the shared video does not need to depend on connections to client devices participating in a collaboration session. Instead, retrieval depends on connections to a storage function, such as cloud storage. Because cloud storage, i.e., storage functions, are designed to be always available, such connections are generally reliable and have high capacity in terms of bandwidth, etc. Furthermore, video frames, for example, individual video frames, are iteratively retrieved from the storage function and then processed (e.g., generating and providing video frames to be sent to a video communication service as a shared video stream). As a result, the memory requirements of the system, for example, a central server, are reduced compared to, for example, if the entire file representing the video had to be stored and managed on the central server. In at least one embodiment of this specification, video frames are processed one by one, and typically only a limited number of video frames need to be held in the system's memory at any given time.

[0018] Furthermore, as a result of iteratively acquiring video frames, processing of a frame can begin as soon as at least one complete video frame is acquired (e.g., received and / or decoded). This reduces video start delays because the system does not have to wait for the entire file, or even just a chunk of the file, to be received.

[0019] Therefore, users participating in collaboration sessions via client devices have the advantage of being able to view videos shared through video communication services with little to no stuttering or delay. This improves user-friendliness.

[0020] Furthermore, according to some embodiments, the shared video has the advantage of being provided at a desired resolution and frame rate. For example, as will be described in the detailed description, the desired frame rate and resolution can be adapted to the client device (e.g., the technical capabilities of the client device and / or the level of the service contract).

[0021] In addition, various embodiments of the first embodiment are described below.

[0022] A method is provided which is performed by a system for managing the sharing of video in a collaboration session in which at least two client devices are involved, wherein the system includes the two client devices and a central server capable of hosting the collaboration session, the video files representing the video and stored in a storage function are accessible from the central server, the video is shared as a shared video stream having a desired resolution and a desired frame rate, and the method includes: - The central server receives a request to share the video within the collaboration session; - The central server obtains the synchronization timestamp of the audio frame associated with the collaboration session, which is used to synchronize the shared video stream to the collaboration session; - The central server determines, based on the video frame rate difference between the desired frame rate and the source frame rate of the video file, whether the shared video stream requires more, fewer, or the same number of video frames as the video file; - The central server sets up a buffer for providing the shared video stream to the collaboration session, wherein the buffer is capable of buffering a limited number of video frames corresponding to time intervals of less than one second at most; - The central server repeatedly obtains video frames of the video file for each video frame, and, For each video frame, the method includes the following: - Based on each repeatedly obtained video frame by the central server, according to the determination step, generate zero, one or multiple video frames, and set the respective timestamps of the one or more generated video frames based on the synchronization timestamp. Here, the one or more generated frames have the desired resolution, and each timestamp corresponds to the desired frame rate, Here, the method includes the following: - By the central server, insert the one or more generated video frames into the buffer for the two client devices participating in the collaboration session, thereby providing the shared video stream.

[0023] In some embodiments, the method includes the following: - The central server obtains information regarding the desired resolution and the desired frame rate of the shared video stream representing the shared video.

[0024] In some embodiments, the method includes the following: - In a further determination step, the central server determines whether more, fewer, or the same number of video frames are required in the shared video stream than in the video file based on a further video frame rate difference between a further desired frame rate and the source frame rate of the video file. Here, the method includes the following for each repeatedly obtained video frame: - In a further generation step, the central server generates zero, one or more video frames according to the further determination step based on each of the repeatedly acquired video frames, and sets a respective further timestamp of the one or more generated video frames based on the synchronization timestamp, wherein the one or more generated frames have a further desired resolution and each timestamp corresponds to a further desired frame rate. - The central server provides the second video stream by transmitting each of the further video frames to a destination function.

[0025] In some embodiments, the shared video stream may be processed by a generation function to obtain a generated shared video stream, wherein the second video stream is processed by a further generation function to obtain a generated second video stream.

[0026] In some examples, the method includes, for example, before acquisition of video frames, after generation of one or more generated video frames, or at a similar timing, the following: - The generation function processes the video frames to obtain a generated shared video stream.

[0027] In some examples, the method includes, for example, before acquisition of further video frames, after generation of yet one or more generated video frames, or at a similar timing, the following: - A further generation function processes the video frames to obtain a generated shared video stream.

[0028] The generation function and the further generation function may be different from each other.

[0029] In some embodiments, the method includes the following: - The central server receives an update message instructing it to set the desired video frame rate and the desired resolution based on the information contained in the message, thereby obtaining the updated desired video frame rate and the updated desired resolution.

[0030] According to a second aspect, a method according to claim 1, performed by a system, is provided.

[0031] The inventors have found that a contributing factor to time lag and / or stuttering is the optimization of client software for connecting to video communication services. These optimizations may stem from the fact that users initially commonly shared static content such as documents, spreadsheets, and presentations with slides. To ensure good operation even under demanding conditions such as high latency and low bandwidth, known client software provides screen sharing functionality with limited frame rates and resolutions. Therefore, a simple solution would be to upgrade the client software to add features that allow higher frame rates or higher resolutions, for example, in a user-selectable manner. However, this often has a catastrophic impact on performance when sharing static content in demanding connection environments such as poor network conditions and low bandwidth.

[0032] Instead of updating client software, the inventors provide a solution in which a central server connects to a video communication service and shares video from a storage function with a highly reliable connection at, for example, any desired frame rate and resolution. This method enables video sharing in a user-friendly manner with little to no interruption or lag. At the same time, it does not require a trade-off between optimizing static content such as slides and documents and video. Advantageously, in at least some embodiments, the client software remains the same, and only a plugin, add-on, or similar function for the client software and a backend that works with the add-on are required. Furthermore, in at least some embodiments, the operation and functionality are completely transparent to the user, making the solution highly user-friendly. For example, the user does not need to indicate to the client software or add-on whether a video or a document is being shared.

[0033] Furthermore, various embodiments of the second aspect are described below. A method is provided, performed by the system, for managing the sharing of video in a collaboration session involving at least two client devices, the system including a video communication service capable of hosting the collaboration session, two client functions executable on each of the two client devices, two add-on functions associated with each of the two client functions, an add-on backend function configured to interact with and communicate with the two add-on functions, and a sharing function configured to interact with and communicate with the add-on backend function to participate in the collaboration session. The two client functions may be configured to communicate and / or interact with the video communication service. Video files stored in the storage function and representing the video are accessible by the sharing function, where the video is shared as a shared video stream having a desired resolution and a desired frame rate. The method includes: - The add-on backend functionality receives requests to share the video in the collaboration session; - Connect the sharing function to the collaboration session; - The sharing function obtains the synchronization timestamp of the audio frame of the collaboration session, where the audio frame is used to synchronize the shared video stream with the collaboration session. In the determination step, the sharing function determines whether the shared video stream requires more, fewer, or the same number of video frames as the video file, based on the video frame rate difference between the desired frame rate and the source frame rate of the video file. - The sharing function allows for the iterative acquisition of video frames from the video file stored in the storage function, and For each video frame acquired iteratively, the method includes the following: - In accordance with the determination step, 0, 1 or more video frames are generated based on each video frame acquired iteratively, and the timestamp of each of the generated 1 or more video frames is set based on the synchronization timestamp, where the generated 1 or more frames have the desired resolution, and each timestamp corresponds to the desired frame rate. Here, the method includes the following: - The sharing function provides the shared video stream by transmitting the one or more generated video frames to the video communication service; and, - The video communication service provides the shared video stream to the two client devices participating in the collaboration session.

[0034] In some embodiments, the method includes the following: - The central server obtains information regarding the desired resolution and the desired frame rate of the shared video stream representing the shared video. Furthermore, a sharing function is provided which is configured to perform at least a portion of the operations performed by the sharing function in the method described above.

[0035] A method is provided, according to a third aspect, that is performed by the system in accordance with the first item of the attached list of examples according to the third aspect.

[0036] In this method, the video is linked to metadata associated with one or more points in the video. This makes it possible to fast forward (jump, skip, etc.) to a specific point in the video to which the metadata is associated. The metadata can represent events such as mouse clicks or slide changes. This means, for example, that it is easy to find a specific point in the video where the event occurred, and that a specific page or slide from the source file appears in the video at that specific point.

[0037] In the context of video editing and / or creation, there is an advantage in that the video portion representing the content of a file can be easily found, for example, by automated generation functions.

[0038] In further embodiments, even without using metadata, there is the advantage of being able to share and deliver at least partially static content, such as websites and document files, as video supplied to a video communication service to participants. This allows content delivery to participants in a collaboration session to rely on and leverage the advantages of the video communication service. In contrast, when using Teams, if a web page is shared in a collaboration session using so-called "stage view," participants' devices load the web page and its content, such as video, directly from the web server hosting the web page. As a result, the web page content is served from the web server to the stage view, effectively and unfortunately bypassing the video communication service. Consequently, such sharing may not take advantage of any of the benefits offered by video communication services, such as their video content delivery capabilities.

[0039] Furthermore, various embodiments of the third aspect are described below.

[0040] A computer implementation method for managing files is provided, and the method includes the following: - Open the file using the rendering function and render at least a portion of the file as an image; - Retrieve one or more event messages. Here, for each of the one or more event messages, the method includes the following: - The rendering function retrieves the respective images corresponding to each event message, where each image represents the rendered portion of the file, and each image is associated with a sequence instruction; - Generate metadata based on the event message and the sequence instruction; - Provide the aforementioned metadata; and, - A video is supplied which includes a set of images, wherein the set of images includes an image corresponding to each event message, and the video is a representation of the file.

[0041] In some embodiments, the method is performed by a client device, the system includes the client device and a video communication service, the video communication service hosts a collaboration session in which the client device participates.

[0042] In some embodiments, the one or more event messages are received from the rendering function and relate to changes in the image rendered by the rendering function, resulting from input events that can be provided, for example, by a user input device.

[0043] In some embodiments, the method includes the following: The one or more event messages are simulated by generating input events that modify the image rendered by the rendering function.

[0044] In some embodiments, input events arise from actual interaction with a rendered file by a user using a user input device such as a mouse, keyboard, or touchpad.

[0045] In some embodiments, the method includes the following: - The rendering function iteratively captures images; and, - When a difference is detected between two consecutively captured images, one or more event messages are generated, where the event messages indicate that the two consecutively captured images are different from each other.

[0046] In some embodiments, the method includes the following: - The rendering function iteratively captures images; and, - If no difference is detected between two consecutively captured images, an event message is generated for one or more event messages, where the event message indicates that the two consecutively captured images are similar to each other.

[0047] In some embodiments, the one or more event messages include an instruction to start repeated image capture by the rendering function, or an instruction to stop repeated image capture by the rendering function. The method includes the following: - When a start command is received, the rendering function is used to repeatedly capture images. - Until a stop command is received, metadata is generated for each captured image based on the event message and the corresponding sequence command.

[0048] In some embodiments, the method includes the following: - Receive the audio stream of the collaboration session from the video communication service; - Detect keywords within the audio stream; and, - Based on the keywords, generate event messages for one or more event messages.

[0049] A computing function is provided which is configured to perform at least some of the operations performed by the computing function disclosed herein.

[0050] Furthermore, the aforementioned objectives, or any other objectives, are achieved by computer programs and computer program carriers corresponding to the embodiments described above.

[0051] Various aspects of the embodiments disclosed herein, in particular their features and advantages, will be readily apparent from the following detailed description and accompanying drawings. These are briefly described below. [Brief explanation of the drawing]

[0052] [Figure 1] Figure 1 is a schematic diagram illustrating an exemplary system according to various embodiments of this specification. [Figure 2] Figure 1 is a schematic diagram illustrating an exemplary system according to various embodiments of this specification. [Figure 3a] Figure 3a is a schematic diagram illustrating an exemplary system according to various embodiments of this specification. [Figure 3b] Figure 3b is a schematic diagram illustrating an exemplary system according to various embodiments of this specification. [Figure 4] Figure 4 is a combined signal processing diagram and flowchart illustrating exemplary embodiments described herein. [Figure 5] Figure 5 is a combined signal processing diagram and flowchart illustrating exemplary embodiments described herein. [Figure 6] Figure 6 is a combined signal processing diagram and flowchart illustrating exemplary embodiments described herein. [Figure 7] Figure 7 is an illustrative flowchart illustrating an exemplary method described herein. [Figure 8] Figure 8 is a timeline showing the establishment of timestamps for video frames. [Figure 9] Figure 9 is a flowchart illustrating an exemplary method as described herein. [Figure 10] Figure 10 is a combined signal processing diagram and flowchart illustrating an exemplary method described herein. [Figure 11]Figure 11 is a flowchart illustrating an exemplary method in this specification. [Figure 12] Figure 12 is a block diagram illustrating an exemplary system. [Figure 13] Figure 13 is a block diagram illustrating exemplary embodiments in this specification. [Figure 14] Figure 14 is a block diagram illustrating exemplary embodiments of this specification. [Figure 15] Figure 15 is a block diagram illustrating exemplary embodiments of this specification. [Figure 16] Figure 16 is a block diagram illustrating exemplary embodiments in this specification. [Figure 17] Figure 17 is a block diagram illustrating exemplary embodiments in this specification. [Modes for carrying out the invention]

[0053] Throughout the following description, similar reference numerals are used to indicate similar features such as function, operation, module, circuit, component, item, element, and unit, where applicable. In the drawings, features appearing in some embodiments are sometimes indicated by dashed lines.

[0054] Definitions of terms: In this specification, the terms “central server” or “function” refer to the functionality of a computer implementation configured to be accessed in a logically centralized manner, such as through a clearly defined API (Application Programming Interface). Such central server functionality may be implemented purely in computer software, or in combination with software and virtual and / or physical hardware. It may be implemented on a single physical or virtual server computer, or it may be distributed across multiple interconnected physical and / or virtual server computers. The physical or virtual hardware on which the central server functionality, i.e., the computer software defining the central server functionality, runs may itself include conventional CPUs, conventional GPUs, conventional RAM / ROM memory, conventional computer buses, and conventional external communication functions such as internet connectivity.

[0055] In this specification, the term "video communication services" refers to video conferencing services, video communication functions, and collaboration services such as Microsoft Teams, Google Meet, Skype, and Slack. The rights to the names of these services belong to their respective registered owners. Such video communication services can be provided by hosting, conducting, or otherwise providing collaboration sessions such as video conferencing meetings and online meetings.

[0056] In this specification, the terms “video communication” or “collaboration session” may be used interchangeably to refer to an interactive digital communication session comprising at least two, preferably at least three, or at least four or more video streams and preferably corresponding audio streams. These streams are used to generate one or more mixed or joint digital video / audio streams, which are consumed by one or more consumers (e.g., participant clients of the type discussed). These consumers may or may not contribute to the video communication via video and / or audio. Such video communication can take place in real time, with or without a certain delay. At least one, preferably at least two, or at least four participants interactively engage in the video communication, providing and consuming video / audio information. A collaboration session may be a video communication, video communication session, video collaboration session, digital video communication, digital meeting, digital video conference, video conference, online video conference, online digital meeting, etc., provided by a video communication service. In many embodiments, a collaboration session is interactive among multiple different participants in that video and / or audio information is delivered bidirectionally between multiple different participants.

[0057] In this specification, terms such as “client device” and “participant client” may refer to mobile phones, user devices, cell phones, personal digital assistants (PDAs) with wireless communication capabilities, smartphones, laptops or personal computers (PCs) with built-in or external mobile broadband modems, tablet PCs with wireless communication capabilities, portable electronic wireless communication devices, or similar devices.

[0058] In this specification, terms such as “video processing software library” may refer to commercially available or privately owned software libraries that provide ready-to-use routines, functions, procedures, etc., for processing video in any appropriate format. Processing may include color conversion, resolution adjustment, frame rate adjustment, brightness adjustment, contrast adjustment, compression, decompression, and format conversion. Examples of software libraries include FFmpeg, VideoLan, Handbrake, PotPlayer, Amazon Web Services (AWS) Elemental MediaTailor, Compressor, Videojs, Amazon Kinesis Video Streams, and Video Converter Ultimate, but existing rights to these belong to their respective owners. Typically, such libraries are clearly defined as technical entities, even if they exist in multiple different versions over time.

[0059] In this specification, the term “shared video stream” may refer to a “shared video stream” transmitted to a video communication service for distribution to one or more participants, such as client devices or virtual clients, in a collaboration session.

[0060] In this specification, the term “generated video stream” may refer to such a “generated video stream” being produced from one or more collected and / or received videos, and may be combined in some cases with audio streams, subtitle information, metadata, and / or similar. Video includes video streams and / or video files, or similar. Generation (creation) may be fully or at least partially automated and performed by the system’s generation function based on available information.

[0061] This means that a shared video stream can be a generated video stream, and a generated video stream can be a shared video stream, but this is not always the case. Furthermore, though less common, a shared video stream does not necessarily have to be a generated video stream. Therefore, the concepts of "shared video stream" and "generated video stream" are orthogonal.

[0062] In this specification, the term “primary stream” (e.g., primary video stream, primary audio stream, etc.) refers to a stream provided to (transmitted to, for example) a generating function. As described herein, a generating function may obtain at least one primary stream, preferably two or more primary streams.

[0063] In this specification, the term “secondary stream” (e.g., secondary video stream, secondary audio stream, etc.) refers to a stream processed or generated by one or more generating functions, etc. As disclosed herein, a generating function may provide (e.g., generate) at least one secondary stream, but in some examples, multiple secondary streams may be generated by a generating function or multiple generating functions, with or without different delays as described herein. Secondary streams can be supplied to a generating function and subsequently take on the role of a primary stream. In other words, generation (creation) can be performed more than one layer or one iteration.

[0064] In this specification, the term “pattern” may refer to a detectable pattern in an audio and / or video stream, and / or a detectable pattern detected in at least two audio and / or video streams (e.g., two simultaneous audios, the same object in a video but at different angles, or similar).

[0065] Figures 1 to 3a and 3b show examples of how system 100 can be implemented according to various configurations. Features of one example can be easily combined with one or more features of other examples, where reasonably possible.

[0066] Figure 1 shows an exemplary system 100 that conforms to at least some aspects of this specification. According to some embodiments, system 100 is configured to perform one or more exemplary methods for managing video sharing in a collaboration session, for example, sharing video as a digital video stream, such as a shared digital video stream delivered within the collaboration session. In other examples, system 100 may be configured to manage files such as document files, presentation files, portable document format (PDF) files, spreadsheet files, web pages, and HTML files. Files may, but do not necessarily, be shared in the collaboration session.

[0067] System 100 may include a central server 130, which may be configured to perform one or more operations of the exemplary methods described herein.

[0068] System 100 may include a video communication service 110, but in some embodiments, the video communication service 110 may be located outside of System 100. Furthermore, the central server 130 may or may not include the video communication service 110. The video communication service 110 is a computer function in the same sense as the central server 130. The central server 130 and the video communication service 110 are configured to communicate with each other, such as exchanging video and / or audio streams, messages, and / or any other kind of information. Therefore, for example, if the central server 130 includes the video communication service 110, the collaboration session may be provided (e.g., by hosting) by the central server 130. If the central server 130 does not include the video communication service 110, the collaboration session is provided (e.g., by hosting) by the video communication service 110.

[0069] System 100 may include one or more participant client devices 121, but in some embodiments, one, some, or all participant client devices 121 may reside outside of System 100. The term “participant client” may refer to participant client devices such as computers, mobile phones, tablet personal computers (PCs), desktop computers, portable computers, and laptops. Each participant client device 121 is a computer function in the same sense as the central server 130, and comprises the physical and / or virtual hardware on which each participant client 121 operates. In other words, the computer software function that defines a participant client 121 may itself include conventional CPU / GPU, conventional RAM / ROM memory, conventional computer bus, and conventional external communication functions such as internet connectivity.

[0070] Each participant client device 121 typically comprises, or communicates with, the following components: one or more computer screens displaying video content provided to the participant client device 121 as part of an ongoing collaboration session; a loudspeaker (configured to output audio content provided to the participant client 121 as part of the video communication); a video camera; and a microphone (configured to locally record the voice content of a human participant 122 to the video communication). Participant 122 uses the participant client 121 to participate in the video communication. Participant 122 may be a user, operator, filmmaker, influencer, conference presenter, teacher, or similar person who is participating in or intending to participate in the collaboration session.

[0071] Typically, participant 122 operates one or more of the client devices 121. At least one of the two client devices 121 may be operated by a user different from the user of the first mentioned client device 121. The first mentioned client device 121 may be referred to as the first client device associated with the first user. At least one of the two client devices 121 may be referred to as the second client device associated with the second user.

[0072] The number of participant client devices 121 used in the same collaboration session may be one, multiple, at least three, or at least four or more. For the purpose of sharing files or videos, it is desirable to have at least two client devices 121. However, in at least some embodiments, it is possible to have no client devices 121 at all, or at least one.

[0073] In some examples, participant client devices may be divided into at least two distinct groups. Each participant client is assigned to such a corresponding group. The groups may reflect different roles of participant clients, different virtual or physical locations of participant clients, and / or different interaction permissions of participant clients. For example, a first group of participant clients may consist of each member of the first group participating remotely in the collaboration session using their own participant client device, while a second group of participant clients may consist of audience members participating in the collaboration session, each participating remotely using a participant client device shared by all audience members, which is configured to display the collaboration session (including video and / or audio) to the audience using one or more display devices, one or more speakers, and / or one or more microphones. Each participant in the second group may also use their own client device instead. Correspondingly, there may be three or more groups.

[0074] The various roles available include, for example, "Leader" or "Moderator," "Speaker," "Presenter," "Panel Participant," "Interactive Audience," or "Remote Listener." Even when these roles are associated with human users, it should be understood that, in order for this embodiment to make roles available, the roles are assigned to client devices, or in some cases, virtual client functions. Roles can form the basis for creating different groups, such as those described above, within a particular collaboration session.

[0075] Available physical locations include, for example, "on stage," "inside the panel," "among the physically present audience," and "among the physically remote audience."

[0076] While virtual locations may be defined based on physical locations, they may also involve virtual groupings that partially overlap with physical locations. For example, a physically present audience can be divided into first and second virtual groups, and some physically present audience members and some physically distant audience members can be grouped together in the same virtual group.

[0077] Examples of the various interaction permissions available include "Full Interaction" (no restrictions), "Speak Only After Microphone Request" (such as virtual or physical hand raising in video conferencing services), "Speak Only but Write to Shared Chat," or "View and / or Listen Only." Interaction permissions apply to a collaboration session or to one or more such collaboration sessions.

[0078] In some cases, each defined role and / or physical / virtual location may be defined based on specific predetermined interaction permissions. Also, all participants with the same interaction permissions may form a group. Therefore, defined roles, locations, and / or interaction permissions reflect various group assignments, and different groups may not interact with each other or may overlap depending on the situation.

[0079] Therefore, for example, a group for a collaboration session may be based on one or more of the following: physical and / or virtual location, interaction permissions, roles, etc., and may correspond to and / or be associated with them.

[0080] As shown in Figure 1, each participant client 121 can constitute the source (transmitter) of its respective information (video and / or audio) stream 10, which is provided from the participant client 121 to the video communication service 110 as described above. Each participant client 121 can also receive a stream 20 containing one or more video and / or audio streams in order to participate in the collaboration session. Stream 10 may be a primary stream, while stream 20 may be a primary or secondary stream.

[0081] System 100 (e.g., Central Server 130) may also be configured to digitally communicate with destination functions 150, such as a virtual director, automated production and / or generation functions, and in particular to transmit digital information. Destination functions 150 may have the ability to edit received video streams and display them to users or viewers. Central Server 130 may or may not have destination functions 150. For example, digital video and / or audio streams created by Central Server 130 may be continuously provided to one or more destination functions 150 in real-time or near-real-time. Again, the fact that destination functions 150 may be “external” means that destination functions 150 are not provided as part of Central Server 130 and / or are not parties to the video communication. In other examples, destination functions 150 may be presentation functions configured to present video to viewers. In some examples, destination functions 150 may participate in a collaboration session (e.g., as a participant).

[0082] Unless otherwise specified, all functions and communications described herein are provided automatically, digitally and / or electronically, by computer software running on appropriate computer hardware, and transmitted over digital communication networks or channels such as the Internet.

[0083] Therefore, in the configuration of system 100 shown in Figure 1, multiple participant client devices 121 participate in a collaboration session provided by the video communication service 110. Each participant client device 121 maintains a continuous login, session, or similar state with the video communication service 110 and can participate in the same ongoing collaboration session provided by the video communication service 110. In other words, the collaboration session is "shared" among the participant client devices 121 and therefore also shared among the corresponding human participants 122.

[0084] The central server 130 may have a virtual client function 400, as shown in Figures 1 to 3b. This can be considered an automated participant client, an automated client corresponding to participant client 121, but not associated with human participant 122. Instead, the automated participant client 400, or virtual client function, is added to the video communication service 110 as a participant client to participate in the same collaboration session in the same or similar manner as participant client device 121. However, the automated participant client 400 typically runs on the central server 130 (for example, on a virtual machine hosted by the central server 130). For example, this means that the virtual client function 400 may run client function 125 or similar programs. Furthermore, the automated participant client allows programs to access and / or control various data and / or functions within the automated participant client (such as audio and / or video receiving / transmitting streams), respond to and process event messages, send event messages, etc. This means that, as such a participant client, the automated participant client 400 is permitted access to a continuously generated digital video and / or audio stream provided by the video communication service 110 as part of an ongoing collaboration session, and is consumed by the central server 130 via the automated participant client 400. Preferably, the automated participant client 400 receives from the video communication service 110 a common video and / or audio stream that is delivered or may be delivered to each participant client 121; each video and / or audio stream provided from one or more participant clients 121 to the video communication service 110 and relayed by the video communication service 110 in raw or modified form to all participant clients 121 or requesting participant clients 121; and / or a common time reference. For example, the virtual client function 400 is a so-called bot, which may be implemented by a Docker container, etc.

[0085] Furthermore, the central server 130 may include at least one software function, such as a preparation function 111, a sharing function 112, or a recording function 410. Details of the software function will be described later, for example, in relation to its corresponding operation. In some examples, the virtual client function 400 may include the sharing function 112 and the recording function 410. Alternatively, the sharing function 112 and / or the recording function 410 may directly implement the functionality of the virtual client function 400, and the virtual client function 400 may be considered as an integral component of the sharing function 112 and / or the recording function 410. In other words, the sharing function 112 is an automated client corresponding to the participant client device 121, but is not associated with a human user. Instead, the automated participant client 400 is added to the video communication service 110 as a participant client and can participate in the collaboration session in a similar manner to the participant client 121. As such a participant client, the sharing function 112 is permitted access to continuously generated digital video and / or audio streams provided by the video communication service 110 as part of the ongoing session, which the sharing function 112 can consume as an automated participant client. The sharing function 112 may relate to the first and / or second aspects of this specification. Similarly, the recording function 410 is an automated client corresponding to the participant client device 121, but is not associated with a human user. Instead, an automated participant client 400 may be added as a participant client to the video communication service 110 to participate in the same shared collaboration session as participant client 121. While operating as such a participant client, the recording function 410 is permitted access to continuously generated digital video and / or audio streams provided by the video communication service 110 as part of the ongoing session, which the recording function 410 can consume as an automated participant client. The recording function 410 may relate to the third aspect of this specification.

[0086] The central server 130 is configured to be connectable to, and may include, a storage function 140 configured to store video shared by the central server 130. The storage function 140 may also be configured to be connectable to local or remote storage, cloud storage, disk drives, hard drives, etc. In some examples, the central server 130 does not necessarily need to have the storage function 140; for example, the storage function 140 may be located outside the central server 130.

[0087] Figure 2 is similar to Figure 1, but in this example, the video communication service 110 is shown as an independent computer software function not included in the central server 130. In this example, the destination function 150 is also shown as an independent function, but in other examples it may be included in the central server 130. The destination function 150 may be a generating function, a presentation function, or a similar function disclosed herein.

[0088] Figure 3a is similar to Figure 1 and shows a further configuration example of system 100. In this case, various computer software functions within the central server 130 are shown separately.

[0089] Figure 3a also shows that each participant client device 121 is capable of running each client function 125, such as participant functions, software applications, apps, programs, or web applications. Each client function 125 may be installed on the client, be a downloadable or cacheable web application, or take other equivalent forms. Each client function 125 is configured to participate in a collaboration session hosted by the video communication service 110. Thus, each participant client device 121 may have similar login credentials for the collaboration session or the video communication service 110. This allows each participant client device 121 to participate in the collaboration session. In other words, the collaboration session is shared among the participant client devices 121 and therefore among the corresponding users 122. The client function 125 is configured to communicate and interact with the video communication service 110, for example, in a known client-server environment. For example, when using a known video conferencing software solution, the user needs to run dedicated client software on their device. This dedicated client software may be specific to the provider of the video communication service 110 (e.g., a company). As mentioned above, users need to register for the video communication service 110 in order to use it. Registration may be free or it may be a paid service.

[0090] When user 122 sets up a collaboration session, the user can select one or more add-on functions 127 (referred to as "add-ons" for simplicity) that are launched by the client function 125. When an add-on 127 is running on the client device 121, it can communicate with the client function 125 and / or the add-on backend function 113 and is specifically configured to work with the add-on 127.

[0091] In some examples, the add-on backend function 113 is further configured to start the sharing function 112 with appropriate initialization information. This allows the sharing function to communicate with the storage function 140 and the video communication service 110.

[0092] Although not shown in Figures 1 to 3a, the central server 130 may include further functions such as generation functions, automated generation functions, or similar functions. These functions, such as generation functions, are further illustrated and described with reference to Figure 3b and related diagrams.

[0093] Figure 3b shows a further example of a central server 130 that may be part of a system 100 according to any of Figures 1 through 3b. The central server 130 in Figure 3b provides features and functions that may be combined with one or more embodiments of this specification.

[0094] The central server 130 may include a collection function 131 that receives video and / or audio streams from other sources, such as a virtual client 140, client device 121, sharing function 112, and possibly a storage function, performs the processing described below, and provides a generated (and possibly shared) video stream. For example, this generated video stream is consumed by an external consumer, such as a destination function 150, and / or a video communication service 110, and is distributed by the video communication service 110 to one, more, or all of the participant client devices 121.

[0095] The central server 130 may further include an event detection function 132 configured to receive video and / or audio stream data, such as binary stream data, from the collection function 131, etc., and to perform event detection on each individual data stream received. The event detection function 132 may include an AI (artificial intelligence) component 132a that performs the event detection. Event detection may be performed without prior time synchronization of the individual collected streams. The event detection function 132 may detect events. An event may be related to a single video stream. For example, image processing may detect specific features within the video stream (such as a cat or a page / slide transition). Furthermore, audio processing may detect specific features (such as words or sounds). Examples of words include "start," "stop," and "dog," and examples of sounds include cheers and applause.

[0096] The central server 130 further includes a synchronization function 133 for time-synchronizing data streams (e.g., video and / or audio streams, subtitle streams, etc.) that may have been provided by the collection function 131 and processed by the event detection function 132. The synchronization function 133 may include an AI component for performing time synchronization. The synchronization function 133 may also perform synchronization as described with reference to Figures 5 and / or 7, or alternatively.

[0097] The central server 130 may further include a pattern detection function 134 configured to detect patterns in at least one, often at least two, e.g., at least three, or at least four, or e.g., all of the received data streams. Pattern detection may also be based on one, or possibly at least two or more, events detected for each individual data stream by the event detection function 132. Such detected events considered by the pattern detection function 134 may be temporally dispersed with respect to each individual stream collected. The pattern detection function 134 may include an AI component 134a for performing pattern detection. Pattern detection may further be based on the grouping described above, and in particular, may be configured to detect a particular pattern if it occurs in only one group, in only some (but not all) groups, or in all groups. In this context, patterns detected in one or more received video streams may include, for example, multiple participants speaking alternately or simultaneously on the same or different streams, or presentation slide changes occurring simultaneously with another event, such as a statement by another participant. This list is not exhaustive and is illustrative. A pattern may be defined using one or more of the following pieces of information: • The number of people speaking at the same time, • Number of people participating in the conversation • In a set of streams, which stream has a speaker who phonetically dominates (is dominant over) the other speakers in that stream? Image / video recognition of body language in one or more streams (body language includes, for example, hand and / or arm signals such as nodding, shaking heads to indicate yes / no, waving hands, giving a thumbs-up, and time-out signs).

[0098] In some embodiments, patterns can be defined in a specific extension along a time axis. For example, a pattern can be defined by considering a series of consecutive video frames or video stream content over a specific time frame or minimum time duration. Thus, a pattern can be defined based on a series of events occurring within a specific time window. For example, a series of events defining slide changes in a presentation can form a pattern together if they occur relatively close to each other. Such a pattern can be defined to capture a user quickly skipping through a presentation and moving to a different slide by repeatedly changing to the next or previous slide until the desired slide is reached.

[0099] The central server 130 may further include a generation function 135 configured to create generated digital video streams (e.g., shared digital video streams) based on data streams provided by the collection function 131 and, if necessary, on detected events and / or patterns. Generally, the generation function 135 can create secondary streams based on primary input streams. Such generated video streams may include at least one video stream created by reformatting or transforming one or more video streams provided by the collection function 131 in their raw data state, and may also include corresponding audio stream data. Multiple generated video streams may exist, as illustrated below. One such generated video stream may be created in the manner described above, but may also be created based on another already generated video stream. In some examples, the generation function 135 may be included in the sharing function 112, and therefore the operations performed by the generation function are invoked as part of the operations performed by the sharing function 112.

[0100] All generated video streams are preferably generated continuously and preferably in near real-time (after subtracting the types of latency and delays described below).

[0101] The central server 130 may also include a publishing function 136 configured to publish the generated digital video stream, for example, via the API 137 as described above.

[0102] In the pattern detection step performed by the pattern detection function 134, the time-synchronized shared video stream is analyzed and at least one pattern selected from the first pattern set is detected.

[0103] In contrast to the event detection step that can be performed by the event detection function 132, the pattern detection step is preferably performed based on video and / or audio information included as part of at least two time-synchronized primary video streams that are considered together.

[0104] The first set of patterns described above can include any number of pattern types, such as multiple participants speaking alternately or simultaneously, or changes in presentation slides occurring simultaneously with different events, such as different participants speaking. This list is not exhaustive and is illustrative.

[0105] In another embodiment, the detected pattern may relate to information contained in only one primary video stream, rather than information contained in multiple primary video streams. In such cases, it is preferable that such a pattern is detected based on video and / or audio information contained in that single primary video stream, spanning at least two detection events, e.g., two or more consecutively detected presentation slide changes or connection quality changes. For example, multiple slide changes that occur rapidly in time may be detected as a single slide change pattern rather than as separate slide change patterns for each detected slide change event. In some examples, a shared video stream and / or secondary video stream may function as a primary video stream for one or more functions such as collection, event detection, synchronization, pattern detection, and generation.

[0106] It is understood that the first set of events and the first set of patterns described above may comprise events / patterns belonging to a predetermined type defined using a corresponding set of parameters and parameter intervals. As will be discussed later, the events / patterns in the above set may also be defined and detected using various AI tools.

[0107] In a subsequent generation step performed by the generation function 135, the shared digital video stream is created as an output digital video stream based on the sequentially considered frames of the time-synchronized primary digital video stream and the detected pattern.

[0108] Before proceeding to a description of how System 100 manages video sharing, Figure 4 shows how System 100 prepares to make video shareable. This example relates to the first and second aspects of this disclosure.

[0109] During video preparation, the following operations are performed in any appropriate order. Although the operations are described as being performed by the central server 130, they may be performed by one or more of the functions comprising the central server 130 as described herein. For example, one or more steps may also be performed by the preparation function 111 and / or the add-on backend 113, or may be performed alternatively.

[0110] Operation A110 User 122 uses their device to select a video saved in a file. User 122 can also join a collaboration session later and share the video, but this is not required. This is because system 100 allows other users besides user 122 to access and share the video in subsequent collaboration sessions. Therefore, when the video is stored in the storage function 140, it becomes accessible to other users other than user 122, and those other users can share the video in collaboration sessions. The file is saved to any location, such as a local drive, remote drive, or cloud storage. After user 122 selects a video, client device 121 sends a request to the central server 130 to upload the video selected by the user, for example.

[0111] Operation A120 Following operation A110, the central server 130 receives the video file for processing in operation A130. This means that, in some examples, the preparation function 111 and / or add-on backend 113 receive the request and, for example, a reference to the file to be uploaded.

[0112] Operation A130 If the video file is not provided at a fixed frame rate, the central server 130 re-encodes (e.g., converts) the video to a fixed frame rate. The frame rate is either pre-set or selected by the user. The central server 130 can also convert the video to a desired resolution and / or frame rate.

[0113] Operation A140 The central server 130 can add an empty audio track to a video if the video file does not contain audio (e.g., an audio stream). This simplifies handling the video in subsequent processing steps, assuming the video has an audio track. This process is usually unnecessary.

[0114] Operation A150 The central server 130 may store the video. For example, the central server 130 can ensure that the video is stored by requesting a storage function 140 accessible to the central server 130 to save the file. The central server 130 may or may not have a storage function 140. In this way, for example, the video file may be stored in cloud storage or similar, which is highly reliable in terms of accessibility, connectivity, bandwidth, latency, etc.

[0115] Referring to Figures 5 and 6, these illustrate a method of this embodiment implemented in one of the exemplary systems 100 shown in Figures 1 through 3b. System 100 performs a method for managing video sharing in a collaboration session in which at least two client devices 121, 122 are participating. As described above, system 100 may include one or more of a video communication service 110 capable of hosting the collaboration session, a central server 130, and two client devices 121. Video files representing the video, stored in a storage function 140, are accessible from the central server 130. These video files may be stored in the storage function 140 as described above with reference to Figure 4. This video is shared as a shared video stream at a desired resolution and a desired frame rate. The desired frame rate may be a fixed frame rate or a variable frame rate. This description is intended to provide at least some groundwork for further details of the first and / or second embodiments, and optionally a third embodiment, as described herein.

[0116] In Figures 5 and / or 6, the central server 130 is shown separately from the video communication service 110. However, the central server 130 may include the video communication service 110. This means that operations performed by the video communication service 110 may be performed by the central server 130.

[0117] In the initial stage (for example, before operation B110 described later), a collaboration session is started (B100) and hosted, such as being managed by a video communication service 110. The start of a collaboration session may be triggered by a scheduled command or by a request from one of the two client devices 121. One of the client devices 121 may be referred to as the first client device 121, and the other as the second client device 122. It should be noted that the client devices 121 participating in the collaboration session may be organized into one or more groups by manual operation by user 122 or by automated operation based on criteria such as username domain. These groups may be associated with different time zones (with the meaning described later), different generation functions, and / or similar, which will be described later.

[0118] For example, when user 122 shares a video, that video may be shared to one or more different destinations, such as the video communication service 110, the destination function 150, and the generation function. This means that in some embodiments, the video is provided to at least two different destinations or destination functions. For example, one destination function may be the video communication service 110 and the other destination function may be the destination function 150. Therefore, the concepts of “first” and “second” are used to distinguish between features related to sharing with the video communication service 110 and features related to sharing with the destination functions. Unless otherwise clear from the context, “first” is associated with the video communication service 110 and “second” is associated with the destination function 150. As will become clear from the following explanation, for example, the first stream may be associated with a first time zone (as defined below) of the collaboration session, and the second stream may be associated with a second different time zone of the same collaboration session. For example, the following terms are used: “First Destination Frame Counter,” “Second Destination Frame Counter,” “First Desired Video Frame Rate,” “Second Desired Video Frame Rate,” “First Shared Video Stream,” “Second Video Stream,” “First Desired Resolution,” “Second Desired Resolution,” “First Duration,” “Second Duration,” “First Timestamp,” and “Second Timestamp.” Similarly, where applicable, “Shared Video Stream” refers to the “First Shared Video Stream” in the above examples and other features, and “Further Video Stream” refers to the “Second Video Stream.” In this context, it should be noted that, for example, the First Desired Resolution may differ from the Second Desired Resolution.

[0119] Operation B104 The above-mentioned at least two client devices 121 connect to the collaboration session, for example, by connecting to the video communication service 110. In this way, at least two client devices 121 participate in the collaboration session. For the purpose of sharing video, the add-on 127 is typically loaded by a client function 125 that runs on each of the at least two client devices 121.

[0120] Operation B106 The video communication service 110 manages the collaboration session and appropriately accepts connections from at least two client devices 121 in accordance with known methods.

[0121] Operation B110 In this operation, in some cases where the collaboration session is not hosted by the central server 130, the central server 130 may connect to the collaboration session. Therefore, the central server 130 participates in the collaboration session. From the perspective of the video communication service 110, the central server 130 (e.g., virtual client function 400) operates and is recognized in the same way as other clients or client devices. In this example, the video communication service 110 is hosting the collaboration session. This operation may occur after operation B120 below in at least some cases, as shown in Figure 7, for example.

[0122] Operation B118 A user (e.g., user 122) can select a video to share from among several videos displayed on client device 121. The videos to be displayed are prepared by the preparation function 111 and uploaded to the storage function 140, for example, by the preparation function 111. In some cases, the videos to be shared are predetermined, and the user does not need to select a video at all. As another example, the selection of a video to share may be triggered when the user utters a keyword associated with a particular video. For example, if the user says "cinnamon roll," a video explaining how to bake a cinnamon roll will be shared if it has been prepared in advance. As yet another example, the selection of a video may be triggered when a specific point in time (absolute time, the relative start time of the collaboration session, etc.) is reached or has passed. Furthermore, the selection of a video may also be triggered by an external command, for example, a command received via an API such as a central server 130. Thus, this request is also called a trigger command. In addition, the selection of a video may be triggered by other automated methods, for example, by an automated execution algorithm used to create a collaboration session.

[0123] Users can also choose whether or not to loop the video (for example, repeating it a specified number of times, or playing until the user stops it). Users can also adjust the volume of the audio within the video.

[0124] The user can also set the desired resolution of the video shared via the video communication service 110. Optionally, the user can also set the frame rate of the video shared via the video communication service 110.

[0125] Furthermore, the user can set a further desired resolution for the video being shared via the destination function 150, such as a software function that can direct the video (referred to herein as a virtual director). Optionally, the user can also set a further frame rate for the video being shared to the destination function 150.

[0126] The frame rate and / or resolution may also be determined automatically by System 100. For example, it may be determined based on the standard or current frame rate and / or resolution used in the collaboration session.

[0127] Therefore, typically, in response to input from user 122, client device 121 sends a request to the central server 130 to share the selected video. However, in some cases, the request may be sent autonomously (i.e., without user input) by client device 121 to the central server 130. For example, this request may be sent from client device 121 as part of the loading and / or start sequence when joining a collaboration session.

[0128] Operation B120 Following operation B118, the central server 130 receives the request or trigger command sent in operation B118. This request, in the form of a command or similar, instructs the central server 130 to share video within the collaboration session.

[0129] Except when the central server 130 hosts the collaboration session as shown in Figure 1, the central server 130 may connect to the video communication service 110 as shown in Figure 2 and / or Figure 3b, thereby joining the collaboration session. For example, the central server 130 may launch an instance of a sharing function 112 (e.g., an automated participant client) and instruct the sharing function 112 to connect to and join the collaboration session. The sharing function 112 may be referred to as an automated participant client, bot, virtual machine, etc. The sharing function 112 is typically associated with an IP address so that it can send and receive messages over the internet.

[0130] In some examples, the central server 130 provides the sharing function 112 with information about the video file. This information may be the complete file path to a local or remote directory. This allows the sharing function 112 to find and read the video file. In a more detailed example, an add-on 127 on a client device 121 sends a request to the add-on backend 113. The add-on backend 113 then starts up and starts or triggers the sharing function 112 with information about the file to be shared. The sharing function 112 then connects to the video communication service 110 and joins the collaboration session.

[0131] Operation B122 The central server 130 obtains a sync timestamp of an audio frame associated with the collaboration session, which is used to synchronize the shared video stream with the collaboration session. More specifically, the sync timestamp may be obtained from an audio stream provided by a video communication service. This audio stream is the audio of the collaboration session that can be heard by at least some users of the participant client devices. As described herein, participant client devices may be in different time zones (e.g., different delays and / or different generated streams). The purpose of the sync timestamp is to provide a criterion that can be used when setting the timestamp of a frame in the shared video stream. For example, a timestamp related to the criterion can be set by measuring the elapsed time since the sync timestamp was obtained. The elapsed time varies because the time it takes to acquire the first video frame varies due to network load, bandwidth, etc. Alternatively, a timestamp relative to the criterion can also be set by adding a preset offset value that is large enough to accommodate variations in the time it takes to acquire the first video frame. In some examples, this operation may be performed by a sharing function 112, a synchronization function, or a similar function. This operation may be performed later, such as after the acquisition of a single video frame in operation B140.

[0132] Operation B124 The central server 130 determines, at least based on the video frame rate difference, whether the shared video stream requires more, fewer, or the same number of video frames per unit of time as the video file. This means that the central server 130 determines, for example, whether it requires more, fewer, or the same number of video frames per unit of time compared to the number of video frames in the video file. The video frame rate difference is calculated as the difference or mismatch between the desired frame rate and the source frame rate of the video file. This determination result is used to determine when to retain, discard, or duplicate video frames when acquiring video frames, such as after operation B140 described below. In some examples, this operation is performed by the sharing function 112. In the case of a variable frame rate, this determination can be performed multiple times (e.g., repeatedly, or when the frame rate of the video file and / or shared video stream changes).

[0133] Operation B126 In a further determination step, the central server 130 may determine, based on at least a further video frame rate difference, whether the video stream requires more, fewer, or the same number of video frames as the video file, per unit of time. This means that the central server 130 further determines, again, per unit of time, whether it requires more, fewer, or the same number of video frames as the video file. The further video frame rate difference is calculated as the difference between the further desired frame rate and the source frame rate of the video file. This determination is used to determine when to retain, discard, or duplicate video frames when they are acquired, such as after operation B140 described later. In some examples, this operation is performed by the sharing function 112. Again, in the case of a variable frame rate, this determination can be performed multiple times (e.g., repeatedly, or when the frame rate of the video file and / or shared video stream changes).

[0134] Operation B130 According to the first embodiment, the central server 130 sets up a buffer for providing a shared video stream to the collaboration session. This buffer is capable of buffering a limited number of video frames corresponding to time intervals of less than 1 second, preferably less than 0.4 seconds, more preferably less than 0.2 seconds, and most preferably less than 0.1 seconds. The limited number of video frames is determined, for example, by the central server 130 based on a desired frame rate. For example, the limited number of video frames is determined as the product of the desired frame rate and the time interval, or in correspondence thereto. This operation can be performed by the sharing function 112. This operation is optional.

[0135] Operation B140 The central server 130 iteratively acquires video frames (e.g., individual video frames) of the video file. This iterative acquisition may include receiving and decoding the video frames. In some examples, the central server 130 iteratively acquires video frames one frame at a time, i.e., individual frames on a frame-by-frame basis. For example, processing of video frames in operation B150 may begin immediately as soon as a complete video frame is acquired. In some embodiments, the entire video file is not acquired at once, but is instead divided into multiple consecutive acquisitions, with one or more video frames acquired for each acquisition. In some examples, the iteratively acquired video frames are acquired from a buffer maintained by the central server 130. This method reduces the risk of the central server 130 running out of video frames to acquire, for example, in the event of insufficient bandwidth or poor connectivity to storage functions. The buffer mentioned above may be referred to as the “source buffer” to distinguish it from another buffer mentioned herein. The other buffer mentioned above may be referred to as the “destination buffer” for similar reasons.

[0136] In another embodiment, the central server 130 can iteratively acquire video frames from a generated video stream, for example, from multiple video sources (including, for example, a video file and one or more other files and / or video streams). In some examples, this operation is performed by the sharing function 112. For each video frame iteratively acquired in operation B140, the central server 130 preferably performs at least operation B150. According to various examples herein, the central server 130 also performs operations B160 and B170 for at least a portion of the iteratively acquired video frames.

[0137] Operation B150 The central server 130 generates zero, one, or more video frames based on each video frame acquired iteratively, according to the determination step B124. In this process, the timestamp of each of the one or more generated video frames is set based on the synchronization timestamp. The one or more generated frames have the desired resolution, and each timestamp corresponds to the desired frame rate.

[0138] For example, the resolution of a video frame can be adjusted using available software and / or known algorithms (e.g., codecs).

[0139] If the central server 130 generates zero video frames, that is, does not generate any video frames, it may mean that the acquired video frames are skipped, deleted, or discarded because the number of video frames required in the shared video stream (e.g., per unit of time) is insufficient.

[0140] When the central server 130 generates a single video frame (for example, one with the desired resolution), this may mean that the acquired video frame is included in the shared video stream. This can occur if the shared video stream requires more or the same number of video frames (for example, per unit of time) compared to the video stream stored in the video file. Naturally, this can also occur if video frames were skipped in a preceding iteration during the acquisition of the video frames.

[0141] If the central server 130 generates multiple video frames, i.e., additional video frames, it may mean that the generated video frames have been duplicated (copied, referenced, etc.) to fill the shared video stream with more video frames, as the shared video stream requires more video frames. In some cases, this operation may be performed by the sharing function 112.

[0142] Operation B160 The central server 130 provides a shared video stream by sending one or more generated video frames to, for example, the video communication service 110. For example, the central server 130 may insert one or more generated video frames into a buffer for provision to the client device 121. In some examples, the frames in the buffer are sent to the client device 121 via the video communication service 110. In other examples, the central server 130 may send each video frame directly, for example, without buffering, i.e., without using a buffer. This operation may be performed by the sharing function 112.

[0143] Operation B170 Next, the video communication service 110 receives the shared video stream (for example, frame by frame, or when a buffer is released if one is being used). Depending on the actual video communication service 110, the central server 130 may label the shared video stream in various ways. This label is interpreted by the video communication service 110 and used to supply the shared video stream to the client device 121 for display in various ways, such as screen sharing mode or camera mode.

[0144] Operation B174 While receiving video frames in operation B170, the video communication service processes the frames appropriately and provides the video stream to client devices 121 participating in the collaboration session.

[0145] Operation B176 Following operation B174, client device 121 receives a video stream containing video frames and optionally audio frames. More specifically, the client function 125 of client device 121 receives the video stream and displays the video to user 122. The frames and / or audio of the video stream containing the frames can be time-synchronized with other primary and / or secondary streams occurring within system 100. This ensures that each frame in the video stream is simultaneously displayed or used as a frame corresponding to other occurring streams. For example, if a video stream consisting of frames processed in relation to steps B110 to B174 is used as a primary stream, that stream may be time-synchronized with one or more additional primary streams used to create secondary streams and supplied to one or more participants 121 and / or destination functions 150. This time synchronization can be based on the aforementioned timestamps, which can be extracted from the audio track.

[0146] Operation B180 The central server 130 can provide additional video streams by sending one or more additional video frames to the destination function 150. In some examples, this additional video stream may be a second shared video stream, for example, if the destination function 150 is a participant in a collaboration session. For example, a sharing function 112 may perform this operation, causing, for example, the video communication service 110 to provide additional video streams. As described above, in some embodiments, there may therefore be two different shared video streams. These may be referred to as the first shared video stream, for example, in operation B160, and the second video stream or second shared video stream, for example, in operation B180. Again, for example, operations B122, B124, B140, B150, B160, operation B180, and other operations may be performed by the sharing function 112.

[0147] For example, the first shared video stream and the second video stream may be associated with the same video (e.g., the same video file) as mentioned in operation B120. However, the two video streams may differ in resolution, frame rate, or one or more other similar elements. Furthermore, the two video streams may have been processed or will be processed by the same or different generation functions described herein. In some examples, the destination function 150 may be a specific client device 121, which may be associated with a specific time zone and / or a specific intentional delay. The expressions “time zone” and “intentional delay” will be discussed later.

[0148] In some examples, the central server 130 may provide additional video streams by inserting the additional one or more video frames into additional buffers (for example, those associated with a particular client device 121). In all these examples, the time synchronization may be used to ensure that the individual frames corresponding to each other occur or are used simultaneously.

[0149] Operation B190 After operation B180, destination function 150 receives further video streams, for example, frame by frame, via any buffer.

[0150] Referring to one or more of Figures 4 through 6, the central server 130 can process video from, in some examples, storage functions 140, client devices 121, virtual clients, etc. This processing may include, for example, performing speech recognition and then translating it into different languages ​​(provided as text and / or audio for the video), or translating it into text in the same language or a different language (provided as subtitles for the video).

[0151] Figure 7 shows a schematic flowchart illustrating an exemplary method in the central server 130. Thus, the central server 130 implements a method that enables video sharing with at least two client devices 121 via the video communication service 110. For example, a sharing function 112, etc., can perform the operation shown in Figure 7. This example applies to the first and / or second aspects, and to some extent to the third aspect, of this specification.

[0152] Operation C110 The central server 130 connects to the collaboration session. If the central server 130 is equipped with a video communication service 110, this operation is either omitted or performed internally by the central server 130.

[0153] In some cases, add-on 127 may send a start message to add-on backend 113, which then starts the sharing function 112 (e.g., launch). The start message is sent from add-on 127 in response to the completion of loading of add-on 127, or in response to user input. User input could include user 122 selecting a file to share, or user 122 clicking a separate button to prepare for file sharing.

[0154] In some examples, the add-on backend function 113 can provide information about a video file to the sharing function 112. This allows the sharing function 112 to search for and retrieve video frames from the video file. This operation is similar to operation B110.

[0155] Operation C120 The central server 130 receives a request from one of the two client devices 121. This operation is the same as operation B120.

[0156] Operation C130 The central server 130 can obtain a desired resolution and a desired frame rate. The desired frame rate may be fixed or variable. For example, the desired resolution and frame rate may be pre-set by reading them from memory or the like. A predetermined desired resolution and / or frame rate may be provided by the video communication service 110 or the central server 130, for example, according to the service requirements. Therefore, this operation is optional. In another example, the desired resolution and frame rate may be input by the user 122 using, for example, a keyboard, mouse, touchscreen, etc.

[0157] Furthermore, the central server 130 can obtain an instruction for the buffer size (e.g., expressed in frames). In some examples, the central server 130 can obtain the time length to be used for the buffer. The buffer size instruction is then determined by the central server 130 based on the time length and the desired frame rate. For example, the buffer size may be the length in units of time (e.g., seconds) multiplied by the desired frame rate (e.g., frames per second). The buffer length in units of time is less than 2 seconds, preferably less than 1 second, more preferably less than 0.5 seconds, and most preferably less than 0.2 seconds. These values ​​are converted to the corresponding size in units of frames, depending on the desired frame rate.

[0158] Operation C140 The central server 130 can obtain the audio timestamp of the audio stream associated with the collaboration session. The audio stream is received from the video communication service 110 to the central server 130 via the video communication service 110's API, etc. This operation may be included in operation B122.

[0159] The central server 130 can also initiate time measurement for use in operations such as C170. For example, the central server 130 can start a timer to track the time since the audio timestamp was received. In some examples, the central server 130 can also read the current tick count from its operating system or similar source. In this way, the central server 130 can track the time since the audio timestamp was received. Because the central server 130 uses audio timestamps for synchronization, synchronization can be achieved without dedicated synchronization signals from the video communication service. Such dedicated synchronization signals include messages, signals in the received audio stream, and signals in the received video stream.

[0160] Before acquiring video frames (for example, in operation C150 described below), the central server 130 can obtain information about the streams from the video file (e.g., read, fetch, receive, etc.). This information includes instructions regarding the number of streams in the file and instructions regarding the types of streams. Stream types include video, audio, subtitles, etc. For example, if the central server 130 detects that multiple video streams exist, it can select the video stream with the highest resolution from among the video streams in the file.

[0161] As another example, if the central server 130 detects the presence of multiple audio streams, it typically selects the audio stream with the highest resolution from among the audio streams in the file. However, in some examples, the central server 130 may also select any one of the additional audio streams available. In some examples, the central server 130 may select an audio stream with a resolution that corresponds to a desired resolution, for example, the closest or nearly closest resolution. These examples are applicable to at least some embodiments of this specification.

[0162] In embodiments including the sharing function 112, the sharing function 112 receives a message containing information about the source video frame rate of the video to be shared, and optionally information about the source resolution of the video to be shared. This message is generated by the central server 130 using information from the selected stream (e.g., the video frame rate and resolution of the selected stream).

[0163] At this stage, the central server 130 may also determine the video frame rate difference between the desired frame rate and the source frame rate of the video file. In this way, the central server 130 can determine whether, for example, more, fewer, or the same number of video frames are needed per unit of time in the shared video stream, as will be described later.

[0164] Furthermore, in some cases where a second video stream is provided, the central server 130 can also determine a further video frame rate difference between the additional desired frame rate and the source frame rate of the video file. In this way, the central server 130 can determine whether, similar to the shared video stream (i.e., the first video stream), the additional video stream (i.e., the second video stream) requires, for example, more, fewer, or the same number of video frames per unit of time.

[0165] Accordingly, according to one embodiment of this specification, the central server 130 can determine, based on the difference in video frame rates, whether the shared video stream (i.e., the applicable first / second video stream) should have more, fewer, or the same number of video frames per unit of time, for example. This means, for example, a lower, higher, or the same frame rate compared to the video stream of the selected file. See also operations B124 and / or B126. In particular, the central server 130 may consider a difference in video frame rates to be negligible (not significant) if the difference is less than 0.5 frames per second (fps), preferably less than 0.1 fps, and most preferably less than 0.01 fps. Here, the absolute value of the difference in video frame rates is considered; that is, the increase or decrease in the required number of frames is not considered in evaluating the importance of the difference in video frame rates. In the variable frame rate embodiment, a corresponding fixed frame rate can be calculated or estimated and used instead of the variable frame rate.

[0166] Operation C150 The central server 130 iteratively acquires video frames (e.g., receive and / or decode them). In this context, it should be noted that in some examples, the acquired video frames may be provided by a generation function that generates a generated video stream based on the video file and one or more other video / audio sources. This operation is similar to operation B140.

[0167] Operation C160 The central server 130 generates zero, one, or more video frames based on each video frame acquired iteratively, according to the determination step B124. In this process, the timestamp of each of the one or more generated video frames is set based on the synchronization timestamp. The one or more generated frames have the desired resolution, and each timestamp corresponds to the desired frame rate. The timestamps are set according to various methods, such as those described in operation C170. This operation is similar to operation B150.

[0168] Operation C170 The central server 130 can calculate a duration (period) indicating the time elapsed since the audio timestamp was acquired. For example, this duration can be obtained using the timer (read, read value, etc.) described above. Furthermore, as described above, the central server 130 can again read an additional current tick count from the operating system and thereby obtain a measurement of the duration. However, in some examples, operation C170 may acquire an audio timestamp, similar to operation C140, instead of calculating the duration. In any case, the objective is to establish a common time reference between the collaboration session and the shared video stream (e.g., the first and / or second video stream). This operation is usually performed only once, preferably after the first video frame acquired.

[0169] Operation C180 The central server 130 can set the timestamp for each of the generated video frames, based on synchronized timestamps such as timestamps obtained from the audio of the collaboration session.

[0170] For example, the central server 130 may set a timestamp TS for each of the one or more generated video frames based on the duration (period) D, a destination frame counter dFC for counting frames provided to the collaboration session, and the video frame rate difference. The destination frame counter can be used to count video frames contained in a shared video stream. This operation may be part of operation B150.

[0171] For example, each timestamp TS is set as follows: TS= timestampOfAudio + D + desiredTDelta*dFC Here, desiredTDelta is given by the desired video frame rate, which can be obtained, for example, by recalculating the video frame rate to the corresponding time interval between video frames. In some cases, the desired video frame rate is equal to or substantially equal to the source video frame rate. In this case, as mentioned above, the difference in video frame rates is considered negligible. The duration D may be zero in some examples. This occurs, for example, when the synchronization timestampOfAudio is acquired after the first frame is acquired but before (preferably in the vicinity before, more preferably immediately before) operation C180 is executed. Thus, this may occur when operation C180 is performed on the first video frame acquired.

[0172] To achieve the desired frame rate in the shared video stream, the central server 130 evaluates the sign (i.e., positive or negative) and absolute value of the frame rate difference.

[0173] As mentioned above, if the frame rate difference is below the threshold, the difference is considered negligible (not significant), and no frames are added or removed when preparing the shared video stream. Therefore, depending on the situation, if the frame rate difference exceeds or reaches the threshold, it can be considered significant (significant).

[0174] When calculating the frame rate difference by subtracting the source frame rate from the desired video frame rate, a negative sign in the frame rate difference may indicate that the shared video stream will have fewer frames. This occurs when the desired frame rate is smaller than the source frame rate, i.e., significantly smaller. Naturally, when calculating the frame rate difference by subtracting the desired frame rate from the source frame rate, the sign will be positive when fewer frames are needed in the shared video stream. The same consideration applies when more frames are needed in the shared video stream.

[0175] As an example where the video frame rate difference is significant and the number of video frames required in the shared video stream is small, the central server 130 generates 0 or 1 video frame (i.e., each video frame) based on a destination frame counter that tracks video frames for the video communication service, the source video frame rate, the desired video frame rate, and a source frame counter SFC that tracks the number of video frames acquired. Thus, the source frame counter may be the sequence number of each video frame, which is considered the current video frame being processed.

[0176] More specifically, the central server 130 can calculate the source time difference STD as the reciprocal of the source video frame rate, and the desired time difference DTD as the reciprocal of the desired video frame rate. Therefore, if the value obtained by multiplying the source time difference STD by the source frame counter is greater than or equal to the value obtained by multiplying the desired time difference DTD by the destination frame counter, the central server 130 generates one or more video frames for each video frame, for example. Otherwise, the video frame currently being processed is discarded as necessary to reduce the frame rate. Furthermore, if one or more video frames are generated, the central server 130 increments the destination frame counter (e.g., by 1). Otherwise, the central server 130 maintains the current value of the destination frame counter, i.e., the destination frame counter is neither incremented nor decremented.

[0177] As an example of a situation where the difference in video frame rates is significant and more video frames are needed in a shared video stream, the central server 130 can generate one or more generated frames (i.e., each of the above video frames) based on: a destination frame counter for tracking video frames for the video communication service, a source video frame rate, a desired video frame rate, and a source frame counter SFC for tracking the amount of video frames acquired.

[0178] More specifically, for example, the central server 130 calculates the source time difference STD and the desired time difference DTD as described above. Therefore, the central server 130 can generate each video frame if the value obtained by multiplying the source time difference STD by the source frame counter is greater than the value obtained by multiplying the desired time difference DTD by the destination frame counter. Furthermore, the central server 130 can duplicate video frames as needed and provide them to the video communication service 110. Note that the central server 130 can continue duplicating video frames as long as the value obtained by multiplying the source time difference STD by the source frame counter is greater than the value obtained by multiplying the desired time difference DTD by the destination frame counter. The central server 130 can also increment the destination frame counter for each duplicated video frame. Furthermore, the timestamp of the duplicated frame is offset from the timestamp of the original frame by a multiple of the desired time difference DTD. In other words, the first duplicate frame has 1*DTD, the second existing duplicate frame has 2*DTD, and the third existing duplicate frame has 3*DTD.

[0179] In some examples, the central server 130 can maintain a current video frame counter to track the order and number of each video frame generated.

[0180] Operation C190 The central server 130 receives an update message (e.g., receives, reads, etc.), which instructs the central server 130 to set the desired video frame rate and resolution according to the information contained in the message. This results in an updated desired video frame rate and / or an updated desired resolution. The update message may be received from the client device 121 in response to user input, etc. In this way, the resolution and / or frame rate of the shared video stream are adjusted on the fly, for example, when video sharing is already in progress. Changed frame rates in a variable frame rate environment are also handled in a corresponding manner.

[0181] Operation C195 The central server 130 provides a shared video stream, for example, by transmitting one or more generated frames for distribution within a collaboration session by the video communication service 110. This operation is similar to operation B160.

[0182] The central server 130 may, additionally or alternatively, have the first and / or second video streams sent to the storage function 140 for storage. This is beneficial if the resolution and / or frame rate of the first and / or second video streams have been adjusted. This makes it possible to select a stream (e.g., a pre-storage stream) that matches the desired resolution and / or frame rate, for example, as part of operation C140, or as a similar process in one or more embodiments described herein. This is because the file may contain video streams that have been pre-generated or shared. In this example, the central server 130 may also store metadata such as events related to the video.

[0183] Furthermore, the central server 130 can send the first and / or second video streams to the generation function.

[0184] Furthermore, referring to Figure 7, the central server 130 may provide an additional video stream, also called a second video stream. The second video stream has a second desired frame rate and a second desired resolution, which can be predetermined and / or acquired in a similar manner to, for example, the first desired frame rate and the first desired resolution. The second video stream differs from the first shared video stream in at least some characteristics, such as frame rate and resolution. In some cases, one video stream may be a fixed frame rate stream while the other is a variable frame rate stream. In some examples, the second video stream is a second shared video stream, because it may be delivered to at least one of the client devices 121 participating in the collaboration session. The central server 130 may acquire a second desired resolution and / or a second desired frame rate associated with the provided second video stream, such as a user input or a predetermined value. See operation B118 above.

[0185] Operation D160 For example, the central server 130 may generate zero, one, or more additional video frames based on each of the iteratively acquired video frames, according to a further determination step B126. In addition, it may set the timestamp of each of the one or more additional video frames based on the synchronization timestamp, as necessary. The one or more additional video frames may have a further desired resolution, and each timestamp may correspond to a further desired frame rate. The timestamps are set according to various methods, such as those described in operation C170. This operation is similar to operations B150 and C160.

[0186] Operation D170 The central server 130 can calculate a further duration (i.e., a second duration) indicating the time elapsed since the audio timestamp was acquired. For example, the second duration can be obtained by reading the timer described above. Furthermore, as described above, the central server 130 can again read a further current tick count from the operating system and thereby obtain a measurement of the second duration. In some examples, the same duration as operation C170 may be used for the second video stream, especially when the first and second video streams are processed in parallel. However, if the second video stream is intended to be provided to a participant(s) in a specific time zone, it is preferable to calculate a further duration, and optionally, this may be a different value from the duration calculated in operation C170. This operation is the same as operation C170.

[0187] Operation D180 The central server 130 may set additional timestamps (i.e., second timestamps) for each generated video frame based on a second duration, a second destination frame counter that counts the frames provided to the destination function, and a second frame rate difference. This operation is similar to operation C180.

[0188] Operation D190 The central server 130 can receive update messages that include further desired frame rates and / or further desired resolutions. This operation is similar to operation C190.

[0189] Operation D195 The central server 130 can provide a second video stream by transmitting each selected generated frame. This operation is similar to operations B160 and C195.

[0190] Operations D160, D170, D180, D190, and D195 can be performed independently of the corresponding operations related to sharing the first shared video stream, that is, before, after, or simultaneously with one or more of these corresponding operations.

[0191] It should be noted that the first and second embodiments described herein may be combined to form further embodiments. This means, for example, that, where applicable, the shared function may perform some of the operations described in the first embodiment (e.g., those described as being performed by a central server). This means, for example, that some embodiments of the second embodiment may include buffering as described in the first embodiment.

[0192] Figure 8 shows an exemplary timeline in which a central server 130, such as a sharing function 112, receives a request and obtains an audio timestamp of the current audio frame of the audio stream at time t0. This example may apply, at least in part, to the first, second, and / or third aspects of this disclosure. The audio stream is associated with a collaboration session. This means, for example, that the audio stream belongs to or originates from a collaboration session. The audio stream may be the principal or sole audio stream shared by multiple video streams, or all video streams, that originate or are used in the collaboration session. For example, the audio stream may be generated by a video communication service 110 based on two or more audio streams available within the collaboration session. Thus, the audio timestamp can be obtained after the central server 130 has connected to or is connected to the collaboration session.

[0193] At time t1, the central server 130 is assumed to have acquired video frames (e.g., decoded them) by processing the video frame by frame. As described above, the central server 130 performs different processing based on the number of frames required in the shared video stream (e.g., the first and second video streams). In some cases, if a change in the resolution of the video frames is necessary, this may be performed before time t1, but this is optional.

[0194] Alternatively or additionally, the central server 130 may obtain the audio timestamp of the current audio frame at time t1, i.e., after the video frame has been acquired. This allows synchronization to be performed after the delay caused by the acquisition and decoding of video frames from the video file.

[0195] Next, the video frames are provided to destinations such as video communication services and destination functions for distribution within the collaboration session, by methods such as being transmitted or inserted into a video frame transmission buffer.

[0196] Advantageously, at least one embodiment of this specification can provide an improved video sharing experience without requiring any changes to the software of the client function 125 and / or the video communication service 110. For example, the add-on 127, add-on backend 113 and sharing function 112 according to a second aspect of this specification can provide the improved sharing experience described above.

[0197] According to a third aspect of this disclosure, an exemplary flowchart is shown in Figure 9. In this example, an exemplary computer implementation method for managing files may be performed by a computing function 600. According to the first example, the computing function 600 is a recording add-on that runs on a client device 121.

[0198] Operation E110 Computing function 600 opens the file using rendering function 610. Rendering function 610 renders at least a portion of the file as an image. Rendering function 610 may be a browser program capable of opening the file and displaying its contents as intended. Scripts such as JavaScript and TypeScript may assist the browser in rendering the file. In some examples, rendering function 610 may be implemented as a server-side rendering function.

[0199] According to the first example, this action is performed by a recording add-on. For example, the recording add-on opens a file stored locally or remotely. Typically, while the file is open, at least a portion of it is visible to user 122.

[0200] In particular, in some cases, files may be opened using a logical display device, meaning that the contents of the file may not be visible on, for example, the client device 121 or elsewhere. However, it is still possible to obtain an image of the file contents drawn by the rendering function 610. As an example, the rendering function 610 may capture a so-called screen dump of the display device (a real display device or a virtual / logical display device). In other cases, the rendered image is not displayed anywhere and is simply saved after the rendering of the image is complete. The rendered image can take any appropriate format, such as a bitmap, a vector graphics file, or an HTML-encoded file.

[0201] Operation E111 The computing function 600 can read or simulate one or more event messages by generating input events that modify the image rendered by the rendering function 610. As described later, input events include right / left mouse clicks, scrolling, page up / down, other keystrokes, or similar events.

[0202] Operation E112 The computing function 600 can repeatedly capture multiple images rendered by the rendering function 610. In this way, the computing function 600 can monitor changes or differences in the rendered images over time.

[0203] Operation E115 The computing function 600 can receive the audio stream of the collaboration session from the video communication service 110.

[0204] Operation E117 Following operation E115, computing function 600 can detect keywords within the audio stream, as described below.

[0205] Operation E118 The computing function 600 can generate one or more event messages if it detects a difference between two consecutively captured images. The event message indicates that the two consecutively captured images are different from each other.

[0206] Alternatively, if no difference is detected between two consecutively captured images, the computing function 600 may generate one or more event messages as described above. These event messages indicate that the two consecutively captured images are similar to each other (e.g., equal, identical, or nearly identical).

[0207] The detected differences may exceed a defined threshold difference. This threshold difference is calculated based on any appropriate measure, such as the average pixel value difference or the transformation-based difference (e.g., a sufficiently offset image in the screen pixel coordinate system). The detected differences may also be based on the minimum elapsed time since the most recently captured difference, resulting in the image being captured only at a specific minimum acceptable pitch (cadence), or at a lower frequency.

[0208] Operation E120 The computing function 600 receives one or more event messages. These one or more event messages relate to the image rendered by the rendering function 610. More specifically, these one or more event messages relate to events that modified the image rendered by the rendering function 610.

[0209] An event message may contain one or more of the following information: • Left-click and / or right-click, and optionally the location where the click occurred. Window scrolling, • Window zoom operation, • Typing, keystrokes, and optionally, identification information of keys associated with typing. • Current page number, slice number, current sheet title, etc. • Events received from the hosted application (e.g., slide change, page change, current page, current sheet, language change, presentation start / end, etc.) • Timestamp, etc.

[0210] One or more of the above event messages or events may be triggered by the user interacting with the rendered portion of the file. Additionally or alternatively, events may be triggered by an add-on simulating the user interacting with the rendered portion of the file.

[0211] In the first example, this operation is performed by a recording add-on. For example, a recording add-on listens to events from rendering functions, etc.

[0212] In some embodiments, one or more event messages are generated by, for example, a recording add-on acquiring them, such as by receiving them, and detecting one or more keywords. For example, in an audio stream received from a video communication service 110, as in the first example, in a portion of a file currently displayed in an image rendered by a rendering function, etc.

[0213] Keywords may be predetermined. However, keywords may be any or all words detected in the audio stream during a specific period in which the image provided by the rendering function remained unchanged.

[0214] Furthermore, in combination with or alone, keywords may be extracted from images using optical character recognition, and / or keywords may be extracted from parts of a file represented by the image, for example, using rendering functions. The keywords are then included in or referenced in metadata, thereby associating them with the image (e.g., a specific duration (period), point in time, etc., in the video being created by operation E160).

[0215] In some embodiments, one or more event messages are generated by a computing function 600 that monitors changes in an image rendered by a rendering function, for example, periodically, irregularly, frequently, continuously, or in a similar manner.

[0216] This method includes actions E130 and E140 for each event message of one or more event messages.

[0217] Operation E130 The computing function 600 acquires an image corresponding to each event message. For example, an image may be associated with one or more event messages. In some examples, there is a separate image corresponding to each event message, but this is not required. The image, or each image, is either rendered or will be rendered by the rendering function 610. This means that the image, or each image, represents a rendered portion of a file (e.g., the portion currently being rendered, the most recently rendered portion, etc.). The image, or each image, is associated with a sequence indication, such as a counter, a timestamp, or a ticket indicating its position within the image sequence. More specifically, it is preferable that each image, or image, is associated with its corresponding sequence indication. It is preferable that each sequence indication is different from one another. Each sequence indication is used to uniquely identify a defined order of captured images, and possibly a timeline.

[0218] Operation E135 In some examples, one or more event messages include an instruction to start a sequence of repeated image captures by the rendering function 610, or an instruction to stop the repeated image captures by the rendering function 610. When the computing function 600 receives a start instruction, it can repeatedly capture images using the rendering function 610.

[0219] Operation E140 The computing function 600 generates metadata based on event messages and sequence instructions. Sequence instructions may include one or more timestamps, counters, and / or values ​​derived therefrom. In an example where operation E135 is performed, the computing function 600 may generate multiple event messages until a stop instruction is acquired or received. The computing function 600 can generate metadata by associating information derived from event messages with sequence instructions.

[0220] For example, metadata may include the following: • Sequence instruction, or a value based on a sequence instruction. • Event identifiers generated based on event messages (e.g., "click", "double-click", "zoom", "page down / up", "scroll") • Event data generated based on the event message (e.g., "click" location, page number, slide number, zoom level, scroll line count).

[0221] In some examples, metadata defines how a file was processed to produce a rendered image. For instance, after opening, the metadata may include information for each captured image about one or more user commands (e.g., page down or mouse click) that produced the rendered image when applied to the opened document by the software function used to render the document. In some embodiments, the definition provided by the metadata is complete in the sense that by opening the document using the defined software function, it is possible to deterministically arrive at one, more, or all rendered images and then apply the instructions, commands, or inputs specified in the metadata.

[0222] For example, when a value derived from a sequence instruction is used, that value may be a timestamp indicating when the image was captured, or it may be a value extrapolated from that time.

[0223] Operation E150 Computing function 600 provides metadata (e.g., transmission, storage). Metadata may be transmitted to and / or stored in local and / or remote storage. Metadata may be stored, for example, as part of, or associated with, captured images, as part of, the resulting video.

[0224] Operation E160 Computing function 600 provides video, which includes a set of images. The set of images includes a corresponding image for each event message. Video is a representation (display) of a file. Examples include video files and video streams. According to some examples, each image in a video image set is assigned a sequence instruction (e.g., associated, assigned, etc.). Typically, each image or video frame is assigned a corresponding timestamp.

[0225] When using metadata, the following steps may be performed, for example, by computing functions.

[0226] The search term is retrieved. The search term can identify a specific event identifier and / or specific event data. If the search term is found in the metadata, a reference to a specific point in time within the video is obtained using the specific sequence instruction associated with the search term. This allows jumping, skipping, fast-forwarding, etc., to the point in the video indicated by the reference.

[0227] In other embodiments, metadata can be used to interpret instructions, commands, or inputs that result in changes between rendered images, thereby interpreting the differences between the first rendered image and preceding or subsequent rendered images. For example, the detection of a “page down” instruction can be interpreted as meaning that the subsequent image is the “next page” of the rendered document, and this information can be used to skip to the subsequent rendered image in order to skip to the next page of the rendered document.

[0228] Furthermore, exemplary methods according to a third aspect of this disclosure are shown in Figures 10 and 11. Figures 10 and 11 are examples of the method shown in Figure 9. In this example, a system 100, such as the system in Figure 1, Figure 2, Figure 3a, or Figure 3b, performs an exemplary method of managing the content of a file shared in a collaboration session in which at least one client device 121 participates. System 100 comprises a video communication service 110 capable of hosting the collaboration session, a central server 130, and the client device 121. See also Figure 3b for the operation performed by the recording function 410 (for example, provided in the central server 130). In this example, event messages may originate from any participant in the collaboration session, such as client devices or virtual clients participating in the collaboration session. Event messages may also be generated by the client device 121, the central server 110, the recording function, a synchronization add-on, etc.

[0229] Operation F110 In the initial stages, for example, user 122 may request to share the contents of a file via client device 121 using a synchronization add-on 128 running on client device 121. The synchronization add-on enables collaboration among all participants by synchronizing the data to be collaborated on among all participants. For example, this type of synchronization may be handled by a known synchronization framework such as SignalR or LiveShare, depending on the environment. Here, the environment may refer to a video communication service and associated client software for which a software development kit (SDK) is available, but if an SDK is not available, the functionality described herein can be directly incorporated into the video communication service and associated client software. The purpose of a known synchronization framework is to facilitate the development of programs such as the synchronization add-on 128. This eliminates the need for software developers to develop, for example, a synchronization backend or complex synchronization routines to keep data in different locations synchronized. All of these, and possibly more, are provided by the synchronization framework. In other words, client device 121 (e.g., an add-on running on client device 121) may receive a request from user 122 to start the synchronization add-on 128 for a file selected by user 122.

[0230] In some examples, a client device 121 obtains a request to share the contents of a file in a collaboration session using a synchronization add-on 128. This synchronization add-on 128 generates a corresponding instance of the synchronization add-on 128 that runs on each client 121 of at least one client device 121 participating in the collaboration session. Here, the central server 130 maintains a central copy of the file in synchronization with each data portion managed by each instance of the synchronization add-on 128 on each client 121, where each data portion corresponds to at least a portion of the central copy of the file.

[0231] Operation F120 Client device 121 sends a message to central server 130. This message instructs central server 130 to capture image frames of content displayed by an instance of sync add-on 128 running within recording function 410. In some examples, client device 121 sends a message to add-on backend 113 on central server 130. More specifically, the message may be sent to add-on backend 113 associated with add-on 127. This message may contain information indicating that the sync add-on has been incorporated into the collaboration session. Thus, all participants in the collaboration session will have their respective corresponding instances of the sync add-on running. In particular, the functionality and appearance of sync add-on 128 may differ depending on the participating client device. In some examples, sync add-on 128 programmatically detects the client type and operates accordingly. Client types may refer to “Participant,” “Meeting Organizer,” “Presenter,” “Viewer,” “Editor,” client roles described herein, or similar.

[0232] Operation F130 Next, the add-on backend 113 typically receives a message from a client device 121, such as an add-on 127. The add-on backend 113 then sends the message to the recording function 410. This may simply mean that the add-on backend 113 forwards the message to the recording function 410. The message may contain information about the collaboration session (e.g., an identifier to allow the recording function to join the collaboration session). The add-on backend 113 can also start the recording function 410, for example, by acting as a virtual client if the recording function 410 is not already running. For example, the add-on backend 113 can start the recording function 410 by sending a command to start it.

[0233] Operation F140 If not already started, the recording function 410 may be started as described above. Subsequently, the recording function 410 receives messages originating from the client device 121 (for example, an add-on 127 running within the client device 121). For example, this effectively means that the client device 121, for example, an add-on 127, sends a message to the recording function 410 via the add-on backend 113.

[0234] Using the information in the message, the recording function 410 can join the collaboration session. As a result, the recording function 410 can execute a second synchronization add-on 128, i.e., an instance of the synchronization add-on 128. Thus, the message can contain information that identifies the collaboration session, as well as additional information such as a URL, path, and hostname. In this way, the synchronization add-on can process different file types based on the path, for example, by using different paths to call code that processes the current file type.

[0235] In this example, the event message may contain information about one or more of the following: • Left-click and / or right-click of the mouse in an instance of the sync add-on. • Window scrolling within an instance of the sync add-on. • Typing within a sync add-on instance, • Current page number, slice number, current sheet title, etc. • Speech recognition for sound events in the audio of a collaboration session (e.g., "applause," "wow," "cheers," "singing," etc.), specific words (e.g., "notes," "look," "dog," "easy," etc.), background music type (e.g., upbeat, sad, happy, scary, etc.), or similar information. • Or various appropriate events.

[0236] Therefore, the Sync Add-on 128 can perform various analyses on video and / or audio streams to generate metadata, for example, by extracting relevant parts of content files. These analyses include, but are not limited to, speech recognition, optical character recognition, music analysis, and general image processing. The term "event" as used here may refer to any information that may be collected by the recording function, for example, after analysis and / or detection.

[0237] Operation F145 (Figure 11) The central server 130 can obtain the synchronization timestamp of the collaboration session. See the similar operation regarding the acquisition of synchronization timestamps in this specification.

[0238] Operation F150 The central server 130 repeatedly captures each image frame of a portion of the content (for example, the portion currently represented by each instance of the synchronization add-on 128 running within the central server 130). In some examples, the recording function 410 may repeatedly capture images of the synchronization add-on as the recording function 410 runs its own instance. In some examples, the recording function 410 may obtain a synchronization timestamp for each captured image, for example, once, or periodically or irregularly. The synchronization timestamp may be obtained from the timestamp of the audio stream of the collaboration session, as described herein. The synchronization timestamp is not always necessary when the image (e.g., frame) is used independently of the collaboration session (e.g., for saving).

[0239] However, each captured image is associated with a corresponding timestamp, though not always. The corresponding timestamp can be set based on the desired frame rate. For example, if the desired frame rate is a fixed 40 frames per second, the difference between any two consecutive timestamps will be 1 / 40th of a second. The timestamp set for each image is used in operation F160, which will be described later.

[0240] Repeated image capture can be performed at any time interval (e.g., regularly, irregularly, etc.). In some examples, the desired time interval is set to match the desired frame rate of the stream provided by action F173. Alternatively, action F150 is triggered by an event message captured by action F160, for example. This means that action F150 may be executed irregularly, triggered by action F160, for example.

[0241] Operation F155 The central server 130, such as the recording function 410, can set the corresponding frame timestamp for each image frame based on the synchronization timestamp.

[0242] Operation F160 The central server repeatedly retrieves event messages. Each event message includes, for example, information regarding user input and / or content in each instance of the synchronization add-on (128). This means, for example, that the recording function 410 listens for event messages from, for example, the synchronization add-on 128 that is running within the recording function 410.

[0243] When the recording function 410 receives an event message, the recording function 410 determines the respective event timestamp associated with the event message, that is, at least the information included in the event message. The corresponding event timestamp is determined by the recording function 410 in various ways described below, or other ways. As an example, the recording function 410 can set each timestamp equal to the latest timestamp of the most recently captured frame.

[0244] As another example, the recording function 410 can set the corresponding timestamp for each to be equal to the timestamp of the most recently captured frame, or the timestamp of a frame scheduled to be captured in the future, that is closest in terms of, for example, time interval. As yet another example, the recording function 410 can set a relative timestamp with respect to the latest timestamp of the most recently captured frame by, for example, measuring the time elapsed between the capture of the latest frame and the receipt of the event message.

[0245] Operation F165 (Figure 11) The central server 130, which includes recording functions, can determine an event timestamp for each event message in order to retrieve metadata associated with the content. This determination is based on the synchronization timestamp and the time when each event message was received. The metadata includes the respective event timestamp and information about each event message in the event messages.

[0246] Operation F170 In some examples, a central server 130 and / or a recording function 410, etc., provide metadata to a storage function 140 by means of transmission or other means.

[0247] Operation F173 The central server 130 and / or recording function 410, etc., can each provide the captured images or frames as video, such as a video stream or video file. For example, each captured image may be sent as a video stream to a destination function, such as a generation function or a video communication service.

[0248] In some examples, the recording function 410 sends the video stream and / or metadata to the storage function 140 for storage. The video stream and metadata are stored in two different files or in a common file. In other examples, the recording function 410 sends the video stream to the video communication service 110 for distribution within a collaboration session.

[0249] Operation F180 The storage function 140 can receive and acquire video streams and / or metadata, and store them in local or remote storage according to known methods.

[0250] As a result, the content of a file (e.g., any file, document, presentation, spreadsheet, etc.) is converted into a video available from the storage function 140. This makes it possible to share the video representing the file's content as a shared video stream according to the first and / or second aspects of this disclosure. In some examples of the first and / or second aspects of this disclosure, system 100 also retrieves metadata from the storage function 140. System 100 (e.g., add-on 127) can read at least some, preferably all, of the metadata and display user interface objects to user 122. Each user interface object is associated with a corresponding metadata piece, although add-on 127 does not necessarily display the corresponding metadata piece itself. The user interface objects allow user 122 to jump to the portion of the video corresponding to the timestamp associated with the corresponding metadata piece of the user interface object of their choice.

[0251] Advantageously, in at least some embodiments of the third aspect of this specification, metadata makes it possible to locate specific points or portions of a video stream. These specific points can be easily found, for example, by a user, an add-on, etc. In some scenarios, it is envisioned that an add-on, for example using AI, autonomously decides which specific points or portions of a shared video should be used, i.e., which should be used in a production. Furthermore, this means that an add-on can use the video file and associated metadata to select which video streams should be included in a production according to the metadata.

[0252] Generally, a file may or may not have a timeline defined as an integral part of the file. For example, a file may be static (e.g., a plain text document) or dynamic (e.g., a hypertext document or an interactive web page). Using the mechanism described herein, such a file can be converted into a set of rendered images with associated metadata, and such images can be ordered along a timeline. The set of rendered images can form a video stream, or can be converted into a video stream with a desired resolution and / or a desired fixed or variable frame rate using the mechanism described herein. This conversion can be performed in conjunction with or after rendering, and can be run as a background process without displaying the rendered images. It can also be performed when user 122 opens and manipulates a document on their client 121, while the user is not participating in an ongoing collaboration session. Alternatively, it can be performed as a by-effect of a file being shared and manipulated during an ongoing collaboration session. Once the file is converted into a set of rendered images or a video stream, the images / videos can be quickly skipped forward or backward during the collaboration session using the metadata associated with each frame. In this case, the image / video is treated as a primary video stream of the type described herein, but is configured to represent a file in a set of available states (e.g., scrolling down a text document, flipping between different slides in a presentation, performing an interactive task on a web page, viewing different chapters of a video movie, etc.). The video stream representing a set of rendered images may have one frame per point in time determined by the frame rate (one frame per frame), or each rendered image may have a timestamp (variable frame rate). In the former case, the rendered images are duplicated along the timeline.In the latter case, frame duplication is unnecessary; instead, a variable frame rate is used to determine when to display the next frame within the video stream. The video stream thus constructed can be used as a primary video stream in any of the methods disclosed herein. When the video stream is used by participant user 122 (e.g., a share initiated using the corresponding client 121), user 122 can control the video stream using control commands entered via the client (e.g., keyboard or mouse input). This causes the video stream to skip forward or backward in response to the control commands, and the stored metadata of the video stream is used to determine to what point in the video stream to skip based on the sequence of control commands entered by the user. Such processing may create a new video stream (primary video stream) which can be used as input for creating a secondary stream, and similar processing is repeated. The video stream created based on the file is processed as described above to have a predetermined frame rate and / or resolution.

[0253] As shown in Figure 12, participant clients 121 may be organized into two or more participant client groups 121. Figure 12 is a simplified illustration of system 100 in a configuration that performs automatic creation of output video streams when such groups exist.

[0254] In Figure 12, the central server 130 includes a collection function 131 which may be configured as described above.

[0255] The central server 130 further includes a first generation function 135', a second generation function 135'', and a third generation function 135'''. Each generation function 135', 135'', and 135'''' corresponds to generation function 135, and the above-mentioned aspects of generation function 135 also apply to generation functions 135', 135'', and 135''''. Generation functions 135', 135'', and 135'''' can exist as individual functions or be jointly deployed as a single logical function containing multiple functions. Depending on the detailed configuration of the central server 130, there may be more than three generation functions. Generation functions 135', 135'', and 135'''' may, in some cases, be different functional aspects of the same generation function 135. Various communications between generation functions 135', 135'', and 135'''' and other entities such as shared functions are performed via appropriate APIs.

[0256] Furthermore, it is understood that there may be separate collection functions 131 for each generation function 135', 135'', 135'''' or group thereof, and that, depending on the detailed configuration, there may be multiple logically separated central servers 130, each equipped with a corresponding collection function 131.

[0257] Furthermore, the central server 130 includes a first public function 136', a second public function 136'', and a third public function 136'''. Each public function 136', 136'', and 136''' corresponds to public function 136, and the above-mentioned provisions regarding public function 136 also apply to public functions 136', 136'', and 136'''. Public functions 136', 136'', and 136''' may be individual functions, or multiple functions may be jointly arranged as a single logical function, and there may be more than three public functions depending on the detailed configuration of the central server 130. In some cases, public functions 136', 136'', and 136''' may be different functional aspects of a single public function 136.

[0258] Figure 12 shows, for illustrative purposes, three sets or groups of participant clients corresponding to the participant client 121 described above. Thus, there exists a first participant client group 121', a second participant client group 121'', and a third participant client group 121'''. Each of these groups may consist of one, preferably at least two, participant clients. Depending on the detailed configuration, there may be only two such groups, or more than three. The assignments between groups 121', 121'', and 121''' are mutually exclusive, meaning that each participant client 121 is assigned to at most one group 121', 121'', or 121'''. In another configuration, at least one participant client 121 may be assigned to multiple groups 121', 121'', or 121''' simultaneously.

[0259] Figure 12 also shows destination functions 150 such as external consumers, and as mentioned above, it is understood that there may be multiple external consumers 150.

[0260] Although Figure 12 does not show the video communication service 110 for simplification, it is understood that the general type of video communication service described above can be used in combination with the central server 130. For example, this may occur when the central server 130 is used to provide a shared video communication service to each participant client 121 in the manner described above.

[0261] The collection function 131 can collect primary video streams from each participant client 121 (e.g., participant clients of groups 121', 121'', and 121'''). Based on the provided primary video streams, the generation functions 135', 135'', and 135''' generate their respective digital video output streams.

[0262] As shown in FIG. 12, one or more of the generated output streams are provided as corresponding input digital video streams from one or more corresponding generation functions 135', 135''' to another generation function 135'', which further generate a secondary digital output video stream and are published by the publishing function 136''. Thus, the secondary digital output video stream is created based on one or more input primary digital video streams, as well as one or more pre-created input digital video streams.

[0263] In some embodiments, the synchronization function 133 performs the step of intentionally introducing a delay of up to 30 seconds, such as up to 5 seconds, such as up to 1 second, such as up to 0.5 seconds, but longer than 0 seconds (in this context, "delay" and "latency" are intended to have the same meaning), whereby the shared video stream is provided with at least the above delay. In any case, the intentionally introduced delay is at least a plurality of video frames (e.g., at least 3 frames, or at least 5 frames, or even 10 frames). As used herein, the term "intentionally" means that the delay is introduced regardless of the need to introduce a delay based on synchronization problems or similar problems. In other words, the intentionally introduced delay is introduced in addition to the delay introduced, for example, as part of the synchronization of the shared video stream with the audio stream of a collaboration session. The intentionally introduced delay can be a fixed value or a variable value predetermined with respect to a common time reference (e.g., a synchronization timestamp). The delay is measured based on the synchronization timestamp. Further, the delay may be different for the first and second video streams described herein (e.g., operations B160, B180 and throughout the specification).

[0264] In some embodiments, a relatively small delay is introduced, such as less than 0.6 seconds, preferably less than 0.4 seconds, more preferably less than 0.2 seconds, and most preferably 0.1 seconds. This delay is barely noticeable to participants in the collaboration session of the video communication service 110. In other embodiments (for example, when the shared video stream is not used in a bidirectional context and is instead exposed in one-way communication to the destination function 150), a larger delay can be introduced. This may be applied, for example, to a second video stream.

[0265] The intentionally introduced delay (or intentional delay) may be sufficient to give the synchronization function 133 enough time to map each collected primary stream video frame to the correct common time reference timestamp 261. It may also be sufficient to ensure enough time to perform the event detection described above and detect lost primary streams, slide changes, resolution changes, etc. Furthermore, as will be described later, intentionally introducing a delay may improve the performance of the pattern detection function 134. The intentional delay may be sufficient to ensure enough time to perform the desired generation function.

[0266] It is understood that the introduction of this delay involves a process of buffering each collected and time-synchronized primary video stream, and then using the buffered frames to expose the output video stream. In other words, the video and / or audio data of at least one, more, or all primary video streams will reside on the central server 130 in a buffered state, like a cache. However, this is not intended to be used to deal with fluctuating bandwidth conditions (like a conventional cache buffer), but rather for the reasons mentioned above, particularly for use by the pattern detection function 134.

[0267] Therefore, in some embodiments, the pattern detection step includes considering specific information of at least one, for example, more, for example, at least four, or all of the primary digital video streams 210, 301. This specific information resides in frames 213 that are later than frames of the time-synchronized primary digital video stream 210 that have not yet been used to generate the output digital video stream 230. Thus, newly added frames 213 reside in the buffer 250 for a certain delay period before forming part (or the basis) of the output video stream 230. During this period, the information of the frame 213 becomes "future" information with respect to the frames currently being used to generate the current frame of the output video stream 230. When the timeline of the output video stream 230 reaches the frame 213, it may be used to generate the corresponding frame of the output video stream 230 and then discarded.

[0268] In other words, the pattern detection function 134 can utilize a set of video / audio frames 213 that have not yet been used to generate the output video stream 230, and can use this data to detect the aforementioned pattern.

[0269] In summary, by combining event detection based on individual primary video streams, intentionally introduced delays, pattern detection based on multiple time-synchronized primary video streams and detected events, and a generation process based on detected patterns, it is possible to automatically generate output digital video streams to suit diverse tastes and styles. This result is effective in various neural network and / or rule-based analysis techniques used in the event detection function 132, pattern detection function 134, and generation function 135. Furthermore, features that are effective in at least some embodiments include: utilizing a first generated video stream for the automatic generation of a second generated video stream, and adding different intentional delays for different groups of participant clients. In particular, effective in at least some embodiments, detected triggers can cause switching of video streams used in the generated output video stream, or automatic trimming or zooming of video streams used in the output video stream.

[0270] Multiple different generation steps 135', 135'', 135'''' may include introducing corresponding time delays. Typically, at least two time delays are different from each other, but are not necessarily required. In some embodiments, one or more of the output digital video streams generated from these generation steps 135', 135'', 135'''' may be temporally asynchronous with other video streams that may be provided to other participant clients in the publishing step, due to the introduction of the time delays. Such time delays may be intentionally added in any of the manner described herein, and / or may be directly resulting from the generation of such generated digital video streams. As a result, a participant client consuming a time-asynchronous generated output digital video stream consumes it in a “time zone” that is, for example, slightly time-offset to the “time zone” of other participant clients' video stream consumption.

[0271] For example, one of the participant client groups 121', 121'', or 121'''' may consume its respective generated video stream in a first "time zone," while another participant client 121 belonging to a different group 121', 121'', or 121'''' may consume its respective generated video stream in a second "time zone." Since both of these generated video streams may be generated based on at least partially the same primary video stream, all participant clients 121 operate within the same video communication but are located in different "time zones." In other words, the respective timelines for consuming the generated video streams may be temporally offset between different groups 121', 121'', or 121''''.

[0272] For example, some generation steps (135', 135'', etc.) may be direct (without intentionally introduced time delays) and / or involve only computationally relatively lightweight processing before public delivery. On the other hand, other generation steps (135'', etc.) may involve intentionally introduced time delays and / or relatively heavy processing, resulting in the generated digital video stream being produced for the earliest possible publication with a delay relative to the earliest possible delay time for publication of the corresponding digital video streams in the former generation steps 135', 135''.

[0273] Therefore, each participant client 121 belonging to one or more of groups 121', 121'', or 121'''' may be able to interact with each other with the same perceived time delay. At the same time, groups associated with a larger time delay can use the generated video stream from groups with a smaller time delay as an input video stream when generating an output video stream that the groups with a larger time delay will view in their later "time zone".

[0274] The result of this first, larger time-delayed generation (e.g., performed by generation function 135'') is therefore a generated digital video stream of the type described above, which may visually include, for example, one or more of the primary video streams as subparts, in processed or unprocessed form. The generated video stream may include live-captured video streams, slides, externally provided videos and images, etc., as described above with respect to the video output stream generated by the central server 130. It may also be generated based on detected events and / or patterns of an intentionally delayed or real-time input primary video stream provided by the participant client 121, in accordance with the general methods described above.

[0275] In an exemplary example, the participant clients of the first group 121' are part of the discussion panel and communicate using the video communication service 110 with relatively low latency. Each of these participant clients is continuously supplied with a generated video stream (or each other's primary video stream) from the public function 136'. The audience of the discussion panel consists of participant clients of the second group 121'', which are continuously supplied with generated video streams from the generation step 135'', these associated with slightly higher latency. The generated video streams from the generation step 135'' are automatically generated in the general manner described above and automatically shift between the views of individual discussion panel speakers (participant clients assigned to the first group 121', whose views are provided directly from the collection function 131) and the generated view (this view is the first generated video stream) that displays all discussion panel speakers. Thus, the audience receives a high-quality video experience, while the discussion panel speakers are able to interact with each other with minimal latency.

[0276] The delay intentionally added to each primary video stream used by the generation function 136'' is at least 0.1 seconds (e.g., at least 0.2 seconds, at least 0.5 seconds) and may be up to 5 seconds (e.g., up to 2 seconds, up to 1 second). This also depends on the delay inherent in each primary video stream to achieve perfect time synchronization between each primary video stream used and between the generated video stream input from generation step 135' to generation step 135''.

[0277] As outlined above, the primary video stream and the generated video stream from generation step 135' may have an additional intentional delay added to improve pattern detection used in the second generation function 135''.

[0278] Based on the above modifications, embodiments, and examples, some illustrative and non-limiting scenarios are listed here. In the following scenarios, the term “conversion” may refer to converting a video to a video having a desired frame rate and / or resolution, including, for example, receiving video, acquiring video frames, and generating generated video frames. Accordingly, conversion will be described herein with reference to the first and / or second aspects. Furthermore, the term “conversion” may also refer to the conversion of files such as web pages, documents, spreadsheets, and PDF files, as described in one embodiment of this specification (for example, according to the example of the third aspect of this specification).

[0279] Files / videos are converted and shared within a collaboration session, for example, by a user or triggered as described herein. Converted files / videos may be shared for a specific time zone of the collaboration session. Once the converted files / videos are saved, they may be shared again in subsequent collaboration sessions, for example, by a user, whether or not further participants in the collaboration session can interact with them. Furthermore, converted videos may be used as a primary stream for creating secondary streams.

[0280] • Files / videos are converted and shared, for example, within a specific time zone of the collaboration session. The converted files / videos may then be shared, for example, by the user, in a way that allows or does not allow further participants in the collaboration session to interact with them (e.g., using LiveShare).

[0281] The file / video is converted and shared, for example, in a specific time zone of the collaboration session. The converted file / video (i.e., video) may be fed into the generation function with or without intentional delay. The thus generated video stream may then be shared by the user, for example, with or without interaction with further participants in the same or a different collaboration session in the same or a different time zone.

[0282] In one embodiment of this specification, it is not possible to convert only videos stored in a storage function (e.g., cloud storage). It is also possible to convert a primary stream to obtain a converted video, for example, according to the first and second embodiments. The converted video is then transferred to the generation function along with one or more further primary streams. Alternatively or additionally, the converted video may be shared by the user, whether or not it can be interacted with by any further participants in the collaboration session. As a further option or addition, the converted video may be saved for future use, for example, as described herein.

[0283] Referring to the above items, multiple primary streams may, for example, generate a generated stream by using a generation function. The stream thus generated may be transformed, for example, according to the first and / or second aspects described herein. This generated stream may then be fed to a further generation function and shared, stored, or similarly processed by the user, whether or not it can be interacted with by further participants in any of the collaboration sessions.

[0284] The aspects and embodiments of this specification are summarized below with reference to the description and Figures 13 to 17. The same reference numerals are used to indicate the same or corresponding features in various examples.

[0285] Figure 13 shows a schematic block diagram illustrating an example of a central server 130.

[0286] The central server 130 may include, for example, a processing module 7101 which is a means for carrying out the method described herein. This means may be embodied in the form of one or more hardware modules and / or one or more software modules. Thus, the term “module” may refer to circuits, software blocks, etc., depending on the various embodiments described below.

[0287] The central server 130 may also be equipped with memory 7102. The memory may be equipped with, for example, instructions that include computer-readable code units in the form of computer programs 7103.

[0288] According to one embodiment of this specification, the central server 130 and / or processing module 7101 comprises a processing circuit 7104 as an exemplary hardware module, which may include one or more processors. Thus, the processing module 7101 is embodied in the form of the processing circuit 7104, or realized by the processing circuit 7104. Instructions are executable by the processing circuit 7104, thereby enabling the central server 130 to perform the corresponding operations described herein. As another example, when instructions are executed by the central server 130 and / or the processing circuit 7104, the central server 130 can be made to perform the operations that the central server 130 performs as described herein.

[0289] From the above perspective, a central server 130 is provided as an example. Again, the memory 7102 contains instructions that can be executed by the processing circuit 7104, thereby causing the central server 130 to operate to perform the operations described herein.

[0290] Figure 13 further shows a carrier 7105, or program carrier, that provides (e.g., equips, mediates, supplies, etc.) the computer program 7103 described above. The carrier 7105 may be an electronic signal, an optical signal, a wireless signal, or a computer-readable medium.

[0291] In some embodiments, the processing module 7101 includes one or more modules, each configured to perform one or more operations described herein.

[0292] The term "module" may refer to a circuit when it refers to a hardware module. In other examples, one or more of the exemplary hardware modules described above may be implemented as one or more software modules.

[0293] Furthermore, the central server 130 and / or processing module 7101 may include an input / output module 7106, which may be exemplified by a receiving module and / or transmitting module, where applicable.

[0294] Therefore, according to the various examples above, the central server 130 and / or processing module 7101 are configured to perform the operations described herein.

[0295] Figure 14 shows a schematic block diagram illustrating an example of the preparation function 111.

[0296] The preparation function 111 may include a processing module 7101, such as means for performing the methods described herein. This means may be embodied in the form of one or more hardware modules and / or one or more software modules. Thus, the term “module” may refer to circuits, software blocks, etc., depending on the various embodiments described below.

[0297] The preparation function 111 may further include a memory 7102. The memory may include or store instructions, for example, in the form of a computer program 7103, which include computer-readable code units.

[0298] According to one embodiment of this specification, the preparation function 111 and / or processing module 7101 comprises a processing circuit 7104 as an exemplary hardware module, which may include one or more processors. Thus, the processing module 7101 is embodied in the form of the processing circuit 7104, or realized by the processing circuit 7104. Instructions are executable by the processing circuit 7104, thereby causing the preparation function 111 to perform the corresponding operations described herein. As another example, when instructions are executed by the preparation function 111 and / or processing circuit 7104, the preparation function 111 performs the operations that the preparation function 111 performs as described herein.

[0299] From the above perspective, a preparation function 111 is provided as an example. Again, the memory 7102 includes instructions that can be executed by the processing circuit 7104, thereby causing the preparation function 111 to operate to perform the operations described herein.

[0300] Figure 14 further shows a carrier 7105, or program carrier, that provides (e.g., equips, mediates, supplies, etc.) the computer program 7103 described above. The carrier 7105 may be an electronic signal, an optical signal, a wireless signal, or a computer-readable medium.

[0301] In some embodiments, the processing module 7101 includes one or more modules, each configured to perform one or more operations described herein. The term “module” may refer to a circuit when referring to a hardware module. In other examples, one or more of the exemplary hardware modules described above may be implemented as one or more software modules.

[0302] Furthermore, the preparation function 111 and / or processing module 7101 may include an input / output module 7106, which may be exemplified by a receiving module and / or transmitting module, where applicable.

[0303] Therefore, according to the various examples above, the preparation function 111 and / or processing module 7101 are configured to perform the operations described herein.

[0304] Figure 15 shows a schematic block diagram illustrating an example of add-on backend 113.

[0305] The add-on backend 113 may include a processing module 7101, such as means for performing the methods described herein. This means may be embodied in the form of one or more hardware modules and / or one or more software modules. Thus, the term “module” may refer to circuits, software blocks, etc., depending on the various embodiments described below.

[0306] The add-on backend 113 may further include memory 7102. The memory may include or store instructions, for example, in the form of a computer program 7103, which includes computer-readable code units.

[0307] According to one embodiment of this specification, the add-on backend 113 and / or processing module 7101 comprises a processing circuit 7104, which is a hardware module as an example, and which may include one or more processors. Thus, the processing module 7101 is embodied in the form of the processing circuit 7104, or realized by the processing circuit 7104. Instructions are executable by the processing circuit 7104, thereby enabling the add-on backend 113 to perform the corresponding operations described herein. As another example, instructions, when executed by the add-on backend 113 and / or processing circuit 7104, can cause the add-on backend 113 to perform the operations of the add-on backend 113 described herein.

[0308] From the above perspective, an add-on backend 113 is provided as an example. Again, the memory 7102 contains instructions that can be executed by the processing circuit 7104, thereby causing the add-on backend 113 to operate to perform the operations described herein.

[0309] Figure 15 further shows a carrier 7105, or program carrier, that provides (e.g., equips, mediates, supplies, etc.) the computer program 7103 described above. The carrier 7105 may be an electronic signal, an optical signal, a wireless signal, or a computer-readable medium.

[0310] In some embodiments, the processing module 7101 includes one or more modules, each configured to perform one or more operations described herein. The term “module” may refer to a circuit when referring to a hardware module. In other examples, one or more of the exemplary hardware modules described above may be implemented as one or more software modules.

[0311] Furthermore, the add-on backend 113 and / or processing module 7101 may include an input / output module 7106, which may be exemplified by a receive module and / or transmit module, where applicable.

[0312] Accordingly, following the various examples described above, the add-on backend 113 and / or processing module 7101 are configured to perform the operations described herein.

[0313] Figure 16 shows a schematic block diagram illustrating an example of the shared function 112.

[0314] The shared function 112 may include a processing module 7101, such as means for performing the methods described herein. This means may be embodied in the form of one or more hardware modules and / or one or more software modules. Thus, the term “module” may refer to circuits, software blocks, etc., depending on the various embodiments described below. The shared function 112 may further include a memory 7102. The memory may include or store instructions, for example, in the form of a computer program 7103, which include computer-readable code units.

[0315] According to one embodiment of this specification, the shared function 112 and / or processing module 7101 comprises a processing circuit 7104 as an exemplary hardware module, which may include one or more processors. Thus, the processing module 7101 is embodied in the form of the processing circuit 7104, or implemented by the processing circuit 7104. Instructions are executable by the processing circuit 7104, thereby causing the shared function 112 to perform the corresponding operations described herein. As another example, if instructions are executed by the shared function 112 and / or the processing circuit 7104, the shared function 112 can be caused to perform the operations described herein.

[0316] From the above perspective, a shared function 112 is provided as an example. Again, the memory 7102 includes instructions that can be executed by the processing circuit 7104, thereby causing the shared function 112 to operate to perform the operations described herein.

[0317] Figure 16 further shows a carrier 7105, or program carrier, that provides (e.g., equips, mediates, supplies, etc.) the computer program 7103 described above. The carrier 7105 may be an electronic signal, an optical signal, a wireless signal, or a computer-readable medium.

[0318] In some embodiments, the processing module 7101 includes one or more modules, each configured to perform one or more operations described herein. The term “module” may refer to a circuit when referring to a hardware module. In other examples, one or more of the exemplary hardware modules described above may be implemented as one or more software modules.

[0319] Furthermore, the shared function 112 and / or processing module 7101 may include an input / output module 7106, which may be exemplified by a receiving module and / or transmitting module, where applicable.

[0320] Therefore, according to the various examples above, the shared function 112 and / or processing module 7101 are configured to perform the operations described herein.

[0321] Figure 17 shows a schematic block diagram of an example of computing function 600 (e.g., a computer, server, virtual machine, etc.).

[0322] The computing function 600 may include a processing module 7101 which is a means for performing the methods described herein. This means may be embodied in the form of one or more hardware modules and / or one or more software modules. Thus, the term “module” may refer to circuits, software blocks, etc., depending on the various embodiments described below. The computing function 600 may further include a memory 7102. The memory may be configured to include or store instructions, for example, in the form of a computer program 7103, which includes computer-readable code units.

[0323] According to one embodiment of this specification, the arithmetic function 600 and / or processing module 7101 includes a processing circuit 7104 as an exemplary hardware module, which may include one or more processors. Thus, the processing module 7101 is embodied in the form of the processing circuit 7104 or realized by the processing circuit 7104. Instructions are executable by the processing circuit 7104, thereby enabling the arithmetic function 600 to operate to perform the corresponding operations described herein. As another example, instructions, when executed by the arithmetic function 600 and / or processing circuit 7104, can cause the arithmetic function 600 to perform the operations of the arithmetic function 600 described herein.

[0324] From the above perspective, a computing function 600 is provided as an example. Again, the memory 7102 includes instructions that can be executed by the processing circuit 7104, thereby enabling the arithmetic function 600 to perform the operations described herein.

[0325] Figure 17 further shows a carrier 7105, or program carrier, that provides (e.g., equips, mediates, supplies, etc.) the computer program 7103 described above. The carrier 7105 may be an electronic signal, an optical signal, a wireless signal, or a computer-readable medium.

[0326] In some embodiments, the processing module 7101 includes one or more modules, each configured to perform one or more operations described herein. The term “module” may refer to a circuit when referring to a hardware module. In other examples, one or more of the exemplary hardware modules described above may be implemented as one or more software modules.

[0327] Furthermore, the computing function 600 and / or processing module 7101 may include an input / output module 7106, which may be exemplified by a receiving module and / or transmitting module, where applicable.

[0328] Therefore, according to the various examples above, the computing function 600 and / or processing module 7101 are configured to perform the operations described herein.

[0329] In this specification, the term “frame” can generally refer to a frame of a video stream that contains one or more streams of different types. Therefore, the term “frame” can refer to a video frame, an audio frame, etc., as understood from the context in which the term is used. For example, a frame may be an uncompressed, decoded pixel map.

[0330] In this specification, the term "tick" may refer to a computer clock or a unit of time measurement used by an operating system. A tick is typically the smallest unit of time that an operating system or processor can measure and is frequently used as a basis for scheduling tasks, allocating system resources, and measuring time. For example, a tick may be incremented every millisecond, and the operating system may use this tick to track the execution time of each process and schedule tasks accordingly.

[0331] Depending on the context, "tick" can also refer to a specific type of software interrupt that an operating system uses to process hardware events or perform specific tasks. These interrupts are often triggered by the computer's hardware clock, which generates signals at regular intervals.

[0332] In this specification, the term "function," including central server, preparation function, sharing function, and storage function, refers to a computer-implemented functionality configured to be accessed in a logically centralized manner, such as through an API (Application Programming Interface). Such functionality may be implemented purely in computer software, or as a combination of software and virtual / physical hardware. It may be implemented on a single physical / virtual server computer, or distributed across multiple interconnected physical / virtual server computers.

[0333] In this specification, terms such as “add-on,” “add-on software,” “add-in,” “plugin,” and “add-on software functionality” may refer to pieces of software that are loaded (preferably dynamically) into a client function, for example, as code that runs within a so-called iframe embedded in the client function. Add-ons are typically added to a collaboration session, which then runs the corresponding instance of the synchronous add-on for all participants in the collaboration session. The use of add-ons is widely known and documented in the relevant literature.

[0334] In this specification, the terms “computer program carrier,” “program carrier,” or “carrier” may refer to an electronic signal, an optical signal, a radio signal, or a computer-readable medium. In some examples, a computer program carrier may exclude transient and propagating signals such as electronic signals, optical signals, and / or radio signals. Therefore, in these examples, a computer program carrier may be a non-transient carrier, such as a non-transient computer-readable medium.

[0335] In this specification, the term “processing module” may include one or more hardware modules, one or more software modules, or a combination thereof. Whether hardware, software, or a combination of hardware and software modules, any such unit may be a determination means, estimation means, capture means, association means, comparison means, identification means, generation means, retrieval means, receiving means, transmitting means, acquisition means, etc., as disclosed herein. For example, the expression “means” may refer to a module corresponding to one of the modules listed above, in conjunction with the drawings.

[0336] In this specification, the term “software module” may refer to a software application, a dynamic link library (DLL), a software component, a software container, a software object, an object based on the Component Object Model (COM), a software function, a virtual machine, a software engine, an executable binary software file, and the like.

[0337] In this specification, terms such as "backend" and "addon backend" may refer to server functions, server-side functions, etc., associated with an addon. The backend and addon are configured to cooperate and communicate with each other as described herein with reference to the attached diagrams.

[0338] In this specification, the term "processing circuit" may encompass a processing unit including, for example, one or more processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. A processing circuit, etc., may include one or more processor kernels.

[0339] In this specification, the term “local storage” may refer to computer-readable and / or writable media such as hard drives, solid-state drives, magnetic tapes, and hard disks that are located near the Device, for example, connected by short-range wired or wireless technology and situated within the Device's enclosure.

[0340] In this specification, the term “remote storage” may refer to computer-readable and / or writable media such as hard drives, solid-state drives, magnetic tapes, hard disks, cloud storage functions, and online storage services. These are located away from this device, for example, connected by long-range wired or wireless technology and sometimes installed in data centers.

[0341] In this specification, the expression "configured / configured for" may mean that a processing circuit is adapted or configured to perform one or more operations described herein by a software configuration and / or hardware configuration.

[0342] In this specification, the term "action" may refer to actions, steps, operations, responses, reactions, activities, etc. Note that, depending on the application, an action may be divided into two or more sub-actions. Furthermore, depending on the application, two or more actions described herein may be combined into a single action.

[0343] In this specification, the term "memory" may refer to hard disks, magnetic storage media, floppy disks or disks for portable computers, flash memory, random access memory (RAM), etc. Furthermore, the term "memory" may also refer to the internal register memory of a processor or similar device.

[0344] In this specification, the term "input unit" may refer to a keyboard, mouse, touchpad, stylus, touchscreen, etc.

[0345] In this specification, the term “output unit” may refer to a computer screen, liquid crystal display, printer, projection device that projects a computer’s graphical user interface onto a flat surface such as a wall, or other commonly used computer output device.

[0346] In this specification, the expressions “more, fewer, or the same number of video frames” may be interpreted as per unit time, per video, per specific portion of video, or similar.

[0347] In this specification, terms such as "message" and "msg" may refer to any information provided (e.g., transmitted) by one entity, function, device, etc., and intended for consumption (e.g., received) by another entity. A message may have any appropriate format, with or without headers. For example, a message may include binary bits, values, strings, objects, JavaScript notation objects, formatted bit arrays, formatted strings, files, and the like.

[0348] In this specification, the term “computer-readable media” can include, but is not limited to, Universal Serial Bus (USB) memory, Digital Versatile Discs (DVDs), Blu-ray Discs, software units received as data streams, flash memory, hard drives, memory cards such as Memory Sticks, Multimedia Cards (MMCs), and Secure Digital (SD) cards. One or more of the above-mentioned examples of computer-readable media may be provided as one or more computer program products.

[0349] As used herein, the term “computer-readable code unit” may refer to the text of a computer program, part or all of a binary file representing a computer program in compiled form, or anything in between.

[0350] In this specification, terms such as "first," "second," and "third" may be used simply to distinguish features, devices, elements, units, etc. from one another, unless otherwise evident from the context.

[0351] When the term "resolution" is mentioned, it may refer to the total number of pixels in one or more dimensions in an image coordinate system, and / or a specific number of pixels in relation to other coordinate systems, such as a screen coordinate system or a coordinate system that measures physical space in conventional units of length. Therefore, the "resolution" of an image may be, for example, "100x200 pixels" or "300 pixels per inch."

[0352] Features described with reference to one or more examples can be readily combined with one or more other examples, where applicable.

[0353] The embodiments described herein are illustrative and not intended to limit the scope of protection defined by the appended claims.

[0354] List of examples according to the first aspect 1. A method for managing the sharing of video in a collaboration session, performed by a system (100), wherein at least two client devices (121, 122) are participating, the system (100) comprises the two client devices (121) and a central server (130) capable of hosting the collaboration session, the video files representing the video and stored in a storage function (140) are accessible by the central server (130), the video is shared as a shared video stream having a desired resolution and a desired frame rate, and the method includes: The central server (130) receives a request to share the video in the collaboration session (B120); The central server (130) obtains the synchronization timestamp of the audio frame associated with the collaboration session, which is used to synchronize the shared video stream with the collaboration session (B122); The central server (130) determines, based on the video frame rate difference between the desired frame rate and the source frame rate of the video file, whether the shared video stream requires more, fewer, or the same number of video frames as the video file (B124); The central server (130) sets up a buffer (B130) for providing the shared video stream to the collaboration session, the buffer being capable of buffering a limited number of video frames corresponding to time intervals of less than one second at most; The central server (130) iteratively acquires each video frame of the video file (B140), and, For each video frame, the method includes the following: The central server (130) generates (B150) zero, one, or more video frames based on each of the iteratively acquired video frames according to the determination step (B124), and sets the timestamp of each of the one or more generated video frames based on the synchronization timestamp, wherein the one or more generated frames have the desired resolution and each timestamp corresponds to the desired frame rate. Here, the method includes the following: The central server (130) provides the shared video stream to the two client devices (121) participating in the collaboration session by inserting the one or more generated video frames into the buffer (B160).

[0355] 2. The method according to Example 1, wherein the method includes: A method for obtaining information (C130) from the central server (130) regarding the desired resolution and the desired frame rate of a shared video stream representing the shared video.

[0356] 3. The method described in Example 1 or Example 2, wherein the method includes: In a further decision step (B126), the central server (130) determines whether the shared video stream requires more, fewer, or the same number of video frames as the video file, based on the further video frame rate difference between the further desired frame rate and the source frame rate of the video file (B126), Here, for each video frame acquired iteratively, the method includes the following: In a further generation step (D160), the central server (130) generates zero, one, or more video frames based on each of the iteratively acquired video frames according to a further determination step (B126), while setting the timestamp of each of the generated one or more video frames based on the synchronization timestamp, wherein the generated one or more frames have the further desired resolution, and each of the timestamps corresponds to the further desired frame rate; A method for providing the second video stream (C197) by having the central server (130) transmit each of the further video frames to a destination function (150).

[0357] 4. A method according to any one of Examples 1 to 3, wherein the method includes: A method to obtain an update message (C190) instructing the central server (130) to set the desired video frame rate and the desired resolution based on the information contained in the message, thereby obtaining the updated desired video frame rate and the updated desired resolution.

[0358] 5. A central server (130) configured to perform operations performed by the central server (130) in the method of any one of Examples 1 to 4.

[0359] 6. A computer program (7103) which, when executed on a central server (130), includes a computer-readable code unit that causes the central server (130) to perform an operation performed by the central server (130) in the method of any one of Examples 1 to 4.

[0360] 7. A carrier (7105) containing the computer program described in Example 6, wherein the carrier (7105) is one of the following: an electronic signal, an optical signal, a wireless signal, a computer-readable medium, or a computer program product.

[0361] List of examples according to the second aspect 1. A method for managing video sharing in a collaboration session, performed by a system (100), wherein at least two client devices (121, 122) are participating, and the system (100) comprises a video communication service (110) capable of hosting the collaboration session, two client functions (125) executable on each of the two client devices (121, 122), and two add-on functions (127) associated with each of the two client functions (125). The method comprises a storage function (140) and an add-on backend function (113) configured to cooperate and communicate with the two add-on functions (127), and a sharing function (112) configured to cooperate and communicate with the add-on backend function (113) and to participate in the collaboration session, wherein a video file representing the video and stored in the storage function (140) is accessible from the sharing function (112), and the video is to be shared as a shared video stream at a desired resolution and a desired frame rate, and the method includes the following: The add-on backend function (113) receives a request to share the video in the collaboration session (B120); The sharing function (112) is connected to the collaboration session (B110); The sharing function (112) retrieves the synchronization timestamp of the audio frame of the collaboration session (B122), where the audio frame is to be used to synchronize the shared video stream with the collaboration session; In the determination step (B124), the sharing function (112) determines whether the shared video stream needs more, fewer, or the same number of video frames as the video file, based on the video frame rate difference between the desired frame rate and the source frame rate of the video file (B124); The sharing function (112) iteratively acquires each video frame of the video file stored in the storage function (140) (B140), and, For each video frame acquired iteratively, the method includes: Based on each of the iteratively acquired video frames, zero, one, or more video frames are generated (B150) according to the determination step (B124), and the timestamp of each of the one or more generated video frames is set based on the synchronization timestamp, where the one or more generated frames have the desired resolution and each of the timestamps corresponds to the desired frame rate. Here, the method includes the following: The sharing function (112) provides the shared video stream by transmitting the one or more generated video frames to the video communication service (110) (B160); and, The video communication service (110) provides the shared video stream to the two client devices (121) participating in the collaboration session (B174).

[0362] 2. The method according to Example 1 includes the following: A method for obtaining information (C130) from the central server (130) regarding the desired resolution and the desired frame rate of a shared video stream representing the shared video.

[0363] 3. A sharing function (112) configured to perform an operation performed by the sharing function (112) in the method described in Example 1 or Example 2.

[0364] 4. A computer program (7103) which, when executed on a shared function (112), includes a computer-readable code unit that causes the shared function (112) to perform an operation performed by the shared function (112) in the method described in Example 1 or Example 2.

[0365] 5. A carrier (7105) comprising the computer program described in Example 4, wherein the carrier (7105) is one of the following: an electronic signal, an optical signal, a wireless signal, a computer-readable medium, and a computer program product.

[0366] 6. An add-on backend (113) configured to perform an operation performed by the add-on backend (113) in the method described in Example 1 or Example 2.

[0367] 7. A computer program (7103) which, when executed on an add-on backend (113), includes a computer-readable code unit that causes the add-on backend (113) to perform an operation performed by the add-on backend (113) in the method described in Example 1 or Example 2.

[0368] 8. A carrier (7105) comprising the computer program described in Example 7, wherein the carrier (7105) is one of the following: an electronic signal, an optical signal, a wireless signal, a computer-readable medium, and a computer program product.

Claims

1. A computer implementation method for managing files, the method comprising: The rendering function (610) is used to open the file (E110), and at least a portion of the file is rendered as an image; Retrieve one or more event messages (E120), Here, for each of the one or more event messages, the method includes the following: From the rendering function (610), an image corresponding to each event message is obtained (E130), where the image represents the rendered portion of the file and is associated with a sequence instruction; Based on the event message and the sequence instruction, metadata is generated (E140); Provide the metadata (E150); and, A video is provided (E160) which includes a set of images, wherein the set of images includes the image corresponding to each of the event messages, and the video is a representation of the file.

2. A method according to claim 1, wherein the method is performed by a client device (121), the system (100) comprises the client device (121) and a video communication service (110), the video communication service (110) hosts a collaboration session in which the client device (121) participates.

3. A method according to claim 1 or 2, wherein the one or more event messages are received from the rendering function (610) and the one or more event messages relate to a change in the image rendered by the rendering function (610) due to an input event that can be provided, for example, by a user input device.

4. The method according to any one of claims 1 to 3, wherein the method includes: A method for simulating one or more event messages (E111) by generating input events that modify the image rendered by the rendering function (610).

5. A method according to any one of claims 1 to 4, wherein the input event arises from an actual interaction between the user and the rendered file using a user input device such as a mouse, keyboard, or touchpad.

6. The method according to any one of claims 1 to 5, wherein the method includes: The rendering function (610) repeatedly captures the image (E112); and, A method for generating one or more event messages (E118) when a difference is detected between two consecutively captured images, wherein the event message indicates that the two consecutively captured images are different from each other.

7. A method according to any one of claims 1 to 6, the method comprising: The rendering function (610) repeatedly captures the image (E112); and, If no difference is detected between two consecutively captured images, the method generates one or more event messages (E118), where the event messages indicate that the two consecutively captured images are similar to each other.

8. A method according to any one of claims 1 to 7, wherein the one or more event messages include an instruction to start the rendering function (610) iteratively capturing an image, or an instruction to stop the rendering function (610) iteratively capturing an image, the method comprising: When the instruction to start is received, the rendering function (610) is used to repeatedly capture images (E135); and, A method for generating metadata for each captured image based on the event message and the respective sequence instruction until the instruction to stop is received (E140).

9. The method according to any one of claims 1 to 8, wherein the method includes: The audio stream of the collaboration session is received from the video communication service (110) (E115); Detect keywords in the audio stream (E117); and, A method for generating event messages for one or more event messages based on the aforementioned keywords (E118).

10. A computing function (600) configured to perform the method described in any one of claims 1 to 9.

11. A computer program (7103) comprising a computer-readable code unit that, when executed on a computing function (600), causes the computing function (600) to perform the method described in any one of claims 1 to 9.

12. A carrier (7105) comprising the computer program described in claim 11, wherein the carrier (7105) is one of an electronic signal, an optical signal, a wireless signal, a computer-readable medium, and a computer program product.

Images