System and method for producing video stream
Patent Information
- Application Number
- IN202417031460
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-10-15
- Filing Date
- 2024-04-19
- Publication Date
- 2026-08-04
- Estimated Expiration
- 2042-10-14
AI Technical Summary
Current digital video conference systems face challenges in producing a cohesive and synchronized digital video stream due to differences in latency, frame rates, aspect ratios, resolutions, and encoding formats among incoming video streams, leading to poor user experiences and increased hardware requirements.
A method and system that collect, time-synchronize, and analyze digital video streams to detect events and patterns, producing a shared digital video stream based on consecutively considered frames and detected patterns, while continuously providing the output stream to consumers.
This approach reduces latency and synchronization issues, enhances user experience by managing variable stream characteristics, and optimizes hardware requirements, effectively handling complex multi-party video meetings and educational contexts.
Abstract
Description
The present invention relates to a system, computer software product and method for producinga digital video stream, and in particular for producing a digital video stream basedon two or more different digital input video streams. In preferred embodiments, the digitalvideo stream is produced in the context of a digital video conference or a digital video conferenceor meeting system, particularly involving a plurality of different concurrent users.The produced digital video stream may be published externally or within a digital video conferenceor digital video conference system.In other embodiments, the present invention is applied in contexts that are not digital videoconferences, but where several digital video input streams are handled concurrently andcombined into a produced digital video stream. For instance, such contexts may be educationalor instructional.There are many known digital video conference systems, such as Microsoft Teams,Zoom and Google Meet, offering two or more participants to meet virtually using digitalvideo and audio recorded locally and broadcast to all participants to emulate a physicalmeeting.There is a general need to improve such digital video conference solutions, in particular withrespect to the production of viewed content, such as what is shown to whom at what time,and via what distribution channels.For instance, some systems automatically detect a currently talking participant, and showthe corresponding video feed of the talking participant to the other participants. In manysystems it is possible to share graphics, such as the currently displayed screen, a viewingwindow or a digital presentation. As virtual meetings become more complex, however, itquickly becomes more difficult for the service to know what of all currently available informationto show to each participant at each point in time.In other examples a presenting participant moves around on a stage while talking aboutslides in a digital presentation. The system then needs to decide whether to show thepresentation, the presenter or both, or to switch between the two.It may be desirable to produce one or several output digital video streams based on a numberof input digital video streams by an automatic production process, and to provide suchproduced digital video stream or streams to one or several consumers.However, in many cases it is difficult for a dynamic conference screen layout manager orother automated production function to select what information to show, due to a numberof technical difficulties facing such digital video conference systems.Firstly, since a digital video meeting has a real-time aspect, it is important that latency islow. This poses problems when different incoming digital video streams, such as from dif15ferent participants joining using different hardware, are associated with different latencies,frame rates, aspect ratios or resolutions. Many times, such incoming digital video streamsneed processing for a well-formed user experience.Secondly, there is a problem with time synchronisation. Since the various input digital videostreams, such as external digital video streams or digital video streams provided by participants,are typically fed to a central server or similar, there is no absolute time to synchroniseeach such digital video feed to. Like too high latency, unsynchronised digital video feeds willlead to poor user experiences.Thirdly, multi-party digital video meetings can involve different digital video streams havingdifferent encodings or formats, that require decoding and re-encoding in turn producingproblems in terms of latency and synchronisation. Such encoding is also computationallyburdensome and therefore costly in terms of hardware requirements.Fourthly, the fact that different digital video sources may be associated with different framerates, aspect ratios and resolutions may also result in that memory allocation needs mayvary unpredictably requiring continuous balancing. This potentially results in additional latencyand synchronisation problems. The result is large buffer requirements.Fifthly, participants may experience various challenges in terms of variable connectivity,leaving / reconnecting etc., posing further challenges in automatically producing a wellformeduser experience.These problems are amplified in more complex meeting situations, for instance involvingmany participants; participants using different hardware and / or software to connect; ex10ternally provided digital video streams; screen-sharing; or multiple hosts.The corresponding problems arise in said other contexts where an output digital videostream is to be produced based on several input digital video streams, such as in digitalvideo production systems for education and instruction.The present invention solves one or several of the above described problems.Hence, the invention relates to a method for providing a shared digital video stream, themethod comprises the steps in a collecting step, collecting from at least two digital videosources respective primary digital video streams; in an event detection step, individuallyanalysing said primary digital video streams to detect at least one event selected from afirst set of events; in a synchronising step, time-synchronising said primary digital videostreams with respect to a common time reference; in a pattern detection step, analysingthe time-synchronised primary digital video streams to detect at least one pattern selectedfrom a first set of patterns, the pattern detection being based on said detected at least oneevent; in a production step, producing the shared digital video stream as an output digitalvideo stream based on consecutively considered frames of said time-synchronised primarydigital video streams and said detected patterns; and in a publishing step, continuouslyproviding said output digital video stream to a consumer of the shared digital video stream.The invention also relates to a computer software product for providing a shared digitalvideo stream, the computer software function being arranged to, when executing, performa collecting step, wherein respective primary digital video streams are collected from atleast two digital video sources; an event detection step, wherein said primary digital videostreams are individually analysed to detect at least one event selected from a first set ofevents; a synchronising step, wherein said primary digital video streams are time-synchronisedwith respect to a common time reference; a pattern detection step, wherein the timesynchronisedprimary digital video streams are analysed to detect at least one pattern selectedfrom a first set of patterns, the pattern detection being based on said detected atleast one event; a production step, wherein the shared digital video stream is produced asan output digital video stream based on consecutively considered frames of said time-synchronisedprimary digital video streams and said detected patterns; and a publishing step,wherein said output digital video stream is continuously provided to a consumer of theshared digital video stream.Moreover, the invention relates to a system for providing a shared digital video stream, thesystem comprising a central server in turn comprising a collecting function, arranged to collectfrom at least two digital video sources respective primary digital video streams; anevent detection function, arranged to individually analyse said primary digital video streamsto detect at least one event selected from a first set of events; a synchronising function,arranged to time-synchronise said primary digital video streams with respect to a commontime reference; a pattern detection function, arranged to analyse the time-synchronisedprimary digital video streams to detect at least one pattern selected from a first set of patterns,the pattern detection being based on said detected at least one event; a productionfunction, arranged to produce the shared digital video stream as an output digital videostream based on consecutively considered frames of said time-synchronised primary digitalvideo streams and said detected patterns; and a publishing function, arranged to continuouslyprovide said output digital video stream to a consumer of the shared digital videostream.In the following, the invention will be described in detail, with reference to exemplifyingembodiments of the invention and to the enclosed drawings, wherein:Figure 1 illustrates a first system according to the invention;Figure 2 illustrates a second system according to the invention;Figure 3 illustrates a third system according to the invention;Figure 4 illustrates a central server according to the invention;Figure 5 illustrates a central server for use in a system according to the invention;Figures 6a-6f illustrate subsequent states in relation to the different method steps in themethod illustrated in Figure 5; andFigure 7 illustrates, conceptually, a common protocol for used in the present invention.All Figures share reference numerals for the same or corresponding parts.Figure 1 illustrates a system 100 according to the present invention, arranged to perform amethod according to the invention for providing a shared digital video stream.The system 100 may comprise a video communication service 110, but the video communicationservice 110 may also be external to the system 100 in some embodiments.The system 100 may comprise one or several participant clients 121, but one, some or allparticipant clients 121 may also be external to the system 100 in some embodiments.The system 100 comprises a central server 130.As used herein, the term "central server" is a computer-implemented functionality that isarranged to be accessed in a logically centralised manner, such as via well-defined API (ApplicationProgramming Interface). The functionality of such a central server may be implementedpurely in computer software, or in a combination of software with virtual and / orphysical hardware. It may be implemented on a standalone physical or virtual servercomputer or be distributed across several interconnected physical and / or virtual servercomputers.The physical or virtual hardware that the central server 130 runs on, in other words thatcomputer software defining the functionality of the central server 130 may comprise a perse conventional CPU, a per se conventional GPU, a per se conventional RAM / ROM memory,a per se conventional computer bus, and a per se conventional external communicationfunctionality such as an internet connection.The video communication service 110, to the extent it is used, is also a central server in saidsense, that may be a different central server than the central server 130 or a part of thecentral server 130.Correspondingly, each of said participant clients 121 may be a central server in said sense,with the corresponding interpretation, and physical or virtual hardware that each participantclient 121 runs on, in other words that computer software defining the functionalityof the participant client 121, may also comprise a per se conventional CPU / GPU, a per seconventional RAM / ROM memory, a per se conventional computer bus, and a per se conventionalexternal communication functionality such as an internet connection.Each participant client 121 also typically comprises or is in communication with a computerscreen, arranged to display video content provided to the participant client 121 as a part ofan ongoing video communication; a loudspeaker, arranged to emit sound content providedto the participant client 121 as a part of said video communication; a video camera; and amicrophone, arranged to record sound locally to a human participant 122 to said video communication,the participant 122 using the participant client 121 in question to participate insaid video communication.In other words, a respective human-machine interface of each participating client 121 al30lows a respective participant 122 to interact with the client 121 in question, in a videocommunication, with other participants and / or audio / video streams provided by varioussources.In general, each of the participating clients 121 comprises a respective input means 123,that may comprise said video camera; said microphone; a keyboard; a computer mouse ortrackpad; and / or an API to receive a digital video stream, a digital audio stream and / or otherdigital data. The input means 123 is specifically arranged to receive a video stream and / oran audio stream from a central server, such as the video communication service 110 and / orthe central server 130, such a video stream and / or audio stream being provided as a part ofa video communication and preferably being produced based on corresponding digital datainput streams provided to said central server from at least two sources of such digital datainput streams, for instance participant clients 121 and / or external sources (see below).Further generally, each of the participating clients 121 comprises a respective output means124, that may comprise said computer screen; said loudspeaker; and an API to emit a digitalvideo and / or audio stream, such stream being representative of a captured video and / oraudio locally to the participant 122 using the participant client 121 in question.In practice, each participant client 121 may be a mobile device, such as a mobile phone,arranged with a screen, a loudspeaker, a microphone and an internet connection, the mobiledevice executing computer software locally or accessing remotely executed computersoftware to perform the functionality of the participant client 121 in question. Correspondingly,the participant client 121 may also be a thick or thin laptop or stationary computer,executing a locally installed application, using a remotely accessed functionality via a webbrowser, and so forth, as the case may be.There may be more than one, such as at least three, participant clients 121 used in one andthe same video communication of the present type.The video communication may be provided at least partly by the video communication service110 and at least partly by the central server 130, as will be described and exemplifiedherein.As the term is used herein, a "video communication" is an interactive, digital communicationsession involving at least two, preferably at least three, video streams, and preferablyalso matching audio streams that are used to produce one or several mixed or joint digitalvideo / audio streams that in turn is consumed by one or several consumers, that may or maynot also be contributing to the video communication via video and / or audio. Such a videocommunication is real-time, with or without a certain latency or delay. At least one, preferablyat least two, participants 122 to such a video communication is involved in the videocommunication in an interactive manner, both providing and consuming video / audio information.At least one of the participant clients 121, or all of the participant clients 121, comprise alocal synchronisation software function 125, that will be described in closer detail below.The video communication service 110 may comprise or have access to a common time reference,as will also be described in closer detail below.The central server 130 may comprise an API 137, for digitally communicating with entitiesexternal to the central server 130. Such communication may involve both input and output.The system 100, such as the central server 130, may be arranged to digitally communicatewith, and in particular to receive digital information, such as audio and / or video streamdata, from, an external information source 300, such as an externally provided videostream. That the information source 300 is "external" means that it is not provided from oras a part of the central server 130. Preferably, the digital data provided by the externalinformation source 300 is independent of the central server 130, and the central server 130cannot affect the information contents thereof. For instance, the external informationsource 130 may be live captured video and / or audio, such as of a public sporting event oran ongoing news event or reporting. The external information source 130 may also be capturedby a web camera or similar, but not by any one of the participating clients 121. Suchcaptured video may hence depict the same locality as any one of the participant clients 121,but not be captured as a part of the activity of the participant client 121 per se. One possibledifference between an externally provided information source 300 and an internally providedinformation source 120 is that internally provided information sources may be providedas, and in their capacity as, participants to a video communication of the above-definedtype, whereas an externally provided information source 300 is not, but is insteadprovided as a part of a context that is external to said video conferenceThere may also be several external information sources 300, that provide digital informationof said type, such as audio and / or video streams, to the central server 130 in parallel.As shown in Figure 1, each of the participating clients 121 constitutes the source of an in15formation (video and / or audio) stream 120, provided to the video communication service110 by the participating client 121 in question as described.The system 100, such as the central server 130, may be further arranged to digitally communicatewith, and in particular to emit digital information to, an external consumer 150.For instance, a digital video and / or audio stream produced by the central server 130 maybe provided continuously, in real-time or near real-time, to one or several external consumers150 via said API 137. Again, that the consumer 150 is "external" means that the consumer150 is not provided as a part of the central server 130, and / or that it is not a party tothe said video communication.Unless not stated otherwise, all functionality and communication herein is provided digitallyand electronically, effected by computer software executing on suitable computer hardwareand communicated over a digital communication network or channel such as the internet.Hence, in the system 100 configuration illustrated in Figure 1, a number of participant clients121 take part in a digital video communication provided by the video communicationservice 110. Each participant client 121 may hence have an ongoing login, session or similarto the video communication service 110, and may take part in one and the same ongoingvideo communication provided by the video communication service 110. In other words,the video communication is "shared" among the participant clients 121 and therefore alsoby corresponding human participants 122.In Figure 1, the central server 130 comprises an automatic participant client 140, being anautomated client corresponding to participant clients 121 but not associated with a humanparticipant 122. Instead, the automatic participant client 140 is added as a participant clientto the video communication service 110 to take part in the same shared video communicationas participant clients 121. As such a participant client, the automatic participant client140 is granted access to continuously produced digital video and / or audio stream(s) pro15vided as a part of the ongoing video communication by the video communication service110, and can be consumed by the central server 130 via the automatic participant client140. Preferably, the automatic participant client 140 receives, from the video communicationservice 110, a common video and / or audio stream that is or may be distributed to eachparticipant client 121; a respective video and / or audio stream provided to the video com20munication service 110 from each of one or several of the participant clients 121 and relayed,in raw or modified form, by the video communication service 110 to all or requestingparticipant clients 121; and / or a common time reference.The central server 130 comprises a collecting function 131 arranged to receive video and / oraudio streams of said type from the automatic participant client 140, and possibly also fromsaid external information source(s) 300, for processing as described below, and then to providea shared video stream via the API 137. For instance, this shared video stream may beconsumed by the external consumer 150 and / or by the video communication service 110to in turn be distributed by the video communication service 110 to all or any requestingone of the participant clients 121.Figure 2 is similar to Figure 1, but instead of using the automatic client participant 140 thecentral server 130 receives video and / or audio stream data from the ongoing video communicationvia an API 112 of the video communication service 110.Figure 3 is also similar to Figure 1, but shows no video communication service 110. In thiscase, the participant clients 121 communicate directly with the API 137 of the central server130, for instance providing video and / or audio stream data to the central server 130 and / orreceiving video and / or audio stream data from the central server 130. Then, the producedshared stream may be provided to the external consumer 150 and / or to one or several ofthe client participants 121.Figure 4 illustrates the central server 130 in closer detail. As illustrated, said collecting function131 may comprise one or, preferably, several, format-specific collecting functions 131a.Each one of said format-specific collecting functions 131a is arranged to receive a videoand / or audio stream having a predetermined format, such as a predetermined binary encodingformat and / or a predetermined stream data container, and is specifically arrangedto parse binary video and / or audio data of said format into individual video frames, sequencesof video frames and / or time slots.The central server 130 further comprises an event detection function 132, arranged to receivevideo and / or audio stream data, such as binary stream data, from the collecting function131 and to perform a respective event detection on each individual one of the receiveddata streams. The event detection function 132 may comprise an AI (Artificial Intelligence)component 132a for performing said event detection. The event detection may take placewithout first time-synchronising the individual collected streams.The central server 130 further comprises a synchronising function 133, arranged to timesynchronisethe data streams provided by the collecting function 131 and processed by theevent detection function 132. The synchronising function 133 may comprise an AI compo30nent 133a for performing said time-synchronisation.The central server 130 further comprises a pattern detection function 134, arranged to performa pattern detection based on the combination of at least one, but in many cases atleast two, such as at least three, such as all, of the received data streams. The pattern detectionmay be further based on one, or in some cases at least two or more, events detectedfor each individual one of said data streams by the event detection function 132. Such detectedevents taking into consideration by said pattern detection function 134 may be distributedacross time with respect to each individual collected stream. The pattern detectionfunction 134 may comprise an AI component 134a for performing said pattern detection.The central server 130 further comprises a production function 135, arranged to produce ashared digital video stream based on the data streams provided from the collecting function131, and further based on any detected events and / or patterns. The shared video streamat least comprises a video stream produced to comprise one or several of video streamsprovided by the collecting function 131, raw, reformatted or transformed, and may alsocomprise corresponding audio stream data.The central server 130 further comprises a publishing function 136, arranged to publish theproduced shared digital video stream, such as via API 137 as described above.It is noted that Figures 1, 2 and 3 illustrate three different examples of how the centralserver 130 can be used to implement the principles described herein, and in particular toprovide a method according to the present invention, but that other configurations, with orwithout using one or several video communication services 110, are also possible.Hence, Figure 5 illustrates a method according to the present invention, for providing saidshared digital video stream. Figures 6a-6f illustrates different digital video / audio datastream states resulting from the method steps illustrated in Figure 5.In a first step, the method starts.In a subsequent collecting step, respective primary digital video streams 210, 301 are collected,such as by said collecting function 131, from at least two of said digital video sources120, 300. Each such primary data stream 210, 301 may comprise an audio part 214 and / ora video part 215. It is understood that "video", in this context, refers to moving and / or stillimage contents of such a data stream. Each primary data stream 210, 301 may be encodedaccording to any video / audio encoding specification (using a respective codec used by theentity providing the primary stream 210, 301 in question), and the encoding formats maybe different across different ones of said primary streams 210, 301 concurrently used in oneand the same video communication. It is preferred that at least one, such as all, of the pri10mary data streams 210, 301 is provided as a stream of binary data, possibly provided in aper se conventional data container data structure. It is preferred that at least one, such asat least two, or even all of the primary data streams 210, 301 are provided as respective livevideo recordings.It is noted that the primary streams 210, 301 may be unsynchronised in terms of time whenthey are received by the collecting function 131. This may mean that they are associatedwith different latencies or delays in relation to each other. For instance, in case two primaryvideo streams 210, 301 are live recordings, this may imply that they are associated, whenreceived by the collecting function 131, with different latencies with respect to the time ofrecording.It is also noted that the primary streams 210, 301 may themselves be a respective live camerafeed from a web camera; a currently shared screen or presentation; a viewed film clipor similar; or any combination of these arranged in various ways in one and the same screen.The collecting step is shown in Figures 6a and 6b. In Figure 6b, it is also illustrated how thecollecting function 131 can store each primary video stream 210, 301 as bundled audio / video information or as audio stream data separated from associated video streamdata. Figure 6b illustrates how the primary video stream 210, 301 data is stored as individualframes 213 or collections / clusters of frames, "frames" here referring to time-limited partsof image data and / or any associated audio data, such as each frame being an individual stillimage or a consecutive series of images (such as such a series constituting at the most 1second of moving images) together forming moving-image video content.In a subsequent event detection step, performed by the event detection function 132, saidprimary digital video streams 210, 301 are analysed, such as by said event detection function132 and in particular said AI component 132a, to detect at least one event 211 selectedfrom a first set of events. This is illustrated in Figure 6c.It is preferred that this event detections step is performed for at least one, such as at leasttwo, such as all, primary video streams 210, 301, and that it is performed individually foreach such primary video stream 210, 301. In other words, the event detection step preferablytakes place for said individual primary video stream 210, 301 only taking into considerationinformation contained as a part of that particular primary video stream 210, 301 inquestion, and particularly without taking into consideration information contained as a partof other primary video streams. Furthermore, the event detection preferably takes placewithout taking into consideration any common time reference 260 associated with the severalprimary video streams 210, 301.On the other hand, the event detection preferably takes into consideration informationcontained as a part of the individually analysed primary video stream in question across acertain time interval, such as a historic time interval of the primary video stream that islonger than 0 seconds, such as at least 0.1 seconds, such as at least 1 second.The event detection may take into consideration information contained in audio and / orvideo data contained as a part of said primary video stream 210, 301.Said first set of events may contain any number of types of events, such as a change of slidesin a slide presentation constituting or being a part of the primary video stream 210, 301 inquestion; a change in connectivity quality of the source 120, 300 providing the primaryvideo stream 210, 301 in question, resulting in an image quality change, a loss of image dataor a regain of image data; and a detected movement physical event in the primary videostream 210, 301 in question, such as the movement of a person or object in the video, achange of lighting in the video, a sudden sharp noise in the audio or a change of audio quality.It is realised that this is not intended to be an exhaustive list, but that these examplesare provided in order to understand the applicability of the presently described principles.In a subsequent synchronising step, performed by the synchronisation function 133, theprimary digital video streams 210 are time-synchronised with respect to a common timereference 260. As illustrated in Figure 6d, this time-synchronisation involves aligning theprimary video streams 210, 301 in relation to each other, using said common time reference260, so that they can be combined to form a time-synchronised context. The common timereference 260 may be a stream of data, a heartbeat signal or other pulsed data, or a timeanchor applicable to each of the individual primary video streams 210, 301. What is importantis that the common time reference can be applied to each of the individual primaryvideo streams 210, 301 in a way so that the informational contents of the primary videostream 210, 301 in question can be unambiguously related to the common time referencewith respect to a common time axis. In other words, the common time reference allows theprimary video streams 210, 301 to be aligned, via time shifting, so as to be time-synchronisedin the present sense.As illustrated in Figure 6d, the time-synchronisation may comprise determining, for eachprimary video streams 210, 301, one or several timestamps 261, in relation to the commontime reference 260.in a subsequent pattern detection step, performed by the pattern detection function 134,the hence time-synchronised primary digital video streams 210, 301 are analysed to detectat least one pattern 212 selected from a first set of patterns. This is illustrated in Figure 6e.In contrast to the event detection step, the pattern detection step may preferably be performedbased on video and / or audio information contained as a part of at least two of thetime-synchronised primary video streams 210, 301 considered jointly.Said first set of patterns may contain any number of types of patterns, such as several participantstalking interchangeably or concurrently; or a presentation slide change occurringconcurrently as a different event, such as a different participant talking. This list is not exhaustive,but illustrative.In alternative embodiments, detected patterns 212 may relate not to information containedin several of said primary video streams 210, 301 but only in one of said primary videostreams 210, 301. In such cases, it is preferred that such pattern 212 is detected based onvideo and / or audio information contained in that single primary video stream 210, 301spanning across at least two detected events 211, for instance two or more consecutivedetected presentation slide changes or connection quality changes. As an example, severalconsecutive slide changes that follow on each other rapidly over time may be detected asone single slide change pattern, as opposed to one individual slide change pattern for eachdetected slide change event.It is realised that the first set of events and said first set of patterns may compriseevents / patterns being of predetermined types, defined using respective sets of parametersand parameter intervals. As will be explained below, the events / patterns in said sets mayalso, or additionally, be defined and detected using various AI tools.In a subsequent production step, performed by the production function 135, the shareddigital video stream is produced as an output digital video stream 230 based on consecutivelyconsidered frames 213 of the time-synchronised primary digital video streams 210,301 and said detected patterns 212.As will be explained and detailed in the following, the present invention allows for the completelyautomatic production of the output digital video stream 230.For instance, such production may involve the selection of what video and / or audio infor30mation from what primary video stream 210, 301 to use to what extent in said output videostream 230; a video screen layout of the output video stream 230; a switching pattern betweendifferent such uses or layouts across time; and so forth.This is illustrated in Figure 6f, that also shows one or several additional pieces of time-related(related to the common time reference 260) digital video information 220, such as anadditional digital video information stream, that can be time-synchronised with said commontime reference 260 and used in concert with the time-synchronised primary videostreams 210, 301 in the production of the output video stream 230. For instance, the additionalstream 220 may comprise information with respect to any video and / or audio specialeffects to use, such as dynamically based on detected patterns; a planned time schedule forthe video communication; and so forth.In a subsequent publishing step, performed by the publishing function 136, the producedoutput digital video stream 230 is continuously provided to a consumer 110, 150 of theshared digital video stream as described above.In a subsequent step, the method ends. However, first the method may iterate any numberof times, as illustrated in Figure 5, to produce the output video stream 230 as a continuouslyprovided stream. Preferably, the output video stream 230 is produced to be consumed inreal-time or near real-time (taking into consideration a total latency added by all steps alongthe way), and continuously (publishing taking place immediately when more information isavailable, however not counting the below-described deliberately added latency). This way,the output video stream 230 may be consumed in an interactive manner, so that the outputvideo stream 230 may be fed back into the video communication service 110 or into anyother context forming a basis for the production of a primary video stream 210 again beingfed to the collection function 131 so as to form a closed feedback loop; or so that the outputvideo stream 230 may be consumed into a different (external to system 100 or at least externalto the central server 130) context but there forming the basis of a real-time, interactivevideo communication.As mentioned above, in some embodiments at least two of said primary digital videostreams 210, 301 are provided as a part of a shared digital video communication, such asprovided by said video communication service 110, the video communication involving arespective remotely connected participant client 121 providing the primary digital videostream 210 in question. In such cases, the collecting step may comprise collecting at leastone of said primary digital video streams 210 from the shared digital video communicationservice 110 itself, such as via an automatic participant client 140 in turn being granted accessto video and / or audio stream data from within the video communication service 110in question; and / or via an API 112 of the video communication service 110.Moreover, in this and in other cases the collecting step may comprise collecting at least oneof said primary digital video streams 210, 301 as a respective external digital video stream301, collected from an information source 300 being external to the shared digital videocommunication service 110. It is noted that one or several used such external video sources300 may also be external to the central server 130.In some embodiments, the primary video streams 210, 301 are not formatted in the samemanner. Such different formatting can be in the form of them being delivered to the collectingfunction 131 in different types of data containers (such as AVI or MPEG), but in pre20ferred embodiments at least one of the primary video streams 210, 301 is formatted accordingto a deviating format (as compared to at least one other of said primary videostreams 210, 301) in terms of said deviating primary digital video stream 210, 301 having adeviating video encoding; a deviating fixed or variable frame rate; a deviating aspect ratio;a deviating video resolution; and / or a deviating audio sample rate.It is preferred that the collecting function 131 is preconfigured to read and interpret all encodingformats, container standards, etc. that occur in all collected primary video streams210, 301. This makes it possible to perform the processing as described herein, not requiringany decoding until relatively late in the process (such as not until after the primary streamin question is put in a respective buffer; not until after the event detection step; or even notuntil after the event detection step). However, in the rare case in which one or several ofthe primary video feeds 210, 301 are encoded using a codec that the collecting function 131cannot interpret without decoding, the collecting function 131 may be arranged to performa decoding and analysis of such primary video stream 210, 301, followed by a conversioninto a format that can be handled by, for instance, the event detection function. It is notedthat, even in this case, it is preferred not to perform any r eencoding at this stage.For instance, primary video streams 220 being fetched from multi-party video events, suchas one provided by the video communication service 110, typically have requirements onlow latency and are therefore typically associated with variable framerate and variable pixelresolution to enable participants 122 to have an effective communication. In other words,overall video and audio quality will be decreased as necessary for the sake of low latency.External video feeds 301, on the other hand, will typically have a more stable framerate,higher quality but therefore possibly higher latency.Hence, the video communication service 110 may, at each moment in time, use a differentencoding and / or container than the external video source 300. The analysis and video productionprocess described herein therefore needs to combine these streams 210, 301 ofdifferent formats into a new one for the combined experience.As mentioned above, the collecting function 131 may comprise a set of format-specific collectingfunctions 131a, each one arranged to process a primary video stream 210, 301 of aparticular type of format. For instance, each one of these format-specific collecting functions131a may be arranged to process primary video streams 210, 301 having been en25coded using a different video respective encoding method / codec, such as Windows Media or DivX.However, in preferred embodiments the collecting step comprises converting at least two,such as all, of the primary digital video streams 210, 301 into a common protocol 240.As used in this context, the term "protocol" refers to an information-structuring standardor data structure specifying how to store information contained in a digital video / audiostream. The common protocol preferably does not, however, specify how to store the digitalvideo and / or audio information as such on a binary level (i.e. the encoded / compresseddata instructive of the sounds and images themselves), but instead forms a structure ofpredetermined format for storing such data. In other words, the common protocol prescribesstoring digital video data in raw, binary form without performing any digital videodecoding or digital video encoding in connection to such storing, possibly by not at allamending the existing binary form apart from possibly concatenating and / or splitting apartthe binary form byte sequence. Instead, the raw (encoded / compressed) binary data contentsof the primary video stream 210, 301 in question is kept, while repacking this rawbinary data in the data structure defined by the protocol. In some embodiments, the commonprotocol defines a video file container format.Figure 7 illustrates, as an example, the primary video streams 210, 301 shown in Figure 6a,restructured by the respective format-specific collecting function 131a and using said commonprotocol 240.Hence, the common protocol 240 prescribes storing digital video and / or audio data in datasets 241, preferably divided into discreet, consecutive sets of data along a time line pertainingto the primary video stream 210, 301 in question. Each such data set may include oneor several video frames, and also associated audio data.The common protocol 240 may also prescribe storing metadata 242 associated with speci25fied time points in relation to the stored digital video and / or audio data sets 241.The metadata 242 may comprise information about the raw binary format of the primarydigital video stream 210 in question, such as regarding a digital video encoding method orcodec used to produce said raw binary data; a resolution of the video data; a video framerate; a frame rate variability flag; a video resolution; a video aspect ratio; an audio compressionalgorithm; or an audio sampling rate. The metadata 242 may also comprise informationon a timestamp of the stored data, in relation to a time reference of the primary videostream 210, 301 in question as such.Using said format-specific collecting functions 131a in combination with said common protocol240 makes it possible to quickly collect the informational contents of the primaryvideo streams 210, 301 without adding latency by decoding / reencoding the receivedvideo / audio data.Hence, the collecting step may comprise using different ones of said format-specific collect10ing functions 131a for collecting primary digital video streams 210, 301 being encoded usingdifferent binary video and / or audio encoding formats, in order to parse the primary videostream 210, 301 in question and store the parsed, raw and binary data in a data structureusing the common protocol, together with any relevant metadata. Self-evidently, the determinationas to what format-specific collecting function 131a to use for what primaryvideo stream 210, 301 may be performed by the collecting function 131 based on predeterminedand / or dynamically detected properties of each primary video stream 210, 301 inquestion.Each hence collected primary video stream 210, 301 may be stored in its own separatememory buffer, such as a RAM memory buffer, in the central server 130.The converting of the primary video streams 210, 301 performed by each format-specificcollecting function 131a may hence comprise splitting raw, binary data of each thus convertedprimary digital video stream 210, 301 into an ordered set of said smaller sets of data241.Moreover, the converting may also comprise associating each (or a subset, such as a regularlydistributed subset along a respective time line of the primary stream 210, 301 in question)of said smaller sets 241 with a respective time of said common time reference 260.This associating may be performed by analysis of the raw binary video and / or audio data inany of the principle ways described below, or in other ways, and may be performed in orderto be able to perform the subsequent time-synchronising of the primary video streams 210,301. Depending on the type of common time reference 260 used, at least part of this associationof each of the data sets 241 may also or instead be performed by the synchronisationfunction 133. In the latter case, the collecting step may instead comprise associating each,or a subset, of the smaller sets 241 with a respective time of a time line specific for theprimary stream 210, 301 in question.In some embodiments, the collecting step also comprises converting the raw binary videoand / or audio data collected from the primary video streams 210, 301 into a uniform qualityand / or updating frequency. This may involve down-sampling or up-sampling of said raw,binary digital video and / or audio data of the primary digital video streams 210, 301, as necessary,to a common video frame rate; a common video resolution; or a common audiosampling rate. It s noted that such re-sampling can be performed without performing a fulldecoding / reencoding, or even without performing any decoding at all, since the format15specific collecting function 131a in question can process the raw binary data directly accordingto the correct binary encoding target format.Preferably, each of said primary digital video streams 210, 301 is stored in an individual datastorage buffer 250, as individual frames 213 or sequences of frames 213 as described above,and also each associated with a corresponding time stamp in turn associated with said commontime reference 260.In a concrete example, provided for illustrative purposes, the video communication service110 is Microsoft Teams, running a video conference involving concurrent participants122. The automatic participant client 140 is registered as a meeting participant in theTeems meeting.Then, the primary video input signals 210 are available to and obtained by the collectingfunction 130 via the automatic participant client 140. These are raw signals in H264 formatand contain timestamp information for every video frame.The relevant format-specific collecting function 131a picks up the raw data over IP (LANnetwork in the cloud) on a configurable predefined TCP port. Every Teems meeting participant,as well as associated audio data, are associated with a separate port. The collectingfunction 131 then uses the timestamps from the audio signal (which is in 50 Hz) and downsamplesthe video data to a fixed output signal of 25 Hz before storing the video stream 220in its respective individual buffer 250.As mentioned, the common protocol 240 stores the data in raw binary form. It can be designedto be very low-level, and to handle the raw bits and bytes of the video / audio data.In preferred embodiments, the data is stored in the common protocol 240 as a simple bytearray or corresponding data structure (such as a slice). This means that the data does notneed to be put in a conventional video container at all (said common protocol 240 not constitutingsuch conventional container in this context). Also, encoding and decoding video iscomputationally heavy, which means it causes delays and requires expensive hardware.Moreover, this problem scales with the number of participants.Using the common protocol 240, it becomes possible to reserve memory in the collectingfunction 131 for the primary video stream 210 associated with each Teams meeting participant122, and also for any external video sources 300, and then to change the amountof memory allocated on the fly during the process. This way, it becomes possible to changethe number of input streams and as a result keep each buffer effective. For instance, sinceinformation like resolution, framerate and so forth may be variable but stored as metadatain the common protocol 240, this information can be used to quickly resize each buffer asneed may be.The following is an example of a specification of a common protocol 240 of the presenttype:Bytes Example Description1 byte 1 0=video; 1=audio4 bytes 1234567 Buffer Length (int)8 bytes 424234234 Timestamp from the incoming audio / video bufferMeasured in ticks, 1tick = 100ns. (long int)1 byte 0 VideoColorFormat{NV12 = 0,Rgb24 = 1,Yuy2 = 2,H264 = 3}4 bytes 720 Video frame pixel height (int)4 bytes 640 Video frame pixel width (int)4 bytes 25.0 Video frame rateNumber of frames per second (float)1 byte 0 Is audio silence? 1 = true; 0 = false1 byte 0 AudioFormat{0 = Pcm16K1 = Pcm44KStereo}1 byte 0 Detected event in, if any0 = no event1, 2, 3, etc. = event of specified type detected30 bytes Reserved for future use8 bytes 1000000 Length of binary data in bytes (long int)Variable 0x87A879... Raw binary video / audio data of this frame(s)4 bytes 1234567 Dominant speaker Port4 bytes 1234567 Active speakerAbove, the "Detected event in, if any" data is included as a part of the common protocol260 specification. However, in some embodiments, this information (regarding detectedevents) may instead be put in a separate memory buffer.In some embodiments, said at least one additional piece of digital video information 220,that may be an overlay or an effect, is also stored in a respective individual buffer 250, asindividual frames or sequences of frames each associated with a corresponding time stampin turn associated with said common time reference 260.As exemplified above, the event detection step may comprise storing, using said commonprotocol 240, metadata 242 descriptive of a detected event 211, associated with the primarydigital video stream 210, 301 in which the event 211 in question was detected.The event detection can be performed in different ways. In some embodiments, performedby the AI component 132a, the event detection step comprises a first trained neural networkor other machine learning component analysing at least one, such as several or evenall, of said primary digital video streams 210, 301 individually in order to automatically detectany of said events 211. This may involve the AI component 132a classifying, in a man15aged classification, the primary video stream 210, 301 data into a set of predefined eventsand / or, in an unmanaged classification, into a dynamically determined set of events.In some embodiments, the detected event 211 is a change of presentation slides in apresentation being or being comprised in the primary video stream 210, 301 in question.For instance, if the presenter of the presentation decides to change the slide in the presentationhe / she is giving at that time to an audience, this means that what is interesting for agiven viewer can change. It may be that the newly shown slide is only a high level picturethat can best be seen briefly in a so-called "butterfly" mode (for instance, displaying in theoutput video stream 230 the slide side-by-side with a video of the presenter). Alternatively,the slide may contain much detail, text with small font sizes, and so forth. In this latter case,the slide should instead be presented in full-screen and perhaps during a somewhat longertime period than what is usually the case. A butterfly mode may not be as appropriate, sincethe slide in this case may be more interesting to a viewer of the presentation than the faceof the presenter.In practice, the event detection step may comprise at least one of the following:Firstly, the event 211 can be detected based on an image analysis of a difference betweena first image of a detected slide and a subsequent second image of a detected slide. Thenature of the primary video stream 220, 301 being that of showing a slide can be automaticallydetermined using per se conventional digital image processing, such as using motiondetection in combination with OCR (Optical Character Recognition).This may involve checking, using automatic computer image processing techniques,whether the detected slide has changed significantly enough to actually categorise it as aslide change. This may be done by checking the delta between current slide and previousslide with respect to RGB colour values. For instance, one may assess how much the RGBvalues have changed globally in the screen area covered by the slide in question, andwhether it is possible to find groups of pixels that belong together and that change in con15cert. This way, relevant slide changes can be detected while, for instance, filtering out irrelevantchanges such as shown computer mouse movements across the screen. This approachalso allows full configurability - for instance, sometimes it is desired to be able tocapture computer mouse movement, for instance when the presenter wishes to presentsomething in detail using the computer mouse to point to different things.Secondly, the event 211 may be detected based on an image analysis of an informationalcomplexity of said second image itself, to determine the type of event with greater specificity.This may, for instance, involve assessing a total amount of textual information on the slidein question, as well as associated font sizes. This may be done by using conventional OCRmethods, such as deep learning-based character recognition techniques.It is noted that, since the raw binary format of the assessed video stream 210, 301 is known,this may be performed directly in the binary domain, without first decoding or reencodingthe video data. For instance, the event detection function 132 may call the relevant formatspecificcollecting function for image interpreting services, or the event detection function132 may itself include functionality for assessing image information, such as on individualpixel level, for a number of different supported raw binary video data formats.In another example, the detected event 211 is a loss of communication connection of aparticipant client 121 to a digital video communication service 110. Then, the detection stepmay comprise detecting that said participant client 121 has lost communication connectionbased on an image analysis of a series of subsequent video frames 213 of a primary digitalvideo stream 210 corresponding to the participant client 121 in question.Because participant clients 121 are associated with different physical locations and differentinternet connections, it can happen that someone will lose connection to the video communicationservice 110 or the central server 130. In that situation, it is desirable to avoidshowing a black or empty screen in the produced output video stream 230.Instead, such connection loss can be detected as an event by the event detection function132, such as by applying a 2-class classification algorithm where the 2 classes used are connected / not connected (no data). In this case, it is understood that "no data" differs fromthe presenter sending out a black screen intentionally. Because a brief black screen, such asof only 1 or 2 frames, may not be noticeable in the end production stream 230, one mayapply said 2-class classification algorithm over time to create a time series. Then, a thresholdvalue, specifying a minimum length for a connection interruption, can be used to decidewhether there is a lost connection or not.As will be explained in the following, detected events of these exemplified types may beused by the pattern detection function 134 to take various actions, as suitable and desired.As mentioned, the individual primary video streams 210, 301 are each related to the commontime reference 260, making it possible for the synchronisation function 133 to time30synchronise them in relation to each other.In some embodiments, the common time reference 260 is based on or comprises a commonaudio signal 111 (see Figures 1-3), the common audio signal 111 being common for theshared digital video communication service 110 involving at least two remotely connectedparticipant clients 121 as described above, each providing a respective one of said primarydigital video streams 210.In the example of Microsoft Teams discussed above, a common audio signal is producedand can be captured by the central server 130 via the automatic participant client 140and / or via the API 112. In this and in other examples, such a common audio signal may beused as a heartbeat signal to time-synchronise the individual primary video streams 220 bybinding each of these to specific time points based on this heartbeat signal. Such a commonaudio signal may be provided as a separate (in relation to each of the other primary videostreams 210) signal, whereby the other primary video streams 210 may each be individuallytime-correlated to the common audio signal, based on audio contained in the other primaryvideo stream 210 in question or even based on image information contained therein (suchas using automatic image processing-based lip syncing techniques).In other words, to handle any variable and / or differing latency associated with individualprimary video streams 210, and to achieve time-synchronisation for the combined videooutput stream 230, such a common audio signal is used as a heartbeat for all primary videostreams 210in the central server 130 (but perhaps not external primary video streams 301).In other words, all other signals are mapped to this common audio time heartbeat to makesure that everything is in time sync.In a different example, the time-synchronisation is achieved using a time synchronisationelement 231 introduced into the output digital video stream 230 and detected by a respectivelocal time-synchronising software function 125 provided as a part of one or several individualones of the participant clients 121, the local software function 125 being arrangedto detect a time of arrival of the time synchronisation element 231 in the output videostream 230. As is understood, in such embodiments the output video stream 230 is fed backinto the video communication service 110 or otherwise made available to each participantclient 121 and the local software function 125 in question.For instance, the time synchronisation element 231 may be a visual marker, such as a pixelchanging colours in a predetermined sequence or manner, placed or updated in the outputvideo 230 at regular time intervals; a visual clock updated and displayed in the output video230; a sound signal (that may be designed to be non-audible to participants 122 by, forinstance, having low enough amplitude and / or high enough frequency) and added to anaudio forming part of the output video stream 230. The local software function 125 is ar10ranged to, using suitable image and / or audio processing, automatically detect respectivetimes of arrival of each of the (or each of the) time synchronisation element(s) 231.Then, the common time reference 260 may be determined at least party based on said detectedtimes of arrival. For instance, each of the local software functions 125 may communi15cate to the central server 130 respective information signifying said detected time of arrival.Such communication may take place via a direct communication link between the participantclient 121 in question and the central server 130. However, the communication mayalso take place via the primary video stream 210 associated with the participant client 121in question. For instance, the participating client 121 may introduce a visual or audible code,such as of the above discussed type, in the primary video stream 210 produced by thatparticipant client 121 in question, for automatic detection by the central server 130 andused to determine the common time reference 260.In yet additional examples, each participant client 121 may perform an image detection ina common video stream available for viewing by all participant clients 121 to the video communicationservice 110 and relay the results of such image detection to the central server130, in a way corresponding to the ones discussed above, to there be used to over timedetermine respective offsets of each participant client 121 in relation to each other. Thisway, the common time reference 260 may be determined as a set of individual relative offsets.For instance, a selected reference pixel of a commonly available video stream may bemonitored by several, or all, participating clients 121, such as by said local software function125, and a current colour of that pixel may be communicated to the central server 130. Thecentral server 130 may calculate a respective time series based on consecutively receivedsuch colour values from each of a number of (or all) the participant clients 121, and performa cross-correlation resulting in an estimated set of relative time offsets across the differentparticipant clients 121.In practice, the output video stream 230 fed into the video communication service 110 maybe included as a part of a shared screen every participant client of the video communicationin question, and may therefore be used to assess such time offset associated with the participantclients 121. In particular, the output video stream 230 fed to the video communicationservice 110 may be available again to the central server via the automatic participantclient 140 and / or the API 112.In some embodiments, the common time reference 260 may be determined at least partlybased on a detected discrepancy between an audio part 214 of a first one of said primarydigital video streams 210, 301 and an image part 215 of said first primary digital videostreams 210, 301. Such discrepancy may, for instance, be based on a digital lip sync videoimage analysis of a talking participant 122 viewed in said first primary digital video stream210, 301 in question. Such lip sync analysis is conventional as such, and may for instanceuse a trained neural network. The analysis may be performed by the synchronisation function133 for each primary video stream 210, 301 in relation to available common audio information,and relative offsets across the individual primary video streams 210, 301 may bedetermined based on this information.In some embodiments, the synchronisation step comprises deliberately introducing a latencyof at the most 30 seconds, such as at the most 5 seconds, such as at the most 1 seconds,such as at the most 0.5 seconds, but longer than 0 s, so that the output digital videostream 230 is provided at least with said latency. At any rate, the deliberately introducedlatency is at least several video frames, such as at least three, or even at least five or even10, video frames, such as this number of frames (or individual images) stored after anyresampling in the collecting step. As used herein, the term "deliberately" means that thelatency is introduced irrespective of any need for introducing such a latency based on synchronisationissues or similar. In other words, the deliberately introduced latency is introducedin addition to any latency introduced as a part of the synchronisation of the primaryvideo streams 210, 301 in order to time-synchronise them one in relation to the other. Thedeliberately introduced latency may be predetermined, fixed or variable in relation to thecommon time reference 260. The latency time may be measured in relation to a least latentone of the primary video streams 210, 301, so that more latent ones of these streams 210,301 as a result of said time-synchronisation are associated with a relatively smaller deliber10ately added latency.In some embodiments, a relatively small latency is introduced, such as of 0.5 seconds orless. This latency will barely be noticeable by participants to a video communication service110 using the output video stream 230. In other embodiments, such as when the outputvideo stream 230 will not be used in an interactive context but is instead published in a onewaycommunication to an external consumer 150, a larger latency may be introduced.This deliberately introduced latency achieves sufficient time for the synchronisation function133 to map the collected individual primary stream 210, 301 video frames onto thecorrect common time reference 260 timestamp 261. It also allows sufficient time to performthe event detection described above, in order to detect lost primary stream 210, 301 signals,slide changes, resolution changes, and so forth. Furthermore, deliberately introducingsaid latency allows for an improved pattern detection function 134, as will be described inthe following.It is realized that the introduction of said latency involves buffering 250 each of the collectedand time-synchronised primary video streams 210, 301 before publishing the output videostream 230 using the buffered frames 213 in question. In other words, video and / or audiodata of at least one, several or even all of the primary video streams 210, 301 will be presentin the central server 130 in a buffered manner, much like a cache but not (like a conventionalcache buffer) used with the intention of being able to handle varying bandwidth situationsbut for the above reasons, and in particular to be used by the pattern detection function134.Hence, in some embodiments said pattern detection step comprises taking into considerationcertain information of at least one, such as several or even all, of the primary digitalvideo streams 210, 301, the certain information being present in a later frame 213 than aframe of a time-synchronised primary digital video stream 210 yet to be used in the productionof the output digital video stream 230. Hence, a newly added frame 213 will existin the buffer 250 in question during a particular latency time before forming part of (or basisfor) the output video stream 230. During this time period, the information in the frame 213in question will constitute information in the "future" in relation to a currently used frameto produce a current frame of the output video stream 230. Once the output video stream230 timeline reaches the frame in question 213, it will be used for the production of thecorresponding frame of the output video stream 230, and may thereafter be discarded.In other words, the pattern detection function 134 has at its disposal a set of video / audioframes 213 that have still not been used to produce the output video stream 230, and usesthis data to detect said patterns.The pattern detection can be performed in different ways. In some embodiments, performedby the AI component 134a, the pattern detection step comprises a second trainedneural network or other machine learning component analysing at least two, such as at leastthree or even all, of said primary digital video streams 120, 301 in concert to automaticallydetect said pattern 212.In some embodiments, the detected pattern 212 comprises a speaking pattern involving atleast two different speaking participants 122, each associated with a respective participantclient 121, to the shared video communication service 110, each of said speaking participants122 being viewed visually in a respective one of said primary digital video streams210, 301.The production step preferably comprises determining, keeping track of and updating a currentproduction state of the output video stream 230. For instance, such a state can dictatewhat, if any, participants 122 are visible in the output video stream 230, and where on thescreen; if any external video stream 300 is visible in the output video stream 230, and whereon the screen; if any slides or shared screens are shown in full-screen mode or in combinationwith any live video streams; and so on. Hence, the production function 135 can beviewed as a state machine with respect to the produced output video stream 230.To generate the output video stream 230 as a combined video experience to be viewed by,for instance, an end consumer 150, it is advantageous for the central server 130 to be ableto understand what happens on a deeper level than merely detecting individual events associatedwith individual primary video streams 210, 301.In a first example, a presenting participant client 121 is changing a currently viewed slide.This slide change is detected by the event detection function 132 as described above, andmetadata 242 is added to the frame in question indicative of a slide change having happened.This happens a number of times, since the presenting participating client 121 turnsout to skip a number of slides forward in rapid succession, resulting in a series of "slidechange" events detected by the even detection function 132 and stored with correspondingmetadata 242 in the individual buffer 250 for the primary video stream 210 in question. Inpractice, each such rapidly forward skipped slide may be visible for only a fraction of a second.The pattern detection function 134, looking at the information in the buffer 250 in question,spanning across several of these detected slide changes, will detect a pattern correspondingto one single slide change (that is, to the last slide in the forward-skipping, the slide reminingvisible once the rapid skipping is finished), rather than a number or rapidly performed slidechanges. In other words, the pattern detection function 134 will note that there are, forinstance, ten slide changes in a very short period of time, why they will be handled as adetected pattern signifying one single slide change. As a result, the production function 135,having access to the patterns detected by the pattern detection function 134, may chooseto show the final slide in full-screen mode in the output video stream 230 for a couple ofseconds, since it determines this slide to be potentially important in said state machine. Itmay also choose not to show the intermediately viewed slides at all in the output stream230.The detection of the pattern with several rapidly changing slides may be detected by a simplerule-based algorithm, but may alternatively be detected using a trained neural networkdesigned and trained to detect such patterns in moving images by classification.In a different example, that may for instance be useful in case the video communication isa talk show, panel debate or similar, it may be desirable to quickly switch visual attentionbetween, one the one hand, a current speaker, while, on the other hand, still giving theconsumer 150 a relevant viewing experience by producing and publishing a calm andsmooth output video stream 230. In this case, the event detection function 132 can contin15uously analyse each primary video stream 210, 301 to at all times determine whether or nota person being viewed in that particular primary video stream 210, 301 is currently speakingor not. This may, for instance, be performed as described above, using per se conventionalimage processing tools. Then, the pattern detection function 134 may be operable to detectparticular overall patterns, involving several of said primary video streams 210, 301, saidpatterns being useful for producing a smooth output video stream 230. For instance, thepattern detection function 134 may detect a pattern of very frequent switches between acurrent speaker and / or patterns involving several concurrent speakers.Then, the production function 135 can take such detected patterns into consideration whentaking automated decisions in relation to said production state, for instance by not automaticallyswitching visual focus to a speaker who only speaks for half a second before againgoing silent, or to switch to a state where several speakers are displayed side by side duringa certain time period when both are speaking interchangeably or concurrently. This statedecision process may in itself be performed using time series pattern recognition tech30niques, or using a trained neural network, but can also be based at least partly on a predeterminedset of rules.In some embodiments, there may be multiple patterns detected in parallel and forming inputto the production function 135 state machine. Such multiple patterns may be used bydifferent AI components, computer vision detecting algorithms, and so forth, by the productionfunction 135. As an example, permanent slide changes can be detected while concurrentlydetecting unstable connections of some participant clients 121, while other patternsdetect a current main speaking participant 122. Using all such available pattern data,a classifier neural network can be trained, and / or a set of rules can be developed, for analysisof a time series of such pattern data. Such a classification may be at least partly, suchas completely, supervised to result in determined desired state changes to be used in saidproduction. For instance, different such predetermined classifiers can be produced, specificallyarranged to automatically produce the output video stream 230 according to variousand different production styles and desires. Training may be based on known productionstate change sequences as desired outputs and known pattern time series data as trainingdata. In some embodiments, a Bayesian model can be used to produce such classifiers. In aconcrete example, information can be a priori gleaned from an experienced producer,providing input such as "in a talkshow I never switch from speaker A to Speaker B directlybut always first show an overview before I focus on the other speaker, unless that the otherspeaker is very dominant and speaking loud." This production logic then be represented asa Bayesian model on the general form "if X is true | given the fact that Y is true | performZ". The actual detection (of whether someone is speaking loudly, etc.) could be performedusing a classifier or threshold-based rules.With large data sets (of pattern time series data), one can use deep learning methods todevelop correct and appealing production formats for use in automated productions ofvideo streams.In summary, using a combination of the event detection based on individual primary videostreams 210, 301; the deliberately introduced latency; the pattern detection based on severaltime-synchronised primary video streams 210, 301 and the detected events; and theproduction process based on the detected patterns, makes it possible to achieve automatedproduction of the output digital video stream 230 according to a wide possible selection oftastes and styles. This result is valid across a wide range of possible neural network and / orrule-based analysis techniques used by the event detection function 132, pattern detectionfunction 134 and production function 135.As exemplified above, the production step may comprise producing the output digital videostream 230 based on a set of predetermined and / or dynamically variable parameters regardingvisibility of individual ones of said primary digital video streams 210, 301 in saidoutput digital video stream 230; visual and / or audial video content arrangement; used visualor audio effects; and / or modes of output of the output digital video stream 230. Suchparameters may be automatically determined by said production function 135 state machineand / or be set by an operator controlling the production (making it semi-automatic)and / or be predetermined based on certain a priori configuration desires (such as a shortesttime between output video stream 230 layout changes or state changes of the above-exemplifiedtypes).In practical examples, the state machine may support a set of predetermined standard layoutsthat may be applied to the output video stream 230, such as a full-screen presenterview (showing a current speaking participant 122 in full-screen); a slide view (showing acurrently shared presentation slide in full-screen); "butterfly view", showing both a cur20rently speaking participant 122 together with a currently shared presentation slide, in aside-by-side view; a multi-speaker view, showing all or a selected subset of participants 122side-by-side or in a matrix layout; and so forth. Various available production formats can bedefined by a set of state machine state changing rules (as exemplified above) together withan available set of states (such as said set of standard layouts). For instance, one such pro25duction format may be "panel discussion", another "presentation", and so forth. By selectinga particular production format via a GUI or other interface to the central server 130, anoperator of the system 100 may quickly select one of a set of predefined such productionformats, and then allow the central server 130 to, completely automatically, produce theoutput video stream 230 according to the production format in question, based on availableinformation as described above.Furthermore, during the production a respective in-memory buffer is created and maintained,as described above, for each meeting participant client 121 or external video source300. These buffers can easily be removed, added, and changed on the fly. The central server130 can then be arranged to receive information, during the production of the output videostream 230, regarding added / dropped-off participant clients 121 and participants 122scheduled for delivering speeches; planned or unexpected pauses / resumes of presentations;desired changes to the currently used production format, and so forth. Such informationmay, for instance, be fed to the central server 130 via an operator GUI or interface,as described above.As exemplified above, in some embodiments at least one of the primary digital videostreams 210, 301 is provided to the digital video communication service 110, and the publishingstep may then comprise providing said output digital video stream 230 to that samecommunication service 110. For instance, the output video stream 230 may be provided toa participant client 121 of the video communication service 110, or be provided, via API 112as an external video stream to the video communication service 110. This way, the outputvideo stream 230 may be made available to several or all of the participants to the videocommunication event currently being achieved by the video communication service 110.As also discussed above, in addition or alternatively the output video stream 230 may beprovided to one or several external consumers 150.In general, the production step may be performed by the central server 130, providing saidoutput digital video stream 230 to one or several concurrent consumers as a live videostream via the API 137.The present invention also relates to a computer software function for providing a shareddigital video stream according to what has been described above. Such a computer softwarefunction is then arranged to, when executing, perform the above-described collecting,event detection, synchronising, pattern detection, production, and publishing steps. Thecomputer software function is arranged to execute on physical or virtual hardware of thecentral server 130, as described above.The present invention also relates to the system 100 as such, being a system for providinga shared digital video stream, and in turn comprising the central server 130. The centralserver 103, in turn, is arranged to perform said collecting, event detection, synchronising,pattern detection, production, and publishing steps. For instance, these steps are performedby the central server 130 executing said computer software function to performsaid steps as described above.Above, preferred embodiments have been described. However, it is apparent to the skilledperson that many modifications can be made to the disclosed embodiments without departingfrom the basic idea of the invention.For instance, many additional functions can be provided as a part of the system 100 describedherein, and that are not described herein. In general, the presently described solutionsprovide a framework on top of which detailed functionality and features can be built,to cater for a wide variety of different concrete application wherein streams of video datais used for communication.One example is a demonstration situation, wherein primary video streams include a view ofthe presenter, a shared digital slide-based presentation, and a live video of a product beingdemonstrated.Another example is a teaching situation, wherein primary video streams include a view of ateacher, a live video of a physical entity being the topic of the teaching, and respective livevideo of several students that may pose questions and engage in dialogue with the teacher.In either of these two examples, a video communication service (that may be or may not bepart of the system) may provide one or several of the primary video streams, and / or severalof the primary video streams may be provided as external video sources of the types discussedherein.In general, all which has been said in relation to the present method is applicable to thepresent system and computer software product, and vice versa as applicable.Hence, the invention is not limited to the described embodiments, but can be varied withinthe scope of the enclosed claims.
Claims
1. Method for providing a shared digital video stream, the method comprises the steps in a collecting step, collecting from at least two digital video sources (120) respective primary digital video streams (210); in an event detection step, individually analysing said primary digital video streams (210) to detect at least one event (211) selected from a first set of events; in a synchronising step, time-synchronising said primary digital video streams (210) with respect to a common time reference (260); in a pattern detection step, analysing the time-synchronised primary digital video streams (210) to detect at least one pattern (212) selected from a first set of patterns, the pattern detection being based on said detected at least one event (211), the pattern detection being based on, firstly, video and / or audio information contained as a part of at least two time-synchronised of the primary video streams (210) considered jointly or, secondly, video and / or audio information contained in a single primary video stream (210) spanning across at least two detected events (211); in a production step, producing the shared digital video stream as an output digital video stream (230) based on consecutively considered frames (213) of said time-synchronised primary digital video streams (210) and said detected patterns (212); and in a publishing step, continuously providing said output digital video stream (230) to a consumer of the shared digital video stream.
2. Method according to claim 1, wherein at least two of said primary digital video streams (210) are provided as a part of a shared digital video communication service (110) involving a respective remotely connected participant client (121) providing the primary digital video stream (210) in question.
3. Method according to claim 2, wherein the collecting step comprises collecting at least one of said primary digital video streams (210) from the shared digital video communication service (110).
4. Method according to claim 2 or 3, wherein the collecting step comprises collecting at least one of said primary digital video streams (210) as an external digital video stream (301) collected from an information source (300) being external to said shared digital video communication service (110).
5. Method according to any one of the preceding claims, wherein at least one of said primary digital video streams (210) has a deviating video encoding, frame rate, aspect ratio and / or resolution.
6. Method according to any one of the preceding claims, wherein the collecting step comprises converting at least two of said primary digital video streams (210) into a common protocol (240), the common protocol (240) prescribing storing digital video data in raw, binary form without performing any digital video decoding or digital video encoding, the common protocol (240) also prescribing storing metadata (242) associated with specified time points in relation to the stored digital video data.
7. Method according to claim 6, wherein said metadata (242) comprises information about the format of the primary digital video stream (210) in question, such as regarding a used digital video encoding, resolution, frame rate or aspect ratio.
8. Method according to claim 6 or 7, wherein the collecting step comprises using different format-specific collecting functions for primary digital video streams (210) being encoded using different encoding formats.
9. Method according to any one of claims 6-8, wherein said converting comprises splitting raw, binary data of each of said primary dig25 ital video streams (210) into smaller sets of data (241), and associating each of said smaller sets (241) with a respective time of said common time reference (260).
10. Method according to any one of claims 6-9, wherein said converting comprises down-sampling or up-sampling said raw, binary digital video data of said primary digital video streams (210), as necessary, to a common frame rate and a common resolution.
11. Method according to claim 10, wherein each of said primary digital video streams (210) is stored in an individual buffer (250), as individual frames or sequences of frames (213) each associated with a corresponding time stamp in turn associated with said common time reference (260).
12. Method according to claim 10 or 11, wherein at least one additional piece of digital video information (220), such as an over10 lay or an effect, is also stored in a respective individual buffer (250), as individual frames or sequences of frames each associated with a corresponding time stamp in turn associated with said common time reference (260).
13. Method according to any one of claims 6-12, wherein the event detection step comprises storing, using said common protocol (240), metadata (242) descriptive of said detected event (211), associated with the primary digital video stream (210) in which the event (211) in question was detected.
14. Method according to any one of the preceding claims, wherein the event detection step comprises a first trained neural network or other machine learning component (132a) analysing each of said primary digital video streams (210) in order to automatically detect said event (211).
15. Method according to claim 14, wherein the event (211) is a change of presentation slides, and wherein the event detection step comprises at least one of firstly, detecting the event (211) based on an image analysis of a difference be25 tween a first image of a slide and a subsequent second image of a slide; and secondly, detecting the event (211) based on an image analysis of an informational complexity of said second image.
16. Method according to claim 14, wherein the event (211) is a loss of communication connection of a participant client (121) to a digital video communication service (110), and wherein the detection step comprises detecting that said participant client (121) has lost communication connection based on an image analysis of a series of subsequent frames (213) of a primary digital video stream (210) corresponding to the participant client (121) in question.
17. Method according to any one of the preceding claims, wherein said common time reference (260) comprises a common audio signal (111), the common audio signal (111) being common for a shared digital video communication service (110) involving at least two remotely connected participant clients (121) each providing a respective one of said primary digital video streams (210).
18. Method according to any one of claims 1-16, wherein at least two of said primary digital video streams (210) are provided by respective participant clients (121) to a shared digital video communication service (110), each such participant client (121) having a respective local synchronisation software (125) arranged to detect a time of arrival of a time synchronisation element (231) provided as a part of the output digital video stream (230) provided to the participant client (121) in question, and wherein said common time reference (260) is determined at least party based on said detected times of arrival.
19. Method according to any one of the preceding claims, wherein said common time reference (260) is determined at least partly based on a detected discrepancy between an audio part (214) of a first one of said primary digital video streams (210) and an image part (215) of said first primary digital video streams (210), the discrepancy being based on a digital lip sync video analysis of a talking participant (122) viewed in said first primary digital video stream (210).
20. Method according to any one of the preceding claims, wherein the synchronisation step comprises deliberately introducing a latency of at the most 30 seconds, such as at the most 5 seconds, such as at the most 1 seconds, such as at the most 0.5 seconds, so that the output digital video stream (230) is provided at least with said latency.
21. Method according to claim 20, wherein the pattern detection step comprises taking into consideration information of said primary digital video streams (210), the information being present in a later frame (213) than a frame of a time-synchronised primary digital video stream (210) yet to be used in the production of the output digital video stream (230).
22. Method according to any one of the preceding claims, wherein the pattern detection step comprises a second trained neural network or other machine learning component (134a) analysing said primary digital video streams (120) in concert to automatically detect said pattern (212).
23. Method according to any one of the preceding claims, wherein said detected pattern (212) comprises a speaking pattern involving at least two different speaking participants (122), each associated with a respective participant client (121), to a shared video communication service (110), each of said speaking participants (122) being viewed in a respective one of said primary digital video streams (210).
24. Method according to any one of the preceding claims, wherein the production step further comprises producing the output digital video stream (230) based on a set of predetermined and / or dynamically variable parameters regarding visibility of individual ones of said primary digital video streams (210) in said output digital video stream (230); visual and / or audial video content arrangement; used visual or audio effects; and / or modes of output of the output digital video stream (230).
25. Method according to any one of the preceding claims, wherein at least one of said primary digital video streams (210) is provided to a digital video communication service (110), and wherein the publishing step comprises providing said output digital video stream (230) to said communication service (110), such as to a participant client (121) of the communication service (110), or to an external consumer (150).
26. Method according to any one of the preceding claims, wherein the production step is performed by a central server (130), providing said output digital video stream (230) to one or several concurrent consumers as a live video stream via an API (137).
27. Computer software product for providing a shared digital video stream, the computer software function being arranged to, when executing, perform a collecting step, wherein respective primary digital video streams (210) are collected from at least two digital video sources (120); an event detection step, wherein said primary digital video streams (210) are individually analysed to detect at least one event (211) selected from a first set of events; a synchronising step, wherein said primary digital video streams (210) are timesynchronised with respect to a common time reference (260); a pattern detection step, wherein the time-synchronised primary digital video streams (210) are analysed to detect at least one pattern (212) selected from a first set of patterns, the pattern detection being based on said detected at least one event (211), the pattern detection being based on, firstly, video and / or audio information contained as a part of at least two time-synchronised of the primary video streams (210) considered jointly or, secondly, video and / or audio information contained in a single primary video stream (210) spanning across at least two detected events (211); a production step, wherein the shared digital video stream is produced as an output digital video stream (230) based on consecutively considered frames (213) of said time-synchronised primary digital video streams (210) and said detected patterns (212); and a publishing step, wherein said output digital video stream (230) is continuously provided to a consumer of the shared digital video stream.
28. System (100) for providing a shared digital video stream, the system (100) comprising a central server (130) in turn comprising a collecting function (131), arranged to collect from at least two digital video sources (120) respective primary digital video streams (210); an event detection function (132), arranged to individually analyse said primary digital video streams (210) to detect at least one event (211) selected from a first set of events; a synchronising function (133), arranged to time-synchronise said primary digital video streams (210) with respect to a common time reference (260); a pattern detection function (134), arranged to analyse the time-synchronised primary digital video streams (210) to detect at least one pattern (212) selected from a first set of patterns, the pattern detection being based on said detected at least one event (211), the pattern detection being based on, firstly, video and / or audio information contained as a part of at least two time-synchronised of the primary video streams (210) considered jointly or, secondly, video and / or audio information contained in a single primary video stream (210) spanning across at least two detected events (211); a production function (135), arranged to produce the shared digital video stream as an output digital video stream (230) based on consecutively considered frames (213) of said time-synchronised primary digital video streams (210) and said detected patterns (212); and a publishing function (136), arranged to continuously provide said output digital video stream (230) to a consumer of the shared digital video stream.
