Method and apparatus for generating a video stream consisting of a shadow video stream

Abstract

A method for generating a generated video stream (PR) comprising: a) capturing a first primary video stream (PV) and a second primary video stream (PV) visually showing a first user (122, 122''); b) automatically generating a generated video stream (PR) based on the first primary video stream (PV) and the second primary video stream (PV); c) downsampling the first primary video stream (PV) and the second video stream (PV), thereby obtaining a first shadow primary video stream and a second shadow primary video stream; d) automatically generating a generated shadow video stream based on the first shadow primary video stream and the second shadow primary video stream, automatically generating based on an automatic generation decision corresponding to a decision used to generate the generated video stream (PR); and e) storing the first shadow primary video stream, the second shadow primary video stream, the generated shadow video stream, and information identifying the automatic generation decision.

Description

Technical Field

[0001] This invention relates to a system, computer program product, and method for generating digital video streams, and more particularly to a system, computer program product, and method for generating digital video streams based on digital input video streams. In some embodiments, the digital video stream is generated in the context of a digital video conferencing or conferencing system, particularly involving multiple different concurrent users. The generated digital video stream can be published externally or within the digital video conferencing or conferencing system. The system can be interactive because it allows participating users to interact in real-time or near real-time, for example, by a second user watching the first user's video as part of a first generated video stream, and the first user simultaneously watching the second user's video as part of a second generated video stream, which may be the same as or different from the first generated video stream.

[0002] In other embodiments, the invention is applied to scenarios that are not digital video conferencing, but in which multiple digital video input streams are processed concurrently and otherwise combined into a generated digital video stream. For example, such scenarios could be educational or instructional. Background Technology

[0003] There are many known digital video conferencing systems, such as Microsoft ® Teams ® Zoom ® and Google ® Meet ® It provides the ability for two or more participants to have a virtual meeting using locally captured digital video and audio, and to broadcast it to all participants to simulate a physical meeting.

[0004] Various digital video conferencing solutions offer a range of automated and non-automated generation capabilities, such as determining when, via, and to which participants what video content to display. Generation decisions can be based on information such as detected currently speaking participants, screen sharing, and interpretations of the content displayed by various available video streams. This generation can become relatively complex as the number of participants increases and available information is used in various ways, especially since the various available information may be in different formats and therefore require conversion; different types of hardware and / or software are involved in the solution; external information is fed into the solution; and information can be processed by more than one host.

[0005] Overall, this solution is expected to offer low latency and high quality in terms of frame rate, pixel resolution, and pixel depth. Furthermore, interactive users require time synchronization of video information shared across these users.

[0006] These issues are particularly prevalent in scenarios involving multiple participants in video conferences or similar events, where either all participants are physically present in the same room or venue, or some participants are physically present while others participate remotely.

[0007] The corresponding problem arises in other scenarios where the output digital video stream is generated based on multiple input digital video streams, such as in digital video generation systems used for education and instruction.

[0008] Swedish patent application SE 2151267-8 discloses various solutions to the problems discussed above. Swedish patent application 2151461-7 discloses various solutions for latency processing in multi-participant digital video environments, such as when different groups of participants are associated with different general latency. Swedish patent application 2250113-4 discloses various solutions for tracking one or more people using one or more cameras. Swedish patent application SE 2250945-9 discloses various methods for load balancing production workloads between local and remote computers. Swedish patent application SE 2350439-2 discloses processing static and dynamic content in video-based systems.

[0009] In the various types of solutions described and mentioned above, there is often a problem with the system's various users and external parties: knowing who is participating in the meeting or interaction as a user.

[0010] Swedish patent application SE 13509476 discloses a solution for cryptographically binding information to a timeline using one-way functions and publicly available information. Summary of the Invention

[0011] The present invention solves one or more of the above-mentioned problems.

[0012] Therefore, the present invention relates to a method for generating a video stream, the method comprising the following steps: a) Authenticate the first user; b) Capture a first main video stream that visually represents the first user, wherein the first main video stream includes a first main authentication video portion that visually represents the authentication steps and / or includes information that visually and / or audibly represents the first authentication token generated by the authentication steps; c) Capturing a fourth main video stream visually representing the first user from a viewpoint different from the first main video stream, wherein the fourth main video stream includes a fourth main authentication video portion that visually illustrates the authentication steps and / or includes information visually and / or audibly representing the first authentication token; and a) The output of the first one-way function is calculated using the first main video stream as a direct or derived input, and the output of the fourth one-way function is calculated using the fourth main video stream as a direct or derived input. The output of the first one-way function is calculated using the first main certified video portion as a direct or derived input, and the output of the fourth one-way function is calculated using the fourth main certified video portion as a direct or derived input.

[0013] In some embodiments, the method includes the following additional steps: b) Automatically generate a new video stream based on the first main video stream.

[0014] In some embodiments, the generated video stream is provided to the receiving user.

[0015] The method may further include performing at least one of the following for each corresponding piece of information representing the direct output or derived output of the first one-way function and / or the fourth one-way function: The visual and / or audible representation of the information is embedded into at least one of a first main video stream, a second main video stream, and a generated video stream (PR), such that the embedding depends on the content in the video stream, which is different from the video stream into which the embedding is performed, and the generated video stream (PR) is generated based on the first main video stream (PV). Store this information; and This information will be released publicly.

[0016] In some examples, a message representing the direct or derived output of the first or fourth one-way function can be referred to as the "first message" or the "fourth message," respectively.

[0017] The present invention also relates to another method for generating a video stream, the method comprising the following steps: a) Capture a first main video stream and a second main video stream that visually represent a first user, wherein the second main video stream optionally visually represents a second user; b) Automatically generate a video stream based on the first main video stream and the second main video stream. The automatic generation is based on an automatic generation decision, which in turn is based on at least one of the following: defined parameters, automatic image processing of the first main video stream and / or the second main video stream, and automatic audio processing of the first main video stream and / or the second main video stream. c) Downsample the first main video stream and the second video stream to obtain the first shadow main video stream and the second shadow main video stream; d) Automatically generate a shadow video stream based on the first shadow main video stream and the second shadow main video stream, automatically generating the shadow video stream based on the corresponding automatic generation decisions used to generate the generated video stream; and e) Store the first shadow main video stream, the second shadow main video stream, the generated shadow video stream, and information for identifying automatically generated decisions.

[0018] The present invention also relates to another method for generating a video stream, the method comprising the following steps: a) Capture a first main video stream and a second main video stream that visually represent a first user, wherein the second main video stream optionally visually represents a second user; b) Automatically generate a video stream based on the first main video stream and the second main video stream. The automatic generation is based on an automatic generation decision, which in turn is based on at least one of the following: defined parameters, automatic image processing of the first main video stream and / or the second main video stream, and automatic audio processing of the first main video stream and / or the second main video stream. c) Downsample the first main video stream and the second video stream to obtain the first shadow main video stream and the second shadow main video stream; d) Optionally store at least one context-related information about the first main video stream and / or the second main video stream, which cannot be derived from the first shadow main video or the second shadow main video; d) At a later point in time, using the context-related information or other available information stored in step d), at least one of the first and second main video streams is upsampled to obtain a first enhanced main video stream and / or a second enhanced main video stream. This upsampling results in the addition of information to the upsampled main shadow video stream, the added information of which cannot be inferred from the upsampled main shadow video stream itself; and f) Automatically generate an enhanced generated video stream based on the first enhanced main video stream and / or the second enhanced main video stream.

[0019] In some embodiments, context-related information includes metadata describing at least one of the following: One or more events or patterns that can be detected in the first main video stream and / or the second main video stream; and One or more things, people or phenomena that are visually shown and / or audibly heard in the first main video stream and / or the second main video stream, such as the identity of a user who is visible or speaking in the first main video stream and / or the second main video stream, or the facial expressions of such a user.

[0020] The present invention also relates to another method for generating a video stream, the method comprising the following steps: a) Authenticate the first user; b) Capture a first main video stream that visually represents the first user, wherein the first main video stream includes a first main authentication video portion that visually represents the authentication steps and / or includes information that visually and / or audibly represents the first authentication token generated by the authentication steps; c) Use the first primary authentication video portion as a direct or derived input to compute the output of the first one-way function; d) Automatically generate a new video stream based on the first main video stream. This automatic generation is based on an automatic generation decision, which in turn is based on at least one of the following: defined parameters, automatic image processing of the first main video stream, and automatic audio processing of the first main video stream; and e) Provide the generated video stream to the receiving user, such as a second or third user.

[0021] The method further includes, for a piece of information representing the direct output or derived output of the first one-way function, performing at least one of the following: embedding a visual and / or audible representation of the information into a generated video stream; storing the information; and publicly publishing the information.

[0022] In some examples, the message representing the direct or derived output of the first one-way function can be referred to as the "first message".

[0023] In some embodiments, step b further includes capturing a second main video stream that visually represents the second user; and step d further includes automatically generating a generated video stream based on the second main video stream. Automatic generation may be based on automatic image processing of the second main video stream. Automatic generation may be based on automatic audio processing of the second main video stream.

[0024] In some embodiments, the method further includes: The output of the second one-way function is computed using the second main video stream as either direct or derived input. For a message representing the direct or derived output of the second one-way function, perform at least one of the following: embed the visual and / or audible representation of the message into the generated video stream; store the message; and publicly publish the message.

[0025] In some examples, the information representing the direct or derived output of the second one-way function can be referred to as the "second information".

[0026] In some embodiments, the method further includes: Provide the first user with at least one of the following: generate a video stream; or automatically generate different video streams based on the first main video stream and the second main video stream, the automatic generation being based on an automatic generation decision, and the automatic generation decision being based on at least one of the following: defined parameters, automatic image processing of the first main video stream and / or the second main video stream, and automatic audio processing of the first main video stream and / or the second main video stream.

[0027] In some embodiments, providing the corresponding generated video stream to the first user and the second user is performed in the context of an interactive video communication with the first user and the second user as participants, and forms part of such interactive video communication.

[0028] In some embodiments, authentication in step a includes capturing images of both the first user and the camera used to capture the first main video stream.

[0029] In some embodiments, authentication includes providing a one-time password to a first user, and the first user provides the one-time password audibly and / or visually, such that it forms part of a first master authentication video segment.

[0030] In some embodiments, the first primary authentication video portion is a part of the first primary video stream, which precedes the automatically generated portion of the primary video stream used to generate the video stream.

[0031] In some embodiments, the method further includes: A set of metadata is associated with and stored with the first main video stream and / or the generated video stream, the set of metadata including information on at least one of the following: the first authentication token; and the current authentication status of the first user.

[0032] In some embodiments, the method further includes: Continuous monitoring is performed, based at least on automatic image and / or audio processing of the first main video stream, to determine whether the first user has been continuously visible in the first main video stream since the authentication step.

[0033] In some embodiments, continuous monitoring is performed in the central server based on a first main video stream provided to the central server from the hardware entity in its raw or pre-processed format. The hardware entity is separate from the central server and is configured to continuously capture the first main video stream using a camera; and / or the continuous monitoring is performed by the hardware entity.

[0034] In some embodiments, the method further includes: Detect interruptions in the communication connection between the central server and the hardware entity.

[0035] In some embodiments, the method further includes: As a result of the detection, the continuous monitoring was switched from being performed by a central server to being performed by hardware entities. Re-establish communication connection; and As a result of regaining the communication connection, the continuous monitoring was switched from being performed by the hardware entity back to being performed by the central server.

[0036] In some embodiments, the method further includes: As a result of the interruption detection, continuous monitoring performed by the central server is suspended. Detect and re-establish communication connection; As a result of the re-acquired detection, the central server catches up with the reception of the main video stream; The central server performs continuous retrospective monitoring of the caught-up portions of the first main video stream; and The central server continues to perform continuous monitoring of the main video stream provided to it.

[0037] In some embodiments, the method further includes: A continuity score is calculated based at least on the automatic image processing and / or audio processing of the first main video stream, to determine the continuity score as a credibility measure of the extent to which the first user has been continuously visible and / or audibly present in the first main video stream since the self-authentication steps, or to determine the continuity score based on the credibility measure.

[0038] In some embodiments, the method further includes: If the continuous score falls within the predetermined range, repeat steps a and b to generate an updated first master authentication video portion, and then use the updated first master authentication video portion as the first master authentication video portion to repeat step c.

[0039] In some embodiments, the continuity score is determined based on at least one of the following: the visual appearance of the first user; the voice fingerprint of the first user; the activity level of the first user; the current conference role of the first user; the interaction between the first user and a second or third user, particularly the verification of the authenticity of the first user by the second or third user, who is also visible in the first main video stream, the second main video stream, or the third main video stream; the video quality and / or continuity of the first main video stream; changes in the main video stream showing the first user; and the time elapsed since the first authenticated video portion, in terms of the timeline of the first main video stream.

[0040] In some embodiments, the method further includes: Both the first and third users were detected to be visible in the first mainstream; and As a result of detecting that both the first user and the third user are visible, and optionally as a result of detecting that the third user actively confirms that the first user is real, the continuity score for the first user is increased.

[0041] In some embodiments, the method further includes: The authentication performed when step a is repeated is a different type of authentication than the authentication performed in the previous iterations of step a.

[0042] In some embodiments, the method further includes: The continuity score is shown in the generated video stream.

[0043] In some embodiments, the method further includes: Sampling is performed on publicly available information sources, and the samples are used as input to compute the output of the first source one-way function; The output of the first joint one-way function is calculated using the direct or derived output of the first source one-way function and the direct or derived output of the first one-way function as inputs; and Publicly release the direct or derived output of the first joint one-way function.

[0044] In some embodiments, the output of the first joint one-way function is iteratively computed using the direct update output or derived update output of the second one-way function as input, which in turn uses the second main video stream as direct input or derived input for computation.

[0045] In some embodiments, the output of the first joint one-way function is iteratively computed using the direct or derived update output of the first source one-way function, the direct or derived update output of the first one-way function, and the direct or derived output of the first joint one-way function from the previous iteration as inputs.

[0046] In some embodiments, the output of the first joint one-way function is also computed using at least one of the following as direct or derived inputs: a first master authentication video portion; a first authentication token; an updated continuity score; the output of a third one-way function, which is computed using the generated video stream as direct or derived inputs; and a set of metadata describing the complete set of generation steps for generating the generated video stream based on the first master video stream and the second video stream.

[0047] In some embodiments, the method further includes: A piece of information representing the direct or derived output of the first joint one-way function is visually and / or audibly embedded into at least one of: a first main video stream before the first main video stream is used to generate the generated video stream; a second main video stream before the second main video stream is used to generate the generated video stream; and the generated video stream.

[0048] In some embodiments, for different portions or frames of the first main video stream, the output of the first joint one-way function is repeatedly calculated over time, and the information is embedded into at least one of the first main video stream, the second main video stream, and the generated video stream.

[0049] In some embodiments, the embedding takes the form of at least one of the following: visual representation, such as alphanumeric information, QR code or barcode; visual watermark; and audible watermark.

[0050] In some embodiments, the first authentication is performed by the first user relative to a third party, and the first authentication is output to generate a first authentication token.

[0051] In some embodiments, step b includes embedding the information representing the token into a visual and / or audible portion of a first main video stream within a defined time period, such as up to 10 seconds, from the first authentication, the time period being defined by the timeline of the main video stream.

[0052] In some embodiments, the method further includes: At least one of the first main video stream and the second video stream is downsampled to obtain a downsampled first shadow main video stream and / or a downsampled second shadow main video stream. Perform all steps relating to the first main video stream, the second video stream, and the generated video stream in a corresponding manner, but using the corresponding first shadow main video stream and / or second shadow main video stream to generate the corresponding generated shadow video stream; and Store the first shadow main video stream and / or the second shadow main video stream, and generate the shadow video stream.

[0053] In some embodiments, the method further includes: Metadata about the automated generation steps performed to generate the shadow video stream is embedded into the generated shadow video stream.

[0054] In some embodiments, the generated shadow video stream is not shown to the first user or the second user.

[0055] In some embodiments, downsampling is configured to reduce the byte size by at least 10 times.

[0056] In some embodiments, downsampling includes at least one of the following: uniformly or selectively reducing the pixel resolution of the shadow video stream in the image plane of the shadow video stream in question compared to the main video stream in question; uniformly or selectively reducing the pixel depth of the shadow video stream in the image plane of the shadow video stream in question compared to the main video stream in question; reducing the frame rate of the shadow video stream in question compared to the main video stream in question; and adding a cropping of the shadow video stream in question compared to the main video stream in question.

[0057] In some embodiments, downsampling is performed without removing more than the amount of information from the main video stream such that the presence and identity of a user visible in the main video stream can also be visually identified in the shadow video stream.

[0058] In some embodiments, downsampling of the first main video stream and / or the second main video stream is dynamic in the sense that it takes into account one or more parameter values ​​for automatically generated decisions, such that, when viewed along the timeline of the main video stream in question, downsampling is applied differently over time.

[0059] In some embodiments, the method further includes: At least a portion of at least one of the first main video stream, the second main video stream, and the generated video stream is discarded, the discarding occurring after the generated video stream is provided to the first user.

[0060] In some embodiments, the generation of the corresponding shadow video stream occurs with a time delay of up to 10 seconds relative to the generation of the generated video stream.

[0061] In some embodiments, the method further includes: The output of the first shadow one-way function is calculated using the first shadow main video stream as a direct or derived input, and the output of the second shadow one-way function is calculated using the second shadow main video stream as a direct or derived input. The output of the first shadow one-way function is calculated using the first shadow authentication video portion of the first shadow main video stream as a direct or derived input. The first shadow authentication video portion corresponds to the first main authentication video portion, illustrating the authentication steps and / or representing the information of the token.

[0062] In some embodiments, the method further includes: For each corresponding piece of information representing the direct or derived output of the first and second shadow one-way functions, perform at least one of the following: embed the visual and / or audible representation of the information into the generated shadow video stream; store the information; and publicly publish the information.

[0063] In some examples, a message representing the direct or derived output of the first or second shadow one-way function may be referred to as the "first shadow message bar" or the "second shadow message bar," respectively.

[0064] In some embodiments, the method further includes: Sampling is performed on publicly available information sources, and the samples are used as input to compute the output of the second source one-way function; The output of the second joint one-way function is calculated using the direct or derived output of the second source one-way function, the direct or derived output of the first shadow one-way function, and the direct or derived output of the second shadow one-way function as inputs; and Publicly release the direct or derived output of the second joint one-way function.

[0065] In some embodiments, the method further includes: Embed the direct or derived output of the first shadow master one-way function into the first master video stream.

[0066] In some embodiments, the method further includes: Embed the direct or derived output of the first one-way function into the first shadow main video stream.

[0067] In some embodiments, the method further includes: Automatically convert the first shadow main video stream, the second shadow main video stream, and / or generate a shadow video stream to obtain a converted video stream; and The converted video stream is provided to the user.

[0068] In some embodiments, the conversion includes at least one of the following: increasing the audio bitrate, pixel resolution, pixel depth, and / or frame rate; introducing de-cropping, image elements, and / or audio elements; and partially or completely animates the participants shown in the converted video stream, the conversion being based on previously stored metadata.

[0069] In some embodiments, the method further includes: Metadata is determined based on at least one of the first main video stream, the second main video stream, and the generated video stream.

[0070] In some embodiments, metadata includes information provided externally.

[0071] In some embodiments, downsampling is irreversible.

[0072] In some embodiments, the method further includes: A fourth main video stream visually representing the first user is captured from a viewpoint different from the first main video stream, wherein the fourth main video stream includes a fourth main authentication video portion that visually illustrates the authentication steps and / or includes information visually and / or audibly representing the token; and The output of the fourth one-way function is computed using the fourth primary authentication video portion as either a direct or derived input.

[0073] In some embodiments, the method further includes: Sampling is performed on publicly available information sources, and the samples are used as input to compute the output of a third-source one-way function; The output of the third joint one-way function is calculated using the direct or derived output of the third source one-way function and the direct or derived output of the fourth one-way function as inputs; and Publicly release the direct or derived output of the third joint one-way function.

[0074] In some embodiments, the automatic generation of the generated video stream is further based on a fourth main video stream.

[0075] In some embodiments, the automatic generation of the generated video stream is not further based on a fourth main video stream.

[0076] In some embodiments, the method further includes: Embed the direct or derived output of the fourth one-way function into the first main video stream.

[0077] In some embodiments, the method further includes: Embed the direct or derived output of the first one-way function into the fourth main video stream.

[0078] The present invention also relates to a system for generating video streams, the system comprising a central server and a client, the system being configured to perform methods of the type described above.

[0079] Furthermore, the present invention relates to a computer program product for generating a video stream, the computer program product including instructions configured to perform a method of the type described above when executed.

[0080] The computer program product may be implemented by a non-transitory computer-readable medium that encodes instructions that cause one or more hardware processors in at least one of the computer hardware devices located in the system to perform the method of the type described. Attached Figure Description

[0081] In the following, the present invention will be described in detail with reference to exemplary embodiments and accompanying drawings, wherein: Figure 1 A first exemplary system is shown; Figure 2 A second exemplary system is shown; Figure 3 A third exemplary system is shown; Figure 4 The central server is shown; Figure 5 The first method is shown; Figures 6a to 6f It shows the relationship with Figure 5 The subsequent states related to different method steps in the method shown; Figure 7 A public protocol is conceptually illustrated; Figure 8 The second method is shown; Figure 9 A fourth exemplary system is shown; Figure 10 The fifth exemplary system is shown; Figure 11 It illustrates the interrelationships between a set of video streams, information sources, and distribution channels along a timeline; Figure 12 A third method is shown; Figure 13 The fourth method is shown; and Figure 14 It shows the interrelationships between a set of video streams, information sources, and distribution channels along the timeline.

[0082] All figures share figure labels for the same or corresponding parts. Detailed Implementation

[0083] Figure 1 A system 100 according to the invention is shown, which is configured to perform the method according to the invention to provide an output digital video stream, such as a shared digital video stream.

[0084] As used herein, the terms “video” and “video stream” include image material, such as a sequence of image frames. “Video” or “video stream” may also include one or more corresponding audio information tracks.

[0085] System 100 may include video communication service 110, but in some embodiments, video communication service 110 may also be external to system 100. As will be discussed, there may be more than one video communication service 110.

[0086] System 100 may include one or more participant clients 121, but in some embodiments, one, some or all of the participant clients 121 may also be outside of system 100.

[0087] System 100 may include a central server 130.

[0088] As used herein, the term "central server" refers to a computer implementation function that is configured for logically centralized access, such as via a well-defined API (Application Programming Interface). Such a central server function can be implemented purely in computer software, or as a combination of software and virtual and / or physical hardware. It can be implemented on a standalone physical or virtual server computer, or distributed across multiple interconnected physical and / or virtual server computers.

[0089] As will be exemplified below, in some embodiments, the central server 130 comprises a single piece of hardware, or is a single piece of hardware, locally located relative to one or more of the participating clients 121. As used herein, "locally located" relative to each other means that they are located in the same place, such as in the same building, for example, in the same room, and are preferably interconnected using dedicated cables or local area network connections for local communication, rather than interconnected via the open Internet. As will be described below, each participating client 121 may be its own system (or "central server" as defined below) in terms of hardware and / or software.

[0090] The physical or virtual hardware running on the central server 130 (in other words, the physical or virtual hardware on which the computer software defining the functionality of the central server 130 executes) may include a CPU, a GPU, RAM / ROM memory, a computer bus, and external communication functions such as Internet connectivity.

[0091] To the extent that each video communication service 110 is used, each video communication service 110 is also a central server in the aforementioned sense, which may be a central server different from central server 130, or a part of central server 130. In particular, video communication service 110, or each video communication service 110, may be locally deployed in relation to one, more, or all participating clients 121.

[0092] Accordingly, each participant client 121 may be a central server in the sense described above, and has a corresponding interpretation, and the physical or virtual hardware running on each participant client 121 (in other words, the physical or virtual hardware on which the computer software defining the functionality of the participant client 121 executes) may also include a CPU / GPU, RAM / ROM memory, computer bus, and external communication functions such as Internet connectivity.

[0093] Each participant client 121 typically also includes: a computer screen or communication with a computer screen configured to display video content provided to the participant client 121 as part of an ongoing video communication; one or more speakers configured to emit sound content provided to the participant client 121 as part of the video communication; one or more video cameras; and one or more microphones configured to locally record the voice of a human participant 122 participating in the video communication using the participant client 121 in question.

[0094] In other words, the corresponding human-machine interface of each participant client 121 allows the corresponding participant 122 to interact with the client 121 in question during video communication, and to interact with other participants and / or audio / video streams provided by various sources.

[0095] In general, each participating client 121 includes a corresponding input device 123, which may include one or more of the video cameras; one or more of the microphones; a keyboard; a computer mouse or touchpad; and / or an API for receiving digital video streams, digital audio streams, and / or other digital data. The input device 123 is specifically configured to receive video and / or audio streams from a central server, such as from video communication service 110 and / or central server 130, which are provided as part of video communication and are preferably generated based on corresponding digital data input streams from at least two sources provided to the central server, for example from participating client 121 and / or external sources (see below).

[0096] Further, in general, each participating client 121 includes a corresponding output device 124, which may include the computer screen; one or more of the speakers; and an API for emitting digital video and / or audio streams, which represent video and / or audio captured locally by the participant 122 using the participating client 121 in question.

[0097] In practice, each participant client 121 may be a mobile device, such as a mobile phone, equipped with a screen, speaker, microphone, and internet connection, which locally executes computer software or accesses remotely executed computer software to perform the functions of the participant client 121 discussed. Accordingly, the participant client 121 may also be a bulk or thin laptop computer or a stationary computer, executing locally installed applications, using remote access functions via a web browser, etc., as appropriate. Each participant client 121 may also include any peripherally connected devices, such as any external camera, microphone, and / or speaker.

[0098] In one or more video communications of this type, more than one participant client 121 may be used, such as at least three or even at least four participant clients 121.

[0099] There may be at least two distinct groups of participating clients. Each participating client can be assigned to such a corresponding group. These groups may reflect different roles of the participating clients, different virtual or physical locations of the participating clients, and / or different interaction permissions of the participating clients.

[0100] Various available roles can be, for example, “leader,” “meeting moderator,” “speaker,” “group participant,” “interactive audience,” “remote audience,” etc.

[0101] Various available physical locations could be, for example, “physically in a room,” “remotely listening,” “on stage,” “in a group,” “among physically present audience members,” “among physically remote audience members,” etc.

[0102] Virtual locations can be defined based on physical locations, but can also involve virtual groups (such as “group rooms,” a common feature in conventional video conferencing solutions) that may partially overlap with the physical locations. For example, physically present viewers can be divided into a first virtual group and a second virtual group, and some physically present viewers can be grouped with physically distant viewers in the same virtual group.

[0103] Various available interactive permissions could be, for example, “fully interactive” (unrestricted), “able to speak but only after requesting a microphone” (such as raising a virtual hand in a video conferencing service), “unable to speak but able to write in a public chat”, or “watch / listen only”. Other examples of interactive permissions that can be granted (or not granted) to various groups include the right to comment on, vote on, or approve / disapprove a topic or issue.

[0104] In some cases, each defined role and / or physical / virtual location may be defined based on certain predefined interaction permissions. In other cases, all participants with the same interaction permissions form a group. Therefore, any defined roles, locations, and / or interaction permissions may reflect various group assignments, and different groups may, as appropriate, not overlap or intersect with each other.

[0105] Video communication may be provided at least in part by video communication service 110 and at least in part by central server 130, as will be described and exemplified herein.

[0106] As used herein, “video communication” refers to an interactive digital communication session involving at least two, preferably at least three, or even at least four video streams, and preferably also matching audio streams, used to generate one or more mixed or combined digital video / audio streams, which are then used by one or more consumers (such as participant clients of the type discussed), who may or may not also contribute to the video communication via video and / or audio. Such video communication is real-time, with or without a certain delay or latency. At least one, preferably at least two, or even at least four participants 122 of such video communication participate interactively in the video communication, both providing and consuming video / audio information.

[0107] At least one or all of the participant clients 121 may include a local synchronization software function 125, which will be described in more detail below.

[0108] The video communication service 110 may include or have access to a public time reference, as will also be described in more detail below.

[0109] Each of at least one of the central servers 130 may include a corresponding API 137 for digital communication with entities outside the central server 130 in question. Such communication may involve both input and output.

[0110] System 100 (such as the central server 130) may be further configured to digitally communicate with external information sources 300, and specifically to receive digital information from external information sources 300, such as audio and / or video streaming data, for example, externally provided video streams. "External" information source 300 means that it is not provided from or as part of the central server 130. Preferably, the digital data provided by the external information source 300 is independent of the central server 130, and the central server 130 cannot influence its information content. For example, the external information source 300 may be live video and / or audio, such as video and / or audio of a public sporting event or ongoing news event or report. The external information source 300 may also be captured by a webcam or the like, but not by any of the participating clients 121. Thus, such captured video may show the same location as any of the participating clients 121, but is not captured as part of the activities of the participating client 121 itself. One possible difference between externally provided information sources 300 and internally provided information sources 120 is that internally provided information sources can be provided as participants in the video communication of the type defined above and in their capacity as participants, while externally provided information sources 300 are not, but are provided as part of a context located outside the video conference. In other embodiments, one or more externally provided information sources 300 take the form of a corresponding digital camera or microphone, configured to capture corresponding digital image / video and / or audio streams in the same location as one or more participating clients 121 and / or corresponding users 122, and in a manner controlled by a central server 130. Therefore, the central server 130 can control the on / off state of such digital image / video / audio capture devices 300, and / or other capture states, such as physical or virtual panning or zooming of the current application. External information sources 300 may also or alternatively provide non-video data, such as one or more still images, sound information, static digital information (such as text and / or numbers), etc.

[0111] Multiple external information sources 300 may also exist, which provide digital information of the type described above, such as audio and / or video streams, to the central server 130 in parallel.

[0112] like Figure 1 As shown, each participating client 121 may constitute a source of a corresponding information (video and / or audio) stream 120, which is provided to the video communication service 110 by the participating client 121 in question as described.

[0113] System 100 (such as a central server 130) may be further configured to digitally communicate with external consumers 150, and specifically to send digital information to external consumers 150. For example, digital video and / or audio streams generated by the central server 130 may be continuously, in real-time, or near real-time provided to one or more external consumers 150 via the API 137. Similarly, "external" for consumer 150 means that consumer 150 is not provided as part of the central server 130, and / or is not a party to the video communication. "Not a party to the video communication" may mean that consumer 150 only accepts input from the video communication, such as in the form of a generated video stream, but cannot interactively provide such information to the video communication to achieve interactivity.

[0114] Unless otherwise stated, all functions and communications herein are provided digitally and electronically, implemented by computer software executing on appropriate computer hardware, and communicated via local or global digital communication networks or channels such as the Internet.

[0115] Therefore, in Figure 1 In the system 100 configuration shown, multiple participant clients 121 participate in digital video communication provided by video communication service 110. Therefore, each participant client 121 may have an ongoing login, session, or similar connection to video communication service 110 and may participate in the same ongoing video communication provided by video communication service 110. In other words, the video communication is "shared" among participant clients 121 and therefore also shared by the corresponding human participants 122.

[0116] exist Figure 1In this configuration, the central server 130 includes an automated participant client 140, which is an automated client corresponding to participant client 121 but not associated with human participant 122. Instead, the automated participant client 140 is added to the video communication service 110 as a participant client to participate in the same shared video communication as participant client 121. As such a participant client, the automated participant client 140 is granted access to continuously generated digital video and / or audio streams as part of ongoing video communication provided by the video communication service 110, and these streams can be consumed by the central server 130 via the automated participant client 140. Preferably, the automated participant client 140 receives public video and / or audio streams from the video communication service 110, which are distributed or may be distributed to one, multiple, or each participant client 121; receives corresponding video and / or audio streams provided to the video communication service 110 from one or more participant clients 121, and relayed by the video communication service 110 in their original or modified form to all or requesting participant clients 121; and / or receives a public time reference.

[0117] The central server 130 may include a collection function 131 configured to receive video and / or audio streams of the type described above from the automatic participant client 140, and possibly also from the external information source 300, for processing as described below, and then provide the generated, such as shared, video stream via API 137. For example, the generated video stream may be consumed by an external consumer 150 and / or by a video communication service 110, and then distributed by the video communication service 110 to all participant clients 121 or any requesting participant client 121.

[0118] Figure 2 Similar to Figure 1 However, instead of using the automatic client participant 140, the central server 130 receives video and / or audio stream data from the ongoing video communication via API 112 of the video communication service 110.

[0119] Figure 3 Similar to Figure 1 However, video communication service 110 is not shown. In this case, participant client 121 communicates directly with API 137 of central server 130, for example, providing video and / or audio stream data to and / or receiving video and / or audio stream data from central server 130. The resulting shared stream can then be provided to external consumer 150 and / or one or more participant clients 121.

[0120] Figure 4 The central server 130 is shown in more detail. As shown, the collection function 131 may include one or more format-specific collection functions 131a. Each of the format-specific collection functions 131a may be configured to receive video and / or audio streams having a predetermined format, such as a predetermined binary encoding format and / or a predetermined stream data container, and may be specifically configured to parse the binary video and / or audio data of the format into individual video frames, sequences of video frames, and / or time slots.

[0121] The central server 130 may further include an event detection function 132 configured to receive video and / or audio stream data, such as binary stream data, from the collection function 131, and perform corresponding event detection for each individual data stream in the received streams. The event detection function 132 may include an AI (artificial intelligence) component 132a for performing the event detection. Event detection can be performed without first synchronizing the individual collection streams in time.

[0122] The central server 130 further includes a synchronization function 133 configured to time-synchronize data streams provided by the collection function 131 and that may be processed by the event detection function 132. The synchronization function 133 may include an AI component 133a for performing the time synchronization.

[0123] The central server 130 may further include a pattern detection function 134, which is configured to perform pattern detection based on a combination of at least one, but in many cases at least two (such as at least three or even at least four, such as all) data streams received. Pattern detection may further be based on one, or in some cases at least two or more, events detected by the event detection function 132 for each individual data stream in the data streams. Such detected events considered by the pattern detection function 134 may be distributed across time relative to each individually collected stream. The pattern detection function 134 may include an AI component 134a for performing the pattern detection. Pattern detection may further be based on the groupings discussed above, and is specifically configured to detect specific patterns occurring only relative to one group; specific patterns occurring only relative to some groups but not all groups; or specific patterns occurring relative to all groups.

[0124] The central server 130 further includes a generation function 135, which is configured to generate digital video streams, such as shared digital video streams, based on data streams or multiple data streams provided from the collection function 131, and further based on any detected events and / or patterns. Such generated video streams may at least include video streams generated to include one or more video streams provided by the collection function 131, which may be raw, reformatted, or converted, and may also include corresponding audio stream data. As will be illustrated below, several generated video streams may exist, one of which may be generated in the manner discussed above, but further based on another already generated video stream.

[0125] All generated video streams are preferably generated continuously, and preferably generated in near real-time (after deducting any latency and delay of the types discussed below).

[0126] The central server 130 may further include a publishing function 136, which is configured to publish the generated digital video stream in question, such as via API 137 as described above.

[0127] Notice, Figure 1 , Figure 2 and Figure 3 Three different examples of how the central server 130 can be used to implement the principles described herein are shown, particularly providing the method according to the invention, but other configurations are also possible, whether or not one or more video communication services 110 are used.

[0128] Figure 5 A method for generating digital video streams is shown. Figures 6a to 6f It shows the result of Figure 5 The steps shown illustrate the different states of digital video / audio data streams produced.

[0129] In the first step S500, the method begins.

[0130] In the subsequent collection step S501, corresponding master digital video streams 210, 301 are collected from at least two of the digital video sources 120, 300, such as by the collection function 131. Each such master data stream 210, 301 may include an audio portion 214 and / or a video portion 215. It should be understood that "video" in this context refers to the moving and / or still image content of such data streams, which may or may not include audio following the visible content of the video. Each master data stream 210, 301 may be encoded according to any video / audio coding specification (using the appropriate codec used by the entity providing the master streams 210, 301 in question), and the encoding format may differ between different master streams 210, 301 used simultaneously in the same video communication. Preferably, at least one (such as all) of the master data streams 210, 301 is provided as a binary data stream, possibly in a data container data structure that is itself conventional. Preferably, at least one (such as at least two, or even all) of the main data streams 210, 301 is provided as a corresponding real-time video recording. One or more main data streams 210, 301 may be provided alternatively or additionally as existing digital video resources, or as digital video resources constructed in real-time in association with the collection (but not recorded using a camera). For example, the main video streams 210, 301 may be digital video streams constructed as a series of images based on 3D data or (time-sequential) rendering of static documents themselves.

[0131] Note that when the main streams 210 and 301 are received by the collection function 131, they may be out of sync in terms of time. This could mean that they are associated with different time delays or latency relative to each other. For example, in the case where the two main video streams 210 and 301 are recorded in real time, this could mean that when they are received by the collection function 131, they are associated with different time delays relative to the recording time.

[0132] Also note that the mainstream 210, 301 can themselves be a corresponding live camera feed from a webcam; a currently shared screen or presentation; a movie clip or similar content being watched; or any combination of these arranged in various ways on the same screen.

[0133] Collection step S501 is shown in Figure 6a and Figure 6b In. Figure 6b The document also illustrates how the collection function 131 can store each main video stream 210, 301 as bundled audio / video information, or as audio stream data separate from the associated video stream data. Figure 6bThis illustrates how the main video stream 210, 301 data is stored as individual frames 213 or collections / clusters of frames, where a “frame” refers to a time-limited portion of image data and / or any associated audio data, such as each frame being a separate still image or a series of consecutive images (such as a series that constitutes at most 1 second of moving images), which together form the moving image video content.

[0134] In the subsequent event detection step S502 performed by event detection function 132, the main digital video streams 210, 301 may be analyzed, such as by event detection function 132 (e.g., the AI ​​component 132a), to detect at least one event 211 selected from the first group of events. This is in Figure 6c As shown in the image.

[0135] Preferably, the event detection step S502 can be performed for at least one (such as at least two, such as all) main video streams 210, 301, and can be performed individually for each such main video stream 210, 301. In other words, the event detection step S502 is preferably performed for the individual main video stream 210, 301, considering only the information contained as part of the particular main video stream 210, 301 in question, and in particular not considering information contained as part of other main video streams. Furthermore, the event detection is preferably performed without considering any common time reference 260 associated with the multiple main video streams 210, 301.

[0136] On the other hand, event detection preferably considers information contained across a certain time interval as part of the main video stream being analyzed separately, such as historical time intervals of the main video stream that are longer than 0 seconds (e.g., at least 0.1 seconds, or at least 1 second).

[0137] Event detection may take into account information contained in the audio and / or video data included as part of the main video streams 210 and 301.

[0138] The first set of events may include any number and type of events, such as changes to slides in the main video streams 210, 301 or slide presentations that constitute part of them; changes in the connection quality of the sources 120, 300 providing the main video streams 210, 301, resulting in changes in image quality, loss of image data, or recovery of image data; and motion-based physical events detected in the main video streams 210, 301, such as movement of people or objects in the video, changes in lighting in the video, sudden sharp noise in the audio, or changes in audio quality. It should be understood that this is not intended as an exhaustive list, but rather these examples are provided to illustrate the applicability of the principles currently described.

[0139] In the subsequent synchronization step S503 performed by synchronization function 133, the main digital video stream 210 can be time-synchronized. This time synchronization can be performed relative to a common time reference 260. Figure 6d As shown, time synchronization may involve aligning the main video streams 210 and 301 relative to each other, for example, using the common time reference 260, so that they can be combined to form a time-synchronized context. The common time reference 260 may be a data stream, a heartbeat signal, or other pulse data, or a time anchor applicable to each of the individual main video streams 210 and 301. The common time reference can be applied to each of the individual main video streams 210 and 301 such that the information content of the main video streams 210 and 301 in question can be explicitly associated with the common time reference relative to a common time axis. In other words, the common time reference allows the main video streams 210 and 301 to be aligned via time shifting for time synchronization in the present sense. In other embodiments, time synchronization may be based on known information about the time difference between the main video streams 210 and 301 in question, such as based on measurements.

[0140] like Figure 6d As shown, time synchronization may include determining one or more timestamps 261 for each master video stream 210, 301, such as relative to a common time base 260, or determining one or more timestamps 261 for each video stream 210, 301 relative to another video stream 210, 301 or other video streams 210, 301.

[0141] In the subsequent pattern detection step S504 performed by the pattern detection function 134, the master digital video streams 210 and 301, which are thus time-synchronized, can be analyzed to detect at least one pattern 212 selected from the first set of patterns. This is in Figure 6e As shown in the image.

[0142] In contrast to event detection step S502, pattern detection step S504 may be performed based on video and / or audio information contained in at least two time-synchronized main video streams 210, 301 that are considered together.

[0143] The first set of patterns can contain any number of patterns of any type, such as multiple participants speaking alternately or simultaneously; or presentation slides changing and different events occurring simultaneously, such as different participants speaking. This list is not exhaustive but illustrative.

[0144] In some embodiments, the detected pattern 212 may not involve information contained in multiple main video streams of the main video streams 210, 301, but only information contained in one of the main video streams 210, 301. In this case, preferably, such pattern 212 is detected based on video and / or audio information contained in at least two detected events 211 in that single main video stream 210, 301, such as two or more consecutively detected presentation slide changes or connection quality changes. As an example, multiple consecutive slide changes occurring rapidly in succession over time can be detected as a single slide change pattern, rather than a separate slide change pattern for each detected slide change event. Other examples include the movement of shown entities or people; and the recognition of spoken phrases by participant users.

[0145] It should be understood that the first group of events and the first group of patterns may include events / patterns of predetermined types, which are defined using corresponding parameter sets and parameter ranges. As will be explained below, the events / patterns in the group may also be defined and detected using various AI tools, or alternatively.

[0146] In the subsequent generation step S505, performed by generation function 135, a digital video stream is generated as an output digital video stream 230 based on frames 213 of the main digital video streams 210 and 301 that may be time-synchronized, and further based on the detected events 211 and / or the detected patterns 212. The generated digital video stream 230 may or may not be a "shared" video stream, meaning it is provided to more than one participant client 121. Generation may also involve generating different output digital video streams 230 for different participant clients 121 and / or external consumers 150.

[0147] As will be explained and detailed below, the present invention allows for the fully automated generation of video streams, such as the fully automated generation of one or more output digital video streams 230.

[0148] For example, this generation may involve selecting what video and / or audio information from which main video streams 210, 301 are used for this output video stream 230 to what extent; the video screen layout of the output video stream 230; switching modes between different uses or layouts across time; and so on.

[0149] This is Figure 6f As shown in the figure, Figure 6fOne or more additional time-dependent (which may be associated with a common time base 260) digital video information 220 are also shown, such as additional digital video information streams, which can be time-synchronized (e.g., synchronized to the common time base 260) and used in conjunction with the time-synchronized main video streams 210, 301 in the generation of the output video stream 230. For example, the additional stream 220 may include information about any video and / or audio effects to be used, such as effects dynamically used based on detected patterns; planned timing of video communications; and so on.

[0150] In the subsequent publishing step S506 performed by publishing function 136, one or more generated output digital video streams 230 are continuously provided to one or more consumers 110, 121, 150 of the generated digital video stream as described above. The generated digital video stream may be provided to one or more participant clients 121, such as via video communication service 110.

[0151] In the subsequent step S507, the method terminates. However, initially, the method can iterate an arbitrary number of times, such as... Figure 5 As shown, the output video stream 230 is generated as a continuously provided stream. Preferably, the output video stream 230 is generated for real-time or near-real-time consumption (considering the total latency added by all steps along the way) and is consumed continuously (published immediately when more information is available, but without calculating any intentionally added latency or delay, see below). In this way, one or more output video streams 230 can be consumed interactively, such that each output video stream 230 can be fed back to the video communication service 110 or any other context that forms the basis for generating the main video stream 210, which is then fed back to the collection function 131 to form a closed feedback loop; or each output video stream 230 can be consumed in a different context (outside of system 100 or at least outside of the central server 130), but in which context forms the basis for real-time interactive video communication.

[0152] As described above, in some embodiments, at least two (such as at least three, at least four, or even at least five) of the main digital video streams 210, 301 are provided as part of a shared digital video communication, such as that provided by the video communication service 110, which involves a participant client 121 providing a corresponding remote connection to the main digital video stream 210 in question. In this case, the collection step S501 may include collecting at least one of the main digital video streams 210 from the shared digital video communication service 110 itself, such as via an automatic participant client 140, which in turn is granted access to video and / or audio stream data within the video communication service 110 in question; and / or via API 112 of the video communication service 110.

[0153] Furthermore, in this and other cases, the collection step S501 may include collecting at least one of the main digital video streams 210, 301 as a corresponding external digital video stream 301, which is collected from an information source 300 outside the shared digital video communication service 110. Note that one or more of these external video sources 300 may also be located outside the central server 130.

[0154] In some embodiments, the main video streams 210 and 301 are not formatted in the same way. This different formatting may be in the form of them being delivered to the collection function 131 in different types of data containers (such as AVI or MPEG), but in a preferred embodiment, at least one of the main video streams 210 and 301 is formatted according to a deviation format (compared to at least one of the main video streams 210 and 301), the aspects of which are that the deviation of the main digital video streams 210 and 301 has a deviation video encoding; a deviation fixed or variable frame rate; a deviation aspect ratio; a deviation video resolution; and / or a deviation audio sampling rate.

[0155] Preferably, the collection function 131 is pre-configured to read and interpret all encoding formats, container standards, etc., appearing in all collected main video streams 210, 301. This allows the processing described herein to be performed without requiring any decoding until relatively late in the processing (e.g., until the main stream in question is placed in its corresponding buffer; until after event detection step S502; or even until after event detection step S502). However, in rare cases where one or more main video feeds 210, 301 are encoded using codecs that cannot be interpreted without decoding by the collection function 131, the collection function 131 may be configured to perform decoding and analysis of such main video streams 210, 301, subsequently converting them into a format that can be processed by, for example, the event detection function. Note that even in this case, it is preferable not to perform any re-encoding at this stage.

[0156] For example, a main video stream 220 obtained from a multi-party video event, such as a main video stream provided by video communication service 110, typically requires low latency and is therefore usually associated with variable frame rates and variable pixel resolutions to enable participants 122 to communicate effectively. In other words, for low latency, the overall video and audio quality will be reduced as needed.

[0157] On the other hand, external video feed 301 will generally have a more stable frame rate and higher quality, but may therefore have higher latency.

[0158] Therefore, the video communication service 110 may use different encodings and / or containers than the external video source 300 at any given time. Consequently, in this case, the analysis and video generation process described herein requires combining these different format streams 210, 301 into a new stream for the combined experience.

[0159] As described above, the collection function 131 may include a set of format-specific collection functions 131a, each format-specific collection function 131a being configured to process the main video streams 210, 301 of a specific type format. For example, each of these format-specific collection functions 131a may be configured to process the main video streams 210, 301 encoded using different video-specific encoding methods / codecs, such as Windows. ® Media ® Or DivX ® .

[0160] However, in some embodiments, the collection step S501 includes converting at least two (such as all) of the main digital video streams 210, 301 into the common protocol 240.

[0161] As used in this context, the term "protocol" refers to an information structuring standard or data structure that specifies how information contained in a digital video / audio stream is stored. However, common protocols preferably do not specify how digital video and / or audio information (i.e., encoded / compressed data indicating the sound and image itself) is stored at the binary level, but rather form a structure in a predetermined format for storing such data. In other words, common protocols specify storing digital video data in its raw binary form without performing any digital video decoding or encoding associated with this storage, possibly without modifying the existing binary form at all except for possibly concatenating and / or splitting the byte sequences. Instead, the raw (encoded / compressed) binary data content of the main video streams 210, 301 under discussion is preserved, while this raw binary data is repackaged into a data structure defined by the protocol. In some embodiments, the common protocol defines a video file container format.

[0162] Figure 7 As an example, it is shown Figure 6a The main video streams 210 and 301 shown are reconstructed by the corresponding format-specific collection function 131a and use the public protocol 240.

[0163] Therefore, public protocol 240 specifies that digital video and / or audio data be stored in dataset 241, preferably divided into discrete, continuous datasets along a timeline relating to the main video streams 210, 301 in question. Each such dataset may include one or more video frames and also include associated audio data.

[0164] The public protocol 240 may also specify the storage of metadata 242 associated with a specified point in time relative to the stored digital video and / or audio dataset 241.

[0165] Metadata 242 may include information about the raw binary format of the main digital video stream 210 in question, such as the digital video encoding method or codec used to generate the raw binary data; the resolution of the video data; the video frame rate; a frame rate variability flag; the video resolution; the video aspect ratio; the audio compression algorithm; or the audio sampling rate. Metadata 242 may also include information about the timestamps of the stored data, such as a time base relative to the main video stream 210, 301 in question or relative to the different video streams discussed above.

[0166] The combination of the format-specific collection function 131a and the common protocol 240 enables the rapid collection of information content from the main video streams 210 and 301 without increasing latency by decoding / re-encoding the received video / audio data.

[0167] Therefore, the collection step S501 may include using different format-specific collection functions in the format-specific collection function 131a to collect the main digital video streams 210, 301 encoded using different binary video and / or audio encoding formats, in order to parse the main video streams 210, 301 in question, and store the parsed raw binary data along with any relevant metadata in a data structure using a common protocol. It is evident that the determination of which format-specific collection function 131a to use for which main video streams 210, 301 can be performed by the collection function 131 based on the predetermined and / or dynamically detected attributes of each main video stream 210, 301 in question.

[0168] Therefore, each collected master video stream 210, 301 can be stored in its own separate memory buffer, such as a RAM memory buffer, in the central server 130.

[0169] Therefore, the conversion of the main video streams 210, 301 performed by each format-specific collection function 131a may include splitting the raw binary data of each such converted main digital video stream 210, 301 into an ordered set of the smaller datasets 241.

[0170] Furthermore, the conversion may also include associating each (or subset, such as a subset regularly distributed along the respective timelines of the main streams 210, 301 in question) of the smaller sets 241 with a corresponding time along a shared timeline, such as relative to the common time reference 260. This association can be performed by analyzing the raw binary video and / or audio data in any of the principles described below or otherwise, and can be performed to enable subsequent time synchronization of the main video streams 210, 301. Depending on the type of common time reference used, at least a portion of this association for each dataset 241 may also be performed, or alternatively, by the synchronization function 133. In the latter case, the collection step S501 may alternatively include associating each or a subset of the smaller sets 241 with a corresponding time along the specific timeline of the main streams 210, 301 in question.

[0171] In some embodiments, the collection step S501 further includes converting the raw binary video and / or audio data collected from the main video streams 210, 301 into a uniform quality and / or update frequency. This may involve downsampling or upsampling the raw binary digital video and / or audio data of the main digital video streams 210, 301 as needed to achieve a common video frame rate; a common video resolution; or a common audio sampling rate. Note that such resampling can be performed without performing full decoding / re-encoding, or even without performing any decoding at all, because the format-specific collection function 131a in question can directly process the raw binary data according to the correct binary encoding target format.

[0172] Each of the main digital video streams 210 and 301 can be stored as a separate frame 213 or a sequence of frames 213 in a separate data storage buffer 250 as described above, and is also associated with a corresponding timestamp, which in turn is associated with the common time base 260.

[0173] In providing specific examples to illustrate these principles, Video Communication Service 110 is Microsoft ® Teams ® Its operation involves a video conference with 122 concurrent participants. Automatic participant client 140 is registered as a Teams client. ® Participants in the meeting.

[0174] The main video input signal 210 is then available to the collection function 130 and is acquired by the collection function 130 via the automatic participant client 140. These are raw signals in H.264 format and contain timestamp information for each video frame.

[0175] The format-specific collection function 131a picks up raw data over IP (LAN network) on a configurable, predefined TCP port. Each Team ® Each conference participant and associated audio data is associated with a separate port. The collection function 131 then uses a timestamp from the audio signal (which is 50 Hz) and downsamples the video data to a fixed output signal of 25 Hz before storing the video stream 220 in its respective separate buffer 250.

[0176] As described above, the common protocol 240 can store data in its raw binary form. It can be designed to be very low-level and handles the raw bits and bytes of video / audio data. In a preferred embodiment, data is stored in the common protocol 240 as a simple array of bytes or a corresponding data structure (such as slices). This means that the data does not need to be placed in a conventional video container (in this context, the common protocol 240 does not constitute such a conventional container). Furthermore, encoding and decoding video is computationally expensive, meaning it causes latency and requires costly hardware. Moreover, this problem scales with the number of participants.

[0177] Using public protocol 240, it is possible to collect data for each Team in function 131. ® The main video stream 210 associated with conference participants 122, as well as any external video sources 300, are reserved in memory, and the amount of memory allocated can be changed on the fly during the process. In this way, the number of input streams can be changed and thus each buffer remains valid. For example, since information such as resolution and frame rate is variable but stored as metadata in the common protocol 240, this information can be used to quickly adjust the size of each buffer as needed.

[0178] The following is an example of the specification for this type of public protocol 240: Byte example description 1 byte = 1 0 = video; 1 = audio 4-byte buffer length (int) 1234567 8 bytes 424234234 Timestamp from the incoming audio / video buffer Measured in ticks, 1 tick = 100 ns. (long int) 1 byte 0 VideoColorFormat { NV12 = 0, Rgb24 = 1, Yuy2 = 2, H264 = 3 } 4 bytes, 720 pixels, video frame height (int) 4 bytes, 640 pixels wide of video frame (int) 4 bytes 25.0 video frame rate Frames per second (float) Does a 1-byte 0 indicate audio mute? 1 = True; 0 = False 1 byte 0 AudioFormat { 0 = Pcm16K 1 = Pcm44Kstereo } 1 byte 0 Detected event, if 0 = No event 1, 2, 3, etc. = Detected an event of the specified type. 30 bytes reserved for future use The length of 8 bytes (1000000) of binary data, in bytes (long int). The variable 0x87A879… represents the raw binary video / audio data(s) of this frame or these frames. 4 bytes 1234567 Dominant speaker port 4 bytes 1234567 Active Speakers The "Detected Events, if any" data is included as part of the Common Protocol 240 specification. However, in some embodiments, this information (regarding detected events) may alternatively be placed in a separate memory buffer.

[0179] In some embodiments, the at least one additional digital video information 220 (which may be an overlay or effect) is also stored in a corresponding separate buffer 250 as a separate frame or a sequence of frames, each separate frame or sequence of frames being associated with a corresponding timestamp, which in turn is associated with the common time base 260.

[0180] As illustrated above, the event detection step S502 may include using the public protocol 240 to store metadata 242 describing the detected event 211, which is associated with the main digital video stream 210, 301 in which the event 211 in question was detected.

[0181] Event detection can be performed in different ways. In some embodiments, event detection step S502 is performed by AI component 132a and includes a first trained neural network or other machine learning component individually analyzing at least one (such as multiple or even all) of the main digital video streams 210, 301 to automatically detect any of the events 211. This may involve AI component 132a classifying the main video streams 210, 301 data into a set of predefined events in a managed classification, and / or into a dynamically determined set of events in an unmanaged classification.

[0182] In some embodiments, the detected event 211 is a change in a presentation slide in the main video stream 210, 301 in question or a presentation included in the main video stream 210, 301 in question.

[0183] For example, if a presenter decides to change the slides in a presentation he / she is currently giving to the audience, this means that the content that is interesting to a given viewer can be changed. The newly shown slide could simply be a high-level image, which is best viewed only briefly in a so-called "butterfly" mode (e.g., displayed side-by-side with the presenter's video in the output video stream 230). Alternatively, the slide could contain a lot of detail, small-print text, and so on. In the latter case, the slide should instead be presented full-screen and for a slightly longer period than usual. Butterfly mode might be less suitable because, in this case, the slide could be more interesting to the viewer of the presentation than the presenter's face.

[0184] In practice, the event detection step S502 may include at least one of the following: First, event 211 can be detected based on image analysis of the difference between the first image of the detected slide and the subsequent second image of the detected slide. The fact that the main video streams 220 and 301 have the property of showing the slide can be automatically determined using conventional digital image processing, such as using motion detection combined with OCR (Optical Character Recognition).

[0185] This can involve using automated computer image processing techniques to examine whether the detected slide has changed significantly enough to actually classify it as a slide change. This can be done by examining the differences in RGB color values ​​between the current slide and the previous slide. For example, one can assess how much the RGB values ​​have changed overall over the screen area covered by the slide in question, and whether groups of pixels belonging to the same set and changing collaboratively can be found. In this way, relevant slide changes can be detected while irrelevant changes, such as the movement of the displayed computer mouse on the screen, are filtered out. This approach also allows for full configurability—for example, sometimes it is desirable to be able to capture computer mouse movements, such as when a presenter wants to use the computer mouse to point at different objects to present certain content in detail.

[0186] Secondly, event 211 can be detected based on image analysis of the information complexity of the second image itself to more specifically determine the event type.

[0187] This could involve, for example, assessing the total amount of text information on a slide under discussion and the associated font size. This can be accomplished using conventional OCR methods, such as deep learning-based character recognition techniques.

[0188] In another example, the detected event 211 is the loss of communication connection between participant client 121 and digital video communication service 110. Then, detection step S502 may include detecting that the participant client 121 has lost its communication connection based on image analysis of a series of subsequent video frames 213 of the main digital video stream 210 corresponding to the participant client 121 in question.

[0189] Because participant client 121 can be associated with different physical locations and different internet connections, it is possible for someone to lose connection with video communication service 110 or central server 130. In this case, it is desirable to avoid displaying a black screen or blank screen in the generated output video stream 230.

[0190] Conversely, this connection loss can be detected as an event by event detection function 132, such as by applying a two-class classification algorithm, where the two categories used are connected / disconnected (no data). In this case, it should be understood that "no data" is different from a presenter intentionally sending a black screen. Since a brief black screen (such as only 1 or 2 frames) may not be significant in the final generated stream 230, the two-class classification algorithm can be applied over time to create a time series. A threshold specifying the minimum length of the connection interruption can then be used to determine whether a connection loss has occurred.

[0191] In another example, the event is the detection of the presence or movement of a participating human user in one or more images of the main digital video streams 210, 301. In yet another example, the event is the detection of movement (such as rotation, zoom, translation, etc.) of the camera used to generate the main digital video streams 210, 301, which may include information about a general motion component and / or a noise motion component. The noise motion component may be, for example, due to manual camera movement. The detection of such presence / movement of a human user and / or the detection of camera movement can be achieved using conventional digital image processing techniques, such as those exemplified above.

[0192] As will be explained below, the events detected in these example types can be used by the pattern detection function 134 to take appropriate and desired actions.

[0193] As described above, each individual main video stream 210, 301 can be associated with a common time base 260 or with each other in the time domain, so that the synchronization function 133 can synchronize them with each other in time.

[0194] In some embodiments, the public time base 260 is based on or includes a public audio signal 111 (see...). Figures 1 to 3The public audio signal 111 is public to the shared digital video communication service 110 involving at least two participant clients 121 with remote connections as described above, each participant client 121 providing a corresponding one of the main digital video streams 210.

[0195] Microsoft discussed above ® Teams ® In the example, a common audio signal is generated, and the central server 130 can capture this common audio signal via the automatic participant client 140 and / or via API 112. In this and other examples, this common audio signal can be used as a heartbeat signal to time-synchronize each of the individual main video streams 220 by binding them to a specific point in time based on this heartbeat signal. This common audio signal can be provided as a separate signal (relative to each of the other main video streams 210), whereby the other main video streams 210 can each be time-correlated with the common audio signal individually, based on the audio contained in the other main video streams 210 in question, or even based on the image information contained therein (such as using lip-syncing techniques based on automatic image processing).

[0196] In other words, to handle any variable and / or different latency associated with individual main video streams 210, and to achieve time synchronization of the combined video output streams 230, this common audio signal can be used as the heartbeat for all main video streams 210 in the central server 130 (but perhaps not for external main video streams 301). In other words, all other signals can be mapped to this common audio time heartbeat to ensure everything is synchronized in time.

[0197] In various examples, time synchronization is achieved using a time synchronization element 231 introduced into the output digital video stream 230, and this time synchronization element 231 is detected by a corresponding local time synchronization software function 125 provided as part of one or more individual participant clients 121, the local software function 125 being configured to detect the arrival time of the time synchronization element 231 in the output video stream 230. It should be understood that in this embodiment, the output video stream 230 is fed back to the video communication service 110, or otherwise made available to each participant client 121 and the local software function 125 in question.

[0198] For example, time synchronization element 231 may be a visual marker, such as pixels that change color in a predetermined sequence or manner, which are placed in or updated in the output video 230 at regular time intervals; a visual clock that is updated and displayed in the output video 230; or an audio signal (which may be designed to be inaudible to participant 122, for example by having a sufficiently low amplitude and / or a sufficiently high frequency) added to the audio forming part of the output video stream 230. Local software function 125 is configured to automatically detect the corresponding arrival time of each time synchronization element 231 using appropriate image and / or audio processing.

[0199] The public time base 260 can then be determined at least in part based on the detected arrival time. For example, each local software function 125 can transmit corresponding information representing the detected arrival time to the central server 130.

[0200] Such communication can occur via a direct communication link between the participant client 121 and the central server 130. However, this communication can also occur via the main video stream 210 associated with the participant client 121. For example, the participant client 121 may introduce visual or audible codes, such as the type discussed above, into the main video stream 210 generated by the participant client 121, for the central server 130 to automatically detect and use in determining the common time reference 260.

[0201] In some additional examples, each participant client 121 may perform image detection in a public video stream available for viewing by all participant clients 121 of the video communication service 110, and relay the results of such image detection to a central server 130 in the manner discussed above, for use at the central server 130 to determine the corresponding offset of each participant client 121 relative to each other over time. In this way, the public time reference 260 can be determined as a set of separate relative offsets. For example, a selected reference pixel of the publicly available video stream may be monitored by multiple or all participant clients 121 (such as by the local software function 125), and the current color of that pixel may be transmitted to the central server 130. The central server 130 may calculate the corresponding time series based on such color values ​​received consecutively from each of the multiple (or all) participant clients 121 and perform cross-correlation to obtain an estimated set of relative time offsets across the different participant clients 121.

[0202] In practice, the output video stream 230, or each output video stream 230, fed into the video communication service 110 may be included as part of a shared screen to two or more, or even each, participant client in the video communication under discussion, and thus can be used to evaluate such time offsets associated with some or all of the participant clients 121. In particular, the output video stream 230 fed into the video communication service 110 may again be available to the central server via the automatic participant client 140 and / or API 112.

[0203] In some embodiments, the common time reference 260 may be determined at least in part based on a difference detected between the audio portion 214 of the first main digital video stream 210, 301 and the image portion 215 of the first main digital video stream 210, 301. This difference may be based, for example, on digital lip-sync video image analysis of a speaking participant 122 as seen in the first main digital video stream 210, 301 in question. This lip-sync analysis is conventional in itself and may, for example, use a trained neural network. This analysis may be performed by the synchronization function 133 for each main video stream 210, 301 relative to available common audio information, and the relative offset between the individual main video streams 210, 301 may be determined based on this information.

[0204] In some embodiments, synchronization step S503 includes intentionally introducing a delay of up to 30 seconds (such as up to 5 seconds, up to 1 second, or up to 0.5 seconds but longer than 0 seconds) (in this context, the terms "delay" and "latency" are intended to mean the same thing) such that the output digital video stream 230 is provided at least with said delay. In any case, the intentionally introduced delay is at least a plurality of video frames, such as at least three frames, or even at least five frames or even 10 frames, such as that number of frames (or individual images) stored after any resampling in collection step S501. As used herein, the term "intentionally" means that the delay is introduced regardless of whether there is any need to introduce such a delay based on synchronization problems or similar problems. In other words, the intentionally introduced delay is introduced in addition to any delay introduced as part of the synchronization of the main video streams 210, 301 to synchronize them with each other in time. The intentionally introduced delay may be predetermined, fixed, or variable relative to a common time reference 260. The delay time can be measured relative to the main video stream with the smallest delay among the main video streams 210 and 301, such that the streams with larger delays among these streams 210 and 301 are associated with a relatively small intentionally added delay due to the time synchronization.

[0205] In some embodiments, a relatively small delay is introduced, such as 0.5 seconds or less. This delay is virtually imperceptible to participants in the video communication service 110 using the output video stream 230. In other embodiments, such as when the output video stream 230 is not used in an interactive context but is published to an external consumer 150 for one-way communication, a larger delay may be introduced.

[0206] The intentionally introduced delay may be sufficient to allow enough time for synchronization function 133 to map the collected individual mainstream 210, 301 video frames to the correct common time base 260 timestamp 261. It may also be sufficient to allow enough time to perform the event detection described above to detect missing mainstream 210, 301 signals, slide changes, resolution changes, etc. Furthermore, the intentionally introduced delay may be sufficient to allow for improved pattern detection function 134, as will be described below.

[0207] It should be understood that introducing the aforementioned delay may involve buffering 250 each of the collected and time-synchronized main video streams 210, 301 before publishing the output video stream 230 using the buffered frame 213 in question. In other words, at least one, several, or even all of the video and / or audio data of the main video streams 210, 301 may subsequently exist in a buffered manner in the central server 130, much like a cache, but not (like a conventional cache buffer) intended to handle varying bandwidth conditions, but for the reasons stated above, and specifically for use by the pattern detection function 134.

[0208] In some embodiments, the pattern detection step S504 includes considering certain information from at least one (such as multiple, at least four, or even all) of the main digital video streams 210, 301, which exists in a frame 213 later than a frame of the time-synchronized main digital video stream 210 that has not yet been used to generate the output digital video stream 230. Therefore, the newly added frame 213 will have a specific latency in the buffer 250 in question before forming part of (or the basis of) the output video stream 230. During this time period, the information in the frame 213 in question will constitute information in the "future" relative to the currently used frame used to generate the current frame of the output video stream 230. Once the timeline of the output video stream 230 reaches the frame 213 in question, it will be used to generate the corresponding frame of the output video stream 230 and can thereafter be discarded.

[0209] In other words, the pattern detection function 134 can use a set of video / audio frames 213 that have not yet been used to generate the output video stream 230, and can use this data to detect the pattern.

[0210] Pattern detection can be performed in different ways. In some embodiments, pattern detection step S504 is performed by AI component 134a and includes a second trained neural network or other machine learning component collaboratively analyzing at least two (such as at least three, such as at least four, or even all) of the main digital video streams 120, 301 to automatically detect the pattern 212.

[0211] In some embodiments, the detected pattern 212 includes speech patterns involving at least two (such as at least three, such as at least four) different speaking participants 122 of the shared video communication service 110, each speaking participant 122 being associated with a corresponding participant client 121, and each of the speaking participants 122 being visually viewed in a corresponding one of the main digital video streams 210, 301.

[0212] The generation step S505 preferably includes determining, tracking, and updating the current generation state of the output video stream 230. For example, this state may specify which participants (if any) of the participants 122 are visible in the output video stream 230 and their positions on the screen; whether the external video stream 300 is visible in the output video stream 230 and its position on the screen; whether any slideshows or shared screens are displayed in full-screen mode or in combination with any live video stream; and so on. Furthermore, this state may specify any cropping or virtual panning / scaling of any of the master digital video streams 210, 301 to be used in any instance. Therefore, the generation function 135 can be viewed as a state machine regarding the generated output video stream 230.

[0213] In order to generate a combined video experience from the output video stream 230 for viewing by, for example, an end consumer 150, it is advantageous that the central server 130 is able to understand what is happening at a deeper level than simply detecting individual events associated with the individual main video streams 210, 301.

[0214] In the first example, the participant client 121, who is giving a presentation, is changing the currently viewed slide. This slide change is detected by the event detection function 132 as described above, and metadata 242 is added to the frame in question, indicating that a slide change has occurred. This can happen multiple times because the participant client 121 is resulting in rapidly and consecutively skipping multiple slides forward, causing a series of "slide change" events to be detected by the event detection function 132, and these events, along with the corresponding metadata 242, are stored in a separate buffer 250 of the main video stream 210 in question. In practice, each such rapidly skipped slide may only be visible for a fraction of a second.

[0215] Pattern detection function 134 examines information in buffer 250 that spans multiple of these detected slide changes and detects patterns corresponding to a single slide change (i.e., the pattern corresponding to the last slide in the forward skip, which remains visible once the fast skip is complete), rather than multiple fast-executed slide changes. In other words, pattern detection function 134 will notice, for example, ten slide changes within a very short time period, and therefore these will be processed as detected patterns representing a single slide change. As a result, the pattern generation function 135, accessible from the pattern detected by pattern detection function 134, can select to display the final slide in full-screen mode for several seconds in the output video stream 230, as it determines that the slide may be important in the state machine. It can also select not to display the intermediate viewing slides in the output stream 230 at all.

[0216] The detection of patterns with multiple rapidly changing slides can be performed by simple rule-based algorithms, but alternatively, a trained neural network designed and trained to detect such patterns in moving images through classification can be used.

[0217] In various examples, such as when video communication is a talk show, panel debate, or similar event, this can be useful: on the one hand, it allows for a rapid shift in visual attention between current speakers; on the other hand, it still provides a relevant viewing experience to the consumer 150 by generating and publishing a smooth and fluid output video stream 230. In this case, the event detection function 132 can continuously analyze each main video stream 210, 301 to always determine whether the person being viewed in that particular main video stream 210, 301 is currently speaking. For example, this can be performed using conventional image processing tools as described above. The pattern detection function 134 can then operate to detect specific overall patterns involving the multiple main video streams 210, 301 that are useful for generating a smooth output video stream 230. For example, the pattern detection function 134 can detect patterns of very frequent switching between current speakers and / or patterns involving multiple concurrent speakers.

[0218] The generation function 135 may then consider this detected pattern when making automatic decisions regarding the generated state, for example, by not automatically switching the visual focus to a speaker who speaks for only half a second and then falls silent again, or by switching to a state where multiple speakers are displayed side-by-side during a period of time in which two speakers speak alternately or simultaneously. This state decision-making process itself can be performed using time-series pattern recognition techniques, or using a trained neural network, but may also be based at least in part on a predetermined set of rules.

[0219] In some embodiments, multiple patterns may exist that are detected in parallel and form the inputs to the state machine of the generation function 135. These multiple patterns may be used by the generation function 135 using different AI components, computer vision detection algorithms, etc. As an example, permanent slide changes may be detected while simultaneously detecting unstable connections of some participant clients 121, while other patterns detect the current main speaker 122. Using all such available pattern data, a classifier neural network may be trained, and / or a set of rules may be developed to analyze the time series of such pattern data. This classification may be at least partially (e.g., fully) supervised to obtain a defined desired state change used in the generation. For example, different such predetermined classifiers may be generated, specifically configured to automatically generate output video streams 230 according to various different generation styles and expectations. Training may be based on known sequences of generated state changes as the desired output and known pattern time series data as training data. In some embodiments, a Bayesian model may be used to generate such a classifier. In a specific example, information can be obtained a priori from an experienced producer, providing input such as, "In stand-up comedy, I never switch directly from speaker A to speaker B, but always show an overview before focusing on the other speaker, unless the other speaker is very dominant and speaking loudly." This generative logic can then be represented as a Bayesian model in the general form of "If X is true | given the fact that Y is true | perform Z." The actual detection (whether someone is speaking loudly, etc.) can be performed using a classifier or threshold-based rules.

[0220] By leveraging large datasets (large datasets of pattern time series data), deep learning methods can be used to develop correct and attractive generation formats for the automatic generation of video streams.

[0221] In some embodiments, generation function 135 may include information about what objects or human participants are shown in the output video stream 230. For example, in some settings, one or more participants 122 may not want or be allowed to be shown to consumers of the output video stream 230. Generation function 135 may then generate the output video stream 230 based on digital image processing techniques used to identify the object or person in question, such that the output video stream 230 does not show such one or more objects or human participants, and automatically trims the main video streams 210, 301 to exclude the object or person before being added as a part of the output video stream 230; or makes a generation decision such that the main video streams 210, 301 currently showing the object or person are not included in the output video stream 230.

[0222] In some embodiments, generation function 130 may be configured to introduce externally provided information, such as in the form of data from the main digital video stream 300 or other externally provided data, in response to patterns detected in one or more of the main digital video streams 210, 301. For example, generation function 130 may be configured to automatically detect a topic of discussion or a predetermined triggering event or pattern (such as a predetermined trigger phrase) via digital processing of images and / or sound included in the main video streams 210, 301. In a specific example, this may include automatically introducing updated text or graphical information from a remote source into the output video stream 230 regarding the topic currently being debated by participant user 122. Generally, the detection of such triggering events or patterns allows generation function 130 to modify its current generation state in any way, depending on the type or characteristics of the detected triggering event or pattern.

[0223] In summary, by combining event detection based on individual main video streams 210 and 301, intentionally introduced delays, pattern detection based on multiple time-synchronized main video streams 210 and 301 and detected events, and a generation process based on detected patterns, it becomes possible to automatically generate the output digital video stream 230 according to a wide range of possible tastes and styles. This result is applicable to a wide range of possible neural networks and / or rule-based analysis techniques used by the event detection function 132, the pattern detection function 134, and the generation function 135.

[0224] As exemplified above, generation step S505 may include: generating one or more output digital video streams 230 based on a set of predetermined and / or dynamically variable parameters relating to the visibility of each of the main digital video streams 210, 301 in the output digital video stream 230; the arrangement of visual and / or audio-visual content; the visual or audio effects used; and / or the output mode of the output digital video stream 230. These parameters may be automatically determined by the state machine of the generation function 135, and / or set by the operator controlling the generation (making it semi-automatic), and / or predetermined based on certain prior configuration expectations (such as the shortest time between a change in the layout of the output video stream 230 or a state change of the type described above).

[0225] In a practical example, the state machine can support a set of predetermined standard layouts applicable to the output video stream 230, such as a full-screen presenter view (showing the current speaker 122 in full screen); a slide view (showing the currently shared presentation slides in full screen); a "butterfly view," which shows the current speaker 122 and the currently shared presentation slides in a side-by-side view; a multi-speaker view, which shows all participants 122 or a subset of selected participants 122 in a side-by-side or matrix layout; and so on. Various available generation formats can be defined by a set of state machine state change rules (as illustrated above) along with a set of available states (such as the aforementioned set of standard layouts). For example, one such generation format could be "group discussion," another could be "presentation," and so on. By selecting a specific generation format via the GUI or other interfaces of the central server 130, the operator of system 100 can quickly select one of these predefined generation formats, and then allow the central server 130 to automatically generate one or more output video streams 230 based on the available information as described above, according to the generation format in question.

[0226] Furthermore, during generation, corresponding in-memory buffers can be created and maintained for each conference participant client 121 or external video source 300 as described above. These buffers can be easily removed, added, and changed in real time. The central server 130 can then be configured to receive information during the generation of the output video stream 230 regarding added / departed participant clients 121 and participants 122 scheduled to give presentations, planned or unexpected presentation pauses / resumptions, expected changes to the currently used generation format, and so on. For example, such information can be fed to the central server 130 via the operator GUI or interface described above.

[0227] As illustrated above, in some embodiments, at least one of the main digital video streams 210, 301 is provided to the digital video communication service 110, and the publishing step S506 may subsequently include providing the one or more output digital video streams 230 to the same communication service 110. For example, the output video stream 230 may be provided to a participant client 121 of the video communication service 110, or provided to the video communication service 110 as a corresponding external video stream via API 112. In this way, the output video stream 230 may be provided to multiple or all participants in the video communication event currently being implemented by the video communication service 110.

[0228] As discussed above, additionally or alternatively, one or more output video streams 230 may be provided to one or more external consumers 150.

[0229] Overall, the generation step S505 can be executed by the central server 130, which provides the output digital video stream 230 as a real-time video stream to one or more concurrent consumers via API 137.

[0230] Figure 8 A method for providing an output digital video stream is shown, which will be described below with reference to what has already been described above. Therefore, in Figure 8 The method shown can apply all the mechanisms and principles described above regarding digital video stream collection, event detection, synchronization, pattern detection, generation, and publishing.

[0231] also, Figure 9 and Figure 10 It is configured to execute Figure 8 A simplified view of the system 100 of the method shown.

[0232] exist Figure 9 The diagram illustrates three distinct central servers 130', 130'', and 130'''. These central servers 130', 130'', and 130''' can be a single integrated central server of the type discussed above, or they can be separate central servers. They may or may not run on the same physical or virtual hardware. In either case, they are configured to communicate with each other.

[0233] In some embodiments, central servers 130' and 130'' may be configured to run on the same physical hardware 402 (by... Figure 9 (shown as a dashed rectangle in the diagram), for example, in the form of a discrete hardware device, such as a conventional computer device itself. In some embodiments, such a discrete hardware device 402 is a computer device 402' located in or physically connected to a conference room (see...). Figure 10 The device is specifically configured to conduct a digital video conference in the room. In other embodiments, the discrete hardware device is a personal computer 402'', 402''', such as a laptop computer, which is used by a single human participant 122, 122'', 122''' of such a digital video conference, who is present in the room under discussion or remotely.

[0234] Each of the central servers 130', 130'', and 130''' includes a corresponding collection function 300', 300'', and 300''', which may be as generally described above. Collection function 300' is configured to collect digital video stream 401 from a digital camera (such as video camera 123 of the type generally described above). This digital camera may be an integrated part of the discrete hardware device 402, or a separate camera connected to the hardware device 402 using a suitable wired or wireless digital communication channel. In any case, the camera is preferably arranged locally relative to the hardware device 402.

[0235] Each of the collection functions 300'' and 300''' can collect the digital video signal corresponding to the digital video stream 401 directly from the digital camera or from the collection function 300'.

[0236] Each of the central servers 130', 130'', and 130''' may also include corresponding generation functions 135', 135'', and 135'''. Each of these generation functions 135', 135'', and 135''' corresponds to generation function 135 described above, and the above description of generation function 135 also applies to generation functions 135', 135'', and 135'''. Depending on the detailed configuration of the central servers 130', 130'', and 130''', there may be more than three generation functions. Various digital communications between generation functions 135', 135'', and 135''' and other entities can be performed via appropriate APIs.

[0237] Furthermore, each of the central servers 130', 130'', and 130''' may include a corresponding publishing function 136', 136'', and 136'''. Each of these publishing functions 136', 136'', and 136''' corresponds to the publishing function 136 described above, and the above description of publishing function 136 also applies to publishing functions 136', 136'', and 136'''. Publishing functions 136', 136'', and 136'''' may be different, or may be arranged together as a single logical function with multiple functions, and depending on the detailed configuration of the central servers 130', 130'', and 130'''', there may be more than three publishing functions. In some cases, publishing functions 136', 136'', and 136'''' may be different functional aspects of the same publishing function 136, as the case may be.

[0238] While publish functions 136'' and 136''' are optional and can be configured to output a video stream different from the video stream output by publish function 136' (potentially more complex, associated with corresponding time delays), publish function 136' is configured to output one or more output digital video streams of the type generally described herein. Further generally, each of generate functions 135'' and 135''' can be configured to process the corresponding incoming video stream to generate the generation control parameters used by generate function 135', so that generate function 135', in turn, generates the output video stream as described herein.

[0239] Figure 9 Three external consumers 150', 150'', and 150''' are also shown, each corresponding to the external consumer 150 described above. It should be understood that there may be fewer than three or more such external consumers 150', 150'', and 150'''. For example, two or more of the publishing functions 136', 136'', and 136''' may output the same or different generated video streams to the same external consumer 150', 150'', and 150''', and each of the publishing functions 136', 136'', and 136''' may output the same or different generated video streams to more than one of the external consumers 150', 150'', and 150''''. It should also be noted that at least publishing function 136' may publish one or more generated video streams back to collecting function 300'. In addition, each of the publishing functions 136', 136'', and 136''' can be configured to publish one or more of the corresponding generated video streams to the participant client 121 of the general type discussed above.

[0240] It should be understood that consumer 150' may be a participant client 121 that also includes a central server 130', for example by setting up the central server 130' (and possibly also the central server 130') on the laptop computers 402'', 402''', and as part of the video communication service in which the corresponding human users 122'', 122''' participate, providing an enhanced real-time output video stream to the corresponding human users 122'', 122''' on the screen of the laptop computers 402'', 402'''.

[0241] also, Figure 9Three external information sources 300', 300'', and 300''' are shown, each corresponding to the external information source 300 described above and providing information to a corresponding one of the collection functions 300', 300'', and 300'''. It should be understood that there may be fewer than three or more such external information sources 300', 300'', and 300'''. For example, one such external information source 300', 300'', and 300'''' may feed to more than one collection function 300', 300'', and 300''''; and each collection function 300', 300''', and 300'''' may feed from more than one external information source 300', 300'', and 300''''.

[0242] For the sake of simplicity, Figure 9 Video communication service 110 is not shown, but it should be understood that the general types of video communication services discussed above can be used in conjunction with central servers 130', 130'', 130''', for example, to provide shared video communication services to participant clients 121 in the manner discussed above. In some embodiments, central server 130''' constitutes, includes, or is included in video communication service 110.

[0243] Figure 10 Three different exemplary hardware devices are shown: a conference room hardware device 402', which includes a digital camera 401' configured to capture images showing one or more human conference participants 122'', 122''', 122'''' in the meeting location, room, or venue; and two laptop computers 402'', 402''', each including a corresponding digital webcam 401'', 401''', configured to capture images showing one of the human conference participants 122'', 122''' using the laptop computers 402'', 402''' in question. It should be understood that... Figure 10 One of many different configurations is shown to illustrate the principles of the invention, and other types of configurations are also possible. For example, only some of participants 122'', 122''', and 122'''' may be visible to camera 401'; additional participant users ( Figure 10 (Not shown in the text) can remotely participate in video communication services; can use external information sources; etc., as illustrated herein. As used herein, the term "remotely" means not a "local" arrangement as defined above. Two entities arranged "remotely" relative to each other are preferably configured to communicate via the Open Internet (WAN).

[0244] One of the exemplary participants, user 122'''', is not visible to cameras 410'' and 401''', but is only visible from camera 401'. Participant user 122'' is visible to both cameras 401'' and 401''. Participant user 122''' is visible to both cameras 401' and 401'''.

[0245] Each of the hardware devices 402', 402'', and 402''' can correspond to Figure 9 The device 402 is shown in more detail below, and each of the hardware devices 402', 402'', and 402''' can be configured to communicate with the video communication service 110 via the Internet 10 or another digital communication network. Note that the video communication service 110 can then use the devices 402', 402'', and 402''' as participant users 121 of the general type described above to provide a shared video communication service. In some embodiments, the video communication service 110 (such as a central server 130''') is remote relative to the central servers 130', 130''.

[0246] return Figure 8 In the first step S800, the method begins.

[0247] In the subsequent step S801, the first participant user 122 is authenticated. There are many different ways to perform this authentication. The authentication may include at least one or more of the following: • User 122 logs into system 100.

[0248] • User 122 logs into a system outside of system 100, such as a system operated by an online service provider (such as a social network or search engine operator).

[0249] • User 122 uses an existing login with a first external party to log in relative to system 100 or a system provided by a second external party, such as using a login token provided by the first external party.

[0250] • User 122 uses a username / password combination to authenticate relative to system 100 or a system provided by an external party.

[0251] • Users receive a one-time password and use it to authenticate with System 100 or a system provided by an external party.

[0252] • Biometric measurements of user 122 are performed relative to fingerprint, palm, iris, 2D or 3D facial contour, voice contour, or the like for authentication relative to system 100 or a system provided by an external party.

[0253] In general, certification can include at least one, at least two, or even all three of the following common certification factor types: “what you have,” “what you know,” and “what you are inherently.”

[0254] The thing “owned” by user 122 can be a mobile device, such as a smartphone or laptop computer, which may or may not be the client device 121 used by user 122 to access video communication service 110. Thus, the mobile device can be used to receive a one-time password that will be entered into a graphical user interface or via a microphone, or the mobile device can be hardware-bound to a piece of information used in authentication, such as via a secure circuit on the mobile device. A smart card, USB drive, or other separate device that supports communication can also be used as something owned by user 122. Authentication of user 122 “owning” the device can be achieved by the device receiving a piece of information (such as a PIN), and user 122 entering that information into system 100 or an external system. In other embodiments, the device “owned” by user 122 can be configured to communicate directly with client 121, such as electronically, digitally, using wires and / or wirelessly, or even using an audio channel outside the frequency of human hearing. This direct communication may be active only when associated with authentication, but may also be active thereafter and can be used for continuous monitoring discussed below.

[0255] What User 122 "knows" can be a PIN, password, or passphrase. It can also be an answer to a question that might be difficult for someone other than User 122 to answer.

[0256] What User 122 "possesses" can be User 122's biometrics, such as fingerprints, facial recognition patterns, iris scan patterns, DNA signatures, and so on.

[0257] It should be understood that there are many different known methods for authenticating users 122. In this document, the terms "authentication method," "authentication mode," "authentication type," etc., are used as synonyms.

[0258] Authentication can typically be relative to system 100 or relative to a different system provided by an external party. In some embodiments, it is not important to which party user 122 is authenticated with; rather, in this case, an interesting aspect can be the reliability of the authentication itself as a reliable proof of user 122's actual identity, and the fact that the authentication can be traced and verified.

[0259] In other words, authentication relative to System 100 or any externally provided system can generate an authentication token (hereinafter referred to as the first authentication token). The token itself can have any suitable format, such as a hash value or a cryptographic signature. This token can then be configured to be used to retroactively verify authentication, allowing a party accessing the first authentication token to securely verify, using available tools, that user 122 has indeed been authenticated, or indeed has an active and valid authentication. In a specific example, a WebAuthn device, such as Yubikey, is used for authentication. ® It operates through a compact personal key using a standard protocol, employing asymmetric cryptography to generate unique digital tokens that are immediately verified by an authentication server. These digital tokens can also be stored for later verification. In another specific example, a third-party authenticator (such as Google...) ® Perform authentication and provide an authentication token, such as an access token according to the OAuth standard, in response.

[0260] As used in this article, "authenticated" or "authenticated" means that a party (such as the operator of system 100) has verified the identity of someone, such as the identity of user 122.

[0261] In a subsequent or concurrent collection step S802 related to step S801, one or more real-time first primary digital video streams 210, 301 are continuously collected. Figure 9 and Figure 10 In the case shown, video streams 210 and 301 are stream 401, which is continuously collected by collection function 131' from any one of cameras 401', 401'', and 401''' (and / or continuously collected by 131'' and 131''' in the case of collecting external information source 300).

[0262] Specifically, a first main video stream can be captured to visually represent the first user 122. A second main video stream can be captured in step S804 to visually represent the second user. The first user and the second user can be... Figure 10 The two users shown are 122', 122'', and 122'''. Even though each user 122', 122'', and 122''' may be visible in one or more main video streams, the first main video stream and the second main video stream are different video streams, preferably captured using different physical cameras. Generally speaking, when a stream includes information representing one or more images of a depicted object, user, etc., the stream can be said to visually represent the object, user, etc.

[0263] The first primary digital video stream is a "real-time" stream, meaning that it is delivered from the capturing camera to the collection function 131 in question without any delay and without any time-consuming image processing before reaching the collection function 131. For example, any data processing and / or communication between the capture of an image frame from the camera sensor and the storage of the corresponding digital video stream in the collection function 131 may take less than 0.1 seconds, such as less than 0.05 seconds.

[0264] As described above, the first primary digital video streams 210 and 301 can be continuously captured by cameras 410', 401'', and 401''' locally arranged relative to the participant client 121 consuming the output video stream, such as by devices 402', 402', 402'', and 402''' themselves.

[0265] In addition, the first main digital video streams 210, 301 can be continuously captured by cameras 401', 401'', 401''', which are configured to capture images of participant users 122'', 122''' showing the device 402'', 402''' (participant client 121) in question.

[0266] The first primary video stream is captured to include a first primary authentication video portion. As used herein, a "video portion" refers to a section of a video stream that includes image and / or audio data and spans at least a certain time period, such as including one or more consecutive video frames and / or corresponding audio content of the video stream. Such a video portion may be limited to only a very short time period, such as a single video frame. However, in some embodiments, the video portion is at least 1 second and / or at least 10 video frames. Preferably, the video portion is configured to allow for the retrospective and explicit detection of information, such as information about the first authentication token stored as part of the video stream.

[0267] In other words, the first primary authentication video portion is captured to meet at least one of two different criteria: the first criterion is that it visually demonstrates the authentication steps. The second criterion is that it includes information that visually and / or audibly represents the first authentication token.

[0268] The capture of the first primary authentication video portion to visually demonstrate the authentication steps means that the video portion contains sufficient visual information to allow a third party to retrospectively and explicitly infer, based on the video portion, that user 122's authentication did indeed occur, what type of authentication was used, and the result of the authentication (user 122 was successfully authenticated). For example, the video portion could include a camera showing user 122 holding up their smartphone with its screen visible, user 122 receiving a one-time password, entering the one-time password into a graphical user interface on the smartphone or a different device's screen, and receiving authentication confirmation on the graphical user interface, where all steps of the authentication process, including any information used, can be clearly verified through visual inspection of the first primary authentication video portion. This could include the smartphone screen being clearly shown in the video portion, achieved by user 122 holding the screen in front of device 121's camera during authentication to demonstrate the necessary steps performed during authentication. Instead of a one-time password, a QR code or barcode could be received by user 122's smartphone, and the smartphone screen displaying the QR code or barcode could be shown in the video portion. In various examples, authentication is performed using a graphical user interface presented on the display screen of client device 121, and the display screen is dumped, recorded, or copied to form part of the video portion. In other examples, the first primary authentication video portion itself may form part of the authentication process and thus demonstrate the authentication of user 122. For example, a one-time password may be provided to user 122 from an authentication entity, which user 122 verbally and / or visually provides to the camera of device 121, and device 121 performs verification of the one-time password or forwards it to an authentication entity capable of verifying the one-time password. Any party viewing the first primary authentication video portion containing the one-time password should then be able to retroactively verify the authenticity of the one-time password.

[0269] The first primary authentication video portion includes information that visually and / or audibly represents the token. This could mean that the token can be shown to the camera of device 121, for example, displayed on the screen of user 122's smartphone, or displayed on a piece of paper placed within the field of view of device 121's camera, and the visual appearance of this information forms part of the first primary authentication video portion; or user 122 speaks the token, such as in the form of a verbally delivered code or password phrase, and the sound of this speech is captured and forms part of the video portion. Preferably, the token is also configured to be explicitly inferred retroactively by a third party. In an alternative embodiment, the token information can be embedded in the first primary authentication video portion by modifying the first primary authentication video portion (e.g., digital modification), for example by digitally adding the token information in a visually readable manner to the video portion in the form of a QR code, barcode, sequence of alphanumeric characters, etc. This addition can take the form of a readable overlay, watermark, or any other suitable method, and can be performed by participant client 121 or any other part of system 100 that can access and modify the first primary video stream before it is used to generate the generated video stream. The information shown can be the token itself, or other information that can be used to definitively locate the token, such as through association, calculation, or similar means.

[0270] In general, authentication can result in the first user 122 audibly and / or visually providing authentication information, such as a one-time password already provided to the first user 122, such that the authentication information (e.g., the one-time password itself as discussed above, or the authentication result, the obtained token, or some other type of authentication proof) becomes part of the first main authentication video portion.

[0271] Therefore, it should be understood that information in the form of explicitly representing a token or explicitly indicating that a positive authentication step has occurred relative to user 122 is made to be present in the first main authentication video portion, which is achieved by physically capturing the information during the capture of the main video stream by a camera and / or microphone used to capture the first main video stream including the video portion; and / or after capture and before the first main video stream is used to generate the generated video stream, adding the information as a modification or overlay to the already recorded main video stream.

[0272] In the subsequent step S803, the first main video stream is used as a direct input or a derived input to calculate the output of the first one-way function. In step S805, the second main video stream can also be used as a direct input or a derived input to calculate the output of the corresponding second one-way function.

[0273] The term "one-way function" is well-known in the art and refers to a function whose input values ​​cannot be determined solely from its output value in practice, and which is essentially one-to-one, meaning that in the practical applications described herein, two different input values ​​will always result in two different output values. Examples include many conventional hash functions, such as SHA hash functions (e.g., SHA-1, SHA-2, and SHA-3), and MD5.

[0274] In various embodiments, such one-way functions are collision-free. Furthermore, they can be as follows: https: / / en.wikipedia.org / wiki / Cryptographic_hash_function A cryptographic hash function in the sense described in [the text].

[0275] Each one-way function described in this article can be the same or different one-way functions (i.e., the same or different one-way functions can be used in different contexts and situations).

[0276] As used in this article, using a piece of information (such as a video stream) "as a direct or derived input" to compute the output of a one-way function means that the result (output) of the one-way function is explicitly dependent on that piece of information, either directly or indirectly. Sometimes the phrase "compute the function" is used, and sometimes "compute the output of the function" is used, but it should be understood that these represent the same meaning.

[0277] For example, some or all of the pixel values ​​from one or more frames of a video stream can be directly used as input to a one-way function, or (more realistically) the hash value of such data can be used as input to a one-way function. However, in more complex embodiments, this information can be used as input to another one-way function (which may be the same or different one-way function), the output of which forms the input to a one-way function to generate the final result. In this way, "chains" of one-way functions can be formed, which can be very long in some cases. One property of such computational chains is that the following information is required to correctly compute the final output of the downstream one-way function: • Data fed into the upstream one-way function; • All one-way functions used in the chain; and • Any additional input information inserted along this chain into one or more one-way functions.

[0278] In other words, "derived" inputs can be derived along a chain of functions, such as a chain of unidirectional functions, which includes two or more functions computed sequentially and potentially over time. Such chains of unidirectional functions can both branch and merge.

[0279] Of course, "derived" inputs can also be derived using non-one-way functions, such as regular integral functions, summation functions, or difference functions.

[0280] Accordingly, the output of a one-way function can be "direct or derived," meaning that the output can be the output value of the one-way function itself, or a value determined in a repeatable and explicit manner based on such a direct output value, for example, via a chain of additional one-way functions and / or other computations.

[0281] Furthermore, the output of the first one-way function can be computed using the first primary authentication video portion as a direct input or a derived input. Specifically, the output of the first one-way function can be computed in such a way that the output of the first one-way function cannot be computed without access to any sub-part of the first primary authentication video portion, which is sufficient to verify the positive authentication of the first user 122 as described above.

[0282] In the subsequent step S806, a generated video stream is automatically generated based on the first and second main video streams. This automatic generation is based on automatic generation decisions of the types explained and exemplified above, such as automatic detection based on events and / or patterns, the use of intentionally introduced delays, and / or based on predetermined and / or dynamic generation parameters. Specifically, this automatic generation may be performed based on at least one of the following: defined generation parameters; automatic image processing of the first and / or second main video streams; and automatic audio processing of the first and / or second main video streams.

[0283] In the subsequent step S807, a generated video stream is provided to a receiving user, who may be a first user, a second user, a different user who is a party to the same video communication service 110, and / or an external user 150.

[0284] Any or all of the main video stream and generated video streams may be stored for future reference, preferably in a persistent and permanent manner, such as on a hard disk drive, flash drive, non-volatile memory storage device, etc. This storage can be performed in a manner that allows for complete retrospective copying of each video stream; in other words, without using any lossy compression algorithms or similar algorithms. This will allow for retrospective verification of the information integrity of the stored data using any relevant output of one or more one-way functions (such as a first one-way function), and in particular, verification of the authentication of the first user 122.

[0285] In some embodiments, a first user and a second user participate in an interactive video session using video communication service 110. A generated video stream showing the first user is then provided to the second user, and a corresponding generated video stream showing the second user is shown to the first user. The two generated video streams may be the same generated video stream showing both the first and second users; or they may be different. In the case of different generated video streams, they may each be generated independently in the general manner discussed above, possibly including a second master authentication video portion in the second master video stream, which includes proof that the second user has been affirmatively authenticated in a manner corresponding to that described above with respect to the first master authentication video portion. The generation of each of the two generated video streams may be performed in a separate central server 130; in a device 121 operated by the first user; and / or in a device 121 operated by the second user.

[0286] Therefore, the generation of a video stream being “provided” to a receiving user can mean that a first primary video stream is generated by a generating entity to generate the video stream, which is then sent to (provided to) the receiving user’s device; or that the first primary video stream is first sent to the receiving user’s device, which generates the video stream, which is then shown to (provided to) the receiving user.

[0287] In general, providing the corresponding generated video stream to the first user and / or the second user can be done in the context of an interactive video communication service with the first user and the second user as participants, and form part of such an interactive video communication service, as discussed in general above.

[0288] In step S808, which is performed after the calculation of the first one-way function and the second one-way function, a piece of information representing the direct or derived output of the first one-way function is embedded as a visual and / or audible representation of that piece of information into the generated video stream; stored (preferably stored persistently and permanently in the sense discussed above); and / or publicly released. A corresponding piece of information representing the direct or derived output of the second one-way function can be processed in a similar manner. It should be understood that the same piece of information can represent the direct or derived output of both the first and second one-way functions.

[0289] The fact that this information is "embedded" into the generated video stream means that the generated video stream has been modified compared to a generated video stream without this information. This modification can occur as a result of generation and / or post-generation and can be digital. For example, the embedded information can be in the form of a QR code or barcode, or digitally added to a portion of the video in a visually readable manner as a sequence of alphanumeric characters, encoding the information. This addition can take the form of a readable overlay, a watermark, or any other suitable method. The information shown can be the information itself, or it can be other information used to explicitly find a direct or derived output representing the first one-way function in question, such as via association, computation, or similar methods. The embedding can also be audible, such that the audio portion of the video stream is modified to allow the embedded information to be retrospectively extracted from the generated video stream. Examples include an audio watermark representing the embedded information or computer-generated speech or signal. The embedding can be visible / audible or hidden. The embedding can also be in the metadata of the video stream or in the metadata associated with the video stream.

[0290] The same applies when information is embedded into the main video stream, but this embedding is always performed before the main video stream is used to generate the generated video stream, so that the main video stream with the embedded information is used as the "main video stream" in the sense discussed above.

[0291] In general, a message representing an authentication token may be embedded in the visual and / or audible portion of the first main video stream for a limited period of time, such as up to one minute, up to ten seconds, or up to one second, from the timeline of the main video stream.

[0292] The term "stored" means that a piece of information is stored on a suitable storage medium, such as a database, for future access. This storage can be part of the video stream, along with the video stream, or separate from it. The storage is preferably persistent and permanent, such as on a conventional hard drive, flash drive, or non-volatile memory. The storage can allow for complete traceability of each piece of information; in other words, it does not use any lossy compression algorithms or similar algorithms.

[0293] A message being "publicly published" means that it is published in such a way that it is readily available to a sufficiently wide audience and has sufficient persistence over time that a third party can likely verify the time of publication (e.g., at a granularity of at most one day or even at most one hour) and that the content of the message or document is completely consistent with what it was published on the date and time, even after a period of time has passed (e.g., several years). For example, the message could be published as a public social media post or on a publicly verifiable blockchain. Public publication can take the form of the message itself and its publication timestamp being accessible online. In general, public publication occurs on and through publicly available channels.

[0294] In the subsequent step S809, the method ends. However, as... Figure 8 As shown, this process can be repeated iteratively over time; capture any main video stream, generate one or more generated video streams, and process the information.

[0295] As described above, the first main video stream captures an image of the first user 122, and in particular, it can capture an image showing the first user 122 being authenticated. Different cameras can also capture the main video stream including an image of the first user 122. For example, if the first main video stream is captured by camera 401'' and shows the first user 122'' being authenticated, another camera 401' also shows the first user 122'' being authenticated.

[0296] In some embodiments, the authentication in step S801 may further include capturing (e.g., captured by camera 401') an image or video showing both the first user 122'' and the camera 401'' used to capture the first main video stream. In other words, camera 401'' is within the field of view of camera 401'. This additional captured image or video (captured by camera 401') can then be processed and used in a manner corresponding to the first main authentication video portion; that is, a one-way function with the captured image or video as a direct or derived input can be computed, and the output of the one-way function can be embedded in the generated video stream, stored, and / or publicly released. In a manner corresponding to what has been discussed above, using chaining and similar techniques, the captured image or video can be combined with the first main authentication video portion into the same one-way function (such as the joint one-way function described below), such that a single one-way function output represents both the captured image or video and the first main authentication video portion. Therefore, in this case, there are two different but simultaneously captured main video streams covering the authentication of user 122 (from cameras 401' and 401'' in the current example): a first main video stream captured by camera 402'' and including the first main authentication video portion, and a main video stream captured by camera 402' and showing both user 122 and camera 402'', where the first main authentication video portion is captured.

[0297] In some embodiments, a portion of the first main video stream may be captured before a certain point in time along the timeline of the first main video stream, at which point the generation of the generated video stream begins. In other embodiments, the first main video stream may be captured during a period when it is not currently used as part of the input to the generated video stream (and / or will never actually be used). This is, for example, in a so-called “lobby” or “waiting room” feature, where users can check in or log into a meeting before being allowed to enter, check audio and video, etc. This can also apply to “green room” or “breakout room” features and similar features.

[0298] In this scenario, the first primary authentication video segment can be a portion of the first primary video stream during a specific time period on the timeline of the primary video stream, such as a period preceding the automatically generated portion of the primary video stream used to generate the video stream. In other words, the authentication steps and / or information representing the token can be shown in, or form part of, a portion of the first primary video stream not used to generate the video stream, and therefore not shown to any other participating user of the video communication service 110 besides the user 122 who may be being authenticated. However, note that the first one-way function is still calculated based on the information in the first primary authentication video segment, such that authentication can be verifiable at a later point in time.

[0299] In some embodiments, metadata (such as receiving, identifying, creating, inferring, or similar) is provided for one or more of the main video stream and / or generated video streams described herein, and this metadata is stored, for example, in step S808. This metadata can be of the general types described above. This metadata can be associated with any of the main video stream and / or generated video streams described herein, and particularly with the first main video stream and / or any generated video stream. This metadata can take different forms and can be provided at any point in time, such as being provided in association with an event; being provided intermittently; being provided continuously; and / or being provided upon request by a part of system 100.

[0300] In some cases, the first authentication token or its derived information is or forms part of this metadata, which helps to clearly infer the first authentication token.

[0301] In some cases, the current authentication status of the first user 122 (such as information about recently performed authentications, current continuity scores, and / or trustworthiness metrics) is part of or forms part of this metadata.

[0302] The method may include the step S810 of performing continuous monitoring. Continuous monitoring may be based at least on automatic image processing and / or audio processing of the first main video stream and / or analysis of sensor data related to the physical camera capturing the first main video stream, to determine the extent to which the first user has been continuously visible and / or audibly (i.e., audibly) present in the first main video stream since the authentication step. Continuous monitoring may involve monitoring the presence of the first user in the video stream, for example, in terms of the visibility of the user and / or the audibility of audio generated by and / or associated with the first user. Continuous monitoring may alternatively or additionally be based on detecting ongoing or active communication between a device held by the user and the participant client 121 of user 122, which has been used as an authentication factor for what the user "owns" as described above.

[0303] This can include one or more of the following: • Analyze consecutive frames of the main video stream to determine the sustained and continuous presence of the first user 122 in the image frames, such as using shape-following image processing as the first user 122 moves around, and in particular, determine whether the first user 122 disappears from the first main video stream. Continuous presence can then affect the continuity score to improve it.

[0304] • Analyze the visual appearance of the first user 122, including facial appearance, facial recognition, hairstyle, clothing, posture, size, body shape, etc., to determine that the appearance will not change unexpectedly over time. There is no such change that could subsequently affect the continuity score and improve it.

[0305] • Analyze the matching between the visual appearance and / or physical movement of the first user 122 and the physical model to detect any unexpected movement of the first user 122. A good match can then affect the continuity score to improve it.

[0306] • Analyzing the speech of the first user 122, possibly combined with detecting the visual lip movements of the first user 122 to perform lip-sync analysis to determine who is speaking, this analysis detects whether the first user 122's speech has changed in an unexpected way. The absence of such a change can subsequently affect the continuity score to improve it.

[0307] • Monitoring client 121 monitors communication between the “owning” factor device to detect communication activity, frequency, and / or status, and to detect whether one or more defined thresholds for communication have been exceeded. This communication may use any suitable wired or wireless communication protocol, such as short-range wireless protocols like NFC, Bluetooth, or any other wireless protocol with a normal range of less than 20 meters, but preferably, once the two devices are paired, the communication is performed automatically by client 121 and factor device without any action from user 122. Exceeding such thresholds can subsequently affect the continuity score, causing it to deteriorate.

[0308] • Use appropriate digital image processing algorithms to measure and analyze general properties of the first video stream, including one or more of lighting conditions, background, orientation, zoom, pan, and focus. Large and / or frequent changes detected in one or more of these general properties can subsequently affect the continuity score and degrade it.

[0309] • Use a suitable digital image processing algorithm to detect events or patterns shown in the first video stream, which can be of the general types discussed above. Detecting such a predetermined event or pattern, or an unexpected event or pattern according to a certain defined metric, can subsequently affect the continuity score and worsen it. For example, an event could be an object (such as a pet) temporarily obscuring the image of authenticated user 122.

[0310] • Measure and analyze camera movement, for example by measuring the camera's acceleration and / or rotation (e.g., by a MEMS sensor, accelerometer, gyroscope, or similar device physically attached to the camera); by measuring translation and / or scaling occurring in the image depicted by the camera (e.g., using corresponding digital image algorithms). Larger and / or more frequent detected camera movements can subsequently affect the continuity score, thus worsening it.

[0311] The continuous monitoring may generate a stream or flow of metadata points in step S811, reflecting the current state of the continuous monitoring and its results. This metadata may then be stored for later reference, and / or used as input to various joint one-way functions discussed below.

[0312] In general, continuous monitoring can be performed on user 122's client 121 (such as device 402''), but it can also be performed on a central server 130 (such as in video communication service 110). This is especially true for continuous monitoring performed based on the content of the first main video stream. It should be noted that client 121, 402'' is a piece of hardware that is separate from such a central server 130 and is configured to continuously capture the first main video stream using camera 401''.

[0313] For example, a first primary video stream can be provided to the central server 130 from clients 121, 402'' in raw or processed format. Continuous monitoring can then be performed in the central server 130 based on the first primary video stream provided by clients 121, 402''. In this and other cases, continuous monitoring can be performed by clients 121, 402'' themselves, or continuous monitoring can be performed using a combination of computational work from the central server 130 and clients 121, 402'', such as clients 121, 402'' performing initial or coarse monitoring and automatically delegating defined higher-level or subsequent tasks to the central server 130.

[0314] Interruptions in the communication connection between the central server 130 and clients 121 and 402'' can be detected. Upon detection of such an interruption, ongoing continuous monitoring, or a portion thereof, performed by the central server 130 can then be automatically switched to be performed by clients 121 and 402''. This means that, for part or all of the duration of the interruption, clients 121 and 402'' can alternatively perform the (partial) continuous monitoring in question.

[0315] Upon detection of a subsequent re-establishment of the communication connection, (partial) continuous monitoring can then be switched back from clients 121 and 402'' to the central server 130. Information regarding the monitoring results performed by clients 121 and 402'' can be transmitted to the central server 130 to ensure continuity.

[0316] The switching between the central server 130 and clients 121 and 402 can also be based on parameters other than the availability of the communication channel, such as load balancing considerations or security settings for the video communication service 110.

[0317] In some embodiments, the continuous monitoring is preferentially performed by the central server 130 of the system 100, and the task is partially or fully delegated to the clients 121, 402'' only when defined parameters require or expect this (such as based on the detection of the availability of the communication connection).

[0318] In cases where continuous monitoring is performed at least in part by the central server 130, in some embodiments, continuous monitoring may also be paused by the central server 130 upon detection of an interruption in the communication connection. Then, upon detection of a re-established communication connection, the central server 130 may catch up on receiving the main video stream from clients 121, 402'', and then retrospectively perform continuous monitoring using the caught-up portion of the first main video stream, which comes from the point in time following the interruption along the timeline of the first main video stream. Thereafter, the central server 130 may subsequently continue performing continuous monitoring on the main video stream provided to the central server using the re-established communication connection.

[0319] At least during periods when the communication connection is unavailable, clients 121, 402'' may perform "interleaving" of a first primary video stream of the type described below, including embedding the direct or derived output of a one-way function into a second video frame of the first primary video stream, which in turn uses the first video frame of the first primary video stream as a direct or indirect input, and then iterating by using the second video frame including the embedded one-way function as a direct or derived input of another one-way function whose direct or derived output is embedded into a third frame, and so on, wherein (if clients 121, 402'' are connected to the Internet) any such one-way function may use sampled information strips from publicly available information sources as direct or derived inputs, and / or (again, if clients 121, 402'' are connected to the Internet) any such one-way function's direct or derived output may be publicly available.

[0320] In practical examples, continuous monitoring may include calculating a continuity score, such as continuously or intermittently. The continuity score may be calculated based at least on the type of automatic image and / or audio processing of the first main video stream. Specifically, the continuity score may be calculated as a defined credibility metric regarding whether the first user 122 has been continuously visible or otherwise present (such as based on voice information) in the first main video stream since the authentication step, or based on that defined credibility metric.

[0321] The purpose of the continuity score is to determine, based on available information, an objective measure of the probability that a previously authenticated user 122 has been continuously present in the first main video stream since authentication (as monitored by the central server 130 and / or clients 121, 402' as discussed above), regardless of whether the first main video stream is used to generate the generated video stream. Authentication itself (which can be partially proven using the first main authentication video) along with continuity monitoring provides a measurable level of authenticity for the user 122 shown in the first main video stream. Therefore, preferably, continuity monitoring is performed uninterruptedly from the authentication step.

[0322] Depending on the specific implementation, various continuity monitoring parameters can be measured, and continuity scores and / or reliability metrics can be calculated based on the measured parameter values. Predefined intervals can be defined for the continuity scores and / or reliability metrics, and in step S812, if any value is detected to fall within or outside such a predetermined interval, the first user 122 can be re-authenticated.

[0323] Such re-authentication can be mandatory, causing a login screen or similar to be presented to the first user 122, and preventing the first user 122 from continuing to participate in the video communication service 110 until re-authentication is performed. In other embodiments, re-authentication is optional, but the presence of the first user 122 in the video communication service 110 can be downgraded in terms of its credibility, which in turn can affect a set of rights associated with the first user 122 with respect to the video communication service 110; can be signaled to other participating users of the video communication service 110; and / or recorded and stored in the form of metadata of the aforementioned type for future reference.

[0324] There may be multiple different such intervals, and / or dynamically calculated intervals. Falling outside these intervals may lead to different defined, predetermined, or parameter-controlled consequences, such as alerting other users and / or requiring re-authentication at different defined security levels.

[0325] Re-authentication can typically be performed in the same or different manner as the authentication steps described above, and / or in the same or different manner compared to any previously performed re-authentication, but the content of the original overall statement regarding the authentication of the first user 122 will also apply to its re-authentication.

[0326] Once re-authentication is complete, the updated first master authentication video portion generated by the re-authentication will be used as the first master authentication video portion from that point in time, including the recalculation of the first one-way function, etc. Therefore, through re-authentication, the surveillance score and / or credibility metric can be immediately improved relative to the first user 122.

[0327] Continuity scores can typically be determined based on one or more of many different types of aspects of the available information. Examples include the following: • The first user 122 disappears from the first main video stream and may reappear, as monitored and detected using standard digital image processing techniques; • The reactions of one or more other users besides the first user 122 to a specific action performed by the first user 122, such as the disappearance and reappearance of the first user 122, as monitored using appropriate digital image processing techniques (e.g., if the first user 122 disappears and reappears, or if the first user 122 changes seats or says something unexpected, as long as no other visible participants are detected to appear startled or surprised by the action in question, the continuity score can remain at a good level). • Since the authentication process began, no words have been detected from the first user 122. • Changes in the visual appearance of the first user 122, such as those monitored and detected using digital image processing technology; • Changes in the voice fingerprint or characteristics of the first user 122, such as those monitored and detected using digital acoustic processing technology; • Changes in the activity level of the first user 122, such as those detected using one or more of image processing, sound processing and speech processing techniques, where activity may refer to speaking, moving, gestures, eye movements, changes in posture, etc. • The factual stance or opinion expressed by the first user 122 (change), such as the first user 122 first expressing a first opinion and then expressing a second incompatible opinion on a factual matter or opinion, which stance or opinion is detected using natural language analysis, such as using a so-called large language model; • The absolute or relative speaking time of the first user 122 in the first main video stream and / or generated video stream, such as relative to the total time or relative to the corresponding speaking time of other users; • The muting behavior of the first user 122, such as the duration of the first user 122 in a muted state or the number of times the first user 122 activates the mute function. • Text-based interactions of the first user 122 with respect to other users, such as the number of times or per unit of time the first user 122 interacts with other users using text input; • The current meeting role of the first user 122, for example, the first user 122 switches from a role with the first set of rights (such as a regular meeting participant) to a role with the second set of rights (such as a meeting administrator), where the second set of rights is more extensive than the first set of rights; • Detected changes in video quality, audio quality, and / or detected interruptions in the continuity of the first main video stream; • This illustrates how the primary video stream of the first user 122 changes from a primary video stream to a different primary video stream (e.g., when the first user 122 switches from using a laptop client 121 to using a mobile client 121 while participating in video communication service 110); and • User 122 of video communication service 110 manually requests to change the required minimum authentication level.

[0328] In some embodiments, the continuity score and / or trustworthiness metric are configured to automatically deteriorate (e.g., decrease) over time (the timeline here being the timeline of the first primary video stream) to reflect the general security and relevance of the first user 122's authentication as time has elapsed since the first authentication or any re-authentication performed. In some embodiments, the first user 122 must re-authenticate after a set period of time in order to maintain the current set of rights in the video communication service 110, or even continue to participate in the service.

[0329] In some embodiments, in addition to the first user 122'', a second user (such as user 122''' or user 122'''') may be visible in the first main video stream, the second main video stream, and / or the third main video stream. This additional user 122''', 122''''' can then verify the authenticity of the first user 122'''s identity. This verification may be performed in association with (or even constitute) the first user 122''s first authentication and / or any re-authentication, or it may be performed during continuous monitoring but not associated with any explicit authentication or re-authentication. This allows the continuity score and / or credibility metric to be altered to indicate a higher probability of continuity or credibility, particularly when the second user 122''' or 122''''' is detected as visible in the same main video stream as the first user 122''.

[0330] This confirmation can take the form of verbal and / or visual communication symbols or phrases, such as user 122''' verbally stating "User 122'' is real," and the trained neural network of system 100 interpreting this phrase as such confirmation based on available main video stream information. In other cases, users 122''', 122''''' can use a graphical user interface or the like to explicitly verify the authenticity of the first user 122'', for example by activating a corresponding control, such as by pressing a button or answering a question. Similarly, in the case where a third user 122'''' or 122''''' actively challenges the authenticity of the first user 122''', this can cause the continuity score and / or credibility metric to be altered to indicate a lower continuity probability or credibility.

[0331] In some embodiments, the continuity score and / or credibility metric can be affected, such as by increasing or otherwise representing an improved continuity probability or credibility, simply by detecting that both the first user 122'' and the third user (e.g., user 122''' or 122'''') are visible in the same master video stream, such as in the first master stream, without requiring any active action by the second user 122'' or 122'''' relative to the first user 122''. Since each master video stream is captured using a single video camera, the fact that two users 122'', 122''', 122'''' are shown in the same master video stream means that these two users 122'', 122''', 122'''' are physically present in the same location, such as in a conference room. This mutual physical presence generally makes it more difficult for the first user 122'' to forge their identity. Therefore, as a result of detecting that both users are visible in the same master video stream, the continuity score and / or credibility score can be modified to reflect the improved continuity probability or credibility. This detection can also be based on digital image processing techniques. It can also be combined with active confirmation or approval from a third user 122''' or 122'''' regarding the authenticity of the first user 122' ...

[0332] Furthermore, the detection of the second user 122''' or 122'''' can also be based, at least in part, on multiple different image frames of the main video stream in question, such as consecutive image frames, detecting movement of one or more of the users 122'', 122''', 122'''', etc., to reduce the risk of fraud by assessing the likelihood that these images actually constitute a real-time video stream of real people rather than, for example, still images shown to the camera in question. Similarly, this analysis can be performed, for example, using standard image processing techniques and / or using a suitably trained neural network of system 100.

[0333] In this way, continuous monitoring can continue over time, generating variable continuity scores and / or credibility metrics, and performing re-authentication of the first user 122'' when needed and / or when the first user 122'' is selected.

[0334] As described above, continuity scores and / or credibility metrics can be stored as metadata, which can then be used and / or processed as generally described below. In some embodiments, continuity scores and / or credibility metrics can be visually represented in the generated video stream. For example, colors, progress bars, icons, symbols, text, or numbers depicted next to the first user 122'' can indicate the currently updated continuity score and / or credibility metric or status of the first user 122'' for one or more other users 122''', 122''''' to view as part of the generated video stream. Continuity scores and / or credibility scores can be stored, including historical change patterns of the continuity scores and / or credibility scores. Continuity scores and / or credibility scores can be stored together with or associated with corresponding metadata indicating one or more detected aspects of the first master video stream and / or generated video stream that caused changes in the continuity scores and / or credibility metrics for retrospective review.

[0335] In some embodiments, each user 122 may be allowed to re-authenticate, for example using the user interface discussed herein, and thus be awarded an improved continuity score and / or credibility metric upon successful re-authentication. Similarly, different users may be allowed to request re-authentication from the first user 122.

[0336] In some embodiments, the invention includes sampling publicly available information sources. As used herein, a “publicly available information source” is an information source that is sufficiently broad and persistently available such that a third party is likely to be able to retrospectively verify the information state of the source at a specific point in time. For example, such information sources include stock market prices, weather data, television / radio news broadcasts, sporting events, public social media feeds, and so on. As used herein, “sampling” such a publicly available information source means reading the current information state of a certain aspect of the source, such as a combination of multiple current stock market prices, and storing information representing that information state. The sampled aspect is one that cannot be reasonably guessed in advance, and is therefore at least detailed enough that such guessing would be futile in practice. Sampling can be performed using a public API. The sampled information is preferably, or at least available via such a public API, for at least several years. The sampled information can be used as is, and / or compressed, hashed, or otherwise processed using one or more one-way functions, wherein preferably, the direct or derived information is configured such that it could not have been known before the sampling time.

[0337] In particular, the sampled information can be used as direct or derived input to compute the output of the source one-way function.

[0338] As used in this article, a "source" one-way function is typically a one-way function that is computed using information sampled from an information source as direct or derived input.

[0339] In some embodiments, the output of a joint one-way function is computed using the direct or derived output of the source one-way function as input, and the source one-way function in turn uses the information sampled from a publicly available information source as direct or derived input.

[0340] As used in this article, a "joint" one-way function is a one-way function that uses two different inputs to compute the output.

[0341] This joint one-way function can also use the corresponding direct or derived outputs of the first one-way function and / or the second one-way function as inputs, and the first one-way function and / or the second one-way function are calculated based on the corresponding main video stream, and specifically based on the corresponding certified video portion (directly or derivedly).

[0342] Then, the direct or derived output of the joint one-way function can be publicly published, as defined above.

[0343] The computation of a joint one-way function can be iterative, meaning that in each iteration, the updated computed values ​​of its inputs are used as direct or derived inputs to repeatedly compute the joint one-way function. The output of the joint one-way function can be publicly published for each iteration, or only for a limited number of iterations.

[0344] In some embodiments, the joint one-way function may use the direct or derived output of the previous iteration of the joint one-way function as input.

[0345] Figure 11 Examples illustrating these principles are shown, where “IS” represents a publicly available information source; “SA” represents sampling; “EM” represents embedding; “OW” represents a one-way function with one or more inputs on its left and an output on its right; “#” represents the result of the one-way function, such as a hash value or the like; “PV” represents the main video stream; “AP” represents the authentication portion of the main video stream; “PR” represents the generated video stream; “PC” represents a publicly available distribution channel; and “PP” represents public release. Figure 11 In the diagram, the horizontal axis represents time.

[0346] It should be understood that Figure 11 Simplified for the purpose of illustrating the principles described in this article. Figure 11Not all features shown are necessary, and additional features described herein may be added. For example, each one-way function OW shown can be a concatenation or chain of any number of one-way functions. A one-way function OW shown as having multiple inputs can be divided into two or more individual one-way functions, each using one or more of these inputs, and can be fed into a common downstream one-way function OW.

[0347] As illustrated for the main video stream (PV) and the generated video stream (PR), the state of any main or generated video stream can be used as a direct or derived input to a one-way function (OW), whose direct or derived output is embedded at a later time point within the same video stream; and / or the state of any main video stream can be used as a direct or derived input to a one-way function, whose direct or derived output is embedded into the generated video stream based on the main video stream in question. The portion of the video stream in which the direct or derived output of the one-way function OW is embedded can be used as a direct or derived input to a subsequent one-way function OW, the output value of which is influenced by the embedded information.

[0348] When a portion of a video stream is used as a direct or derived input to a one-way function OW, any portion of the video stream can be extracted as a direct or derived input to the one-way function OW, such as all or part of a single frame; multiple frames; and / or audio from the video stream. The extracted portion can, for example, be hashed. This hashing may use all the information considered, or only a portion of it (such as using only a defined set of pixels in each frame or the like). Preferably, the extracted information used as a direct or indirect input to the one-way function is configured to depend on the state of the video stream along the timeline of the video stream at the extraction point, which is either an instantaneous state or has a certain duration.

[0349] In general, any information embedded in a video stream PV or PR can be made to affect the state of the video stream PV or PR in question, which serves as a direct or derived input to the later one-way function OW calculation, making it impossible to perform the later one-way function OW calculation if the video stream PV or PR affected by the embedding is not accessible. This can be achieved, for example, by using a portion (such as one or more frames) of the video stream PV or PR with the embedding as a direct or derived input to the one-way function OW, the result of which is subsequently embedded into the same video stream PV or PR at a later time point; and so on.

[0350] like Figure 11The diagram also shows that the direct or derived output of a one-way function OW, which uses a portion of the main video stream PV as a direct or derived input, can be embedded into another main video stream PV, and two different main video streams PV can be cross-linked in both directions in this way.

[0351] Any one-way function OW computation described herein can be iteratively repeated. Among other things, this results in (via the aforementioned joint one-way function): video streams PV and PR with embeddings depending on these computations cannot have existed (with that embedding) prior to the latest sample of a publicly available source of information used as input to the computation; and the information used as input to the computation cannot have appeared after the earliest public release of the computation's output. This locks each piece of information subjected to one-way function computation between the earliest and latest time points relative to the absolute timeline, thereby enabling the determination with high determinism of the time points when information was used as direct or derived input to the joint one-way function, and thus the relative order of any frames in a video stream processed in this way, as well as the time point at which such video stream was created.

[0352] More specifically, for a video stream, such as any main video stream (PV) or generated video stream (PR) whose state is used as a direct or derived input to a joint one-way function, and the direct or derived output of the joint one-way function is embedded into the video stream in such a way that the next state, which is used as the direct or derived input to the next iteration of the joint one-way function computation, is affected by this embedding. This means two things: • Capable of retrospectively determining the relative order of states (such as frames or sets of frames) in a video stream with high confidence; and • It can retrospectively determine the point in time when a video stream was captured or generated with high confidence.

[0353] This determination requires knowing all the information used as input to the one-way function in question, as well as which one-way functions were used at each step. In some embodiments, the invention includes persistently storing all such required information for future use, such as in the form of metadata of the general type discussed herein. However, in general, the output of the one-way function can be made available via embedding in the video stream whose authenticity is to be verified, and therefore does not need to be stored separately.

[0354] For any master or generated video stream, the current state of the video stream can be used as a direct or indirect input to a joint one-way function, which is computed at least once per minute, such as at least once every ten seconds, such as at least once per second, such as at least once every 100 video frames, such as at least once every 10 video frames, such as once per frame. The latest state of a publicly available information source can be sampled at least once per hour, such as at least once every ten minutes, such as at least once per minute, such as at least once every ten seconds, and used as a direct or derived input to the joint one-way function. The updated direct or derived output of the joint one-way function can be publicly published at least once per hour, such as at least once every ten minutes, such as at least once per minute, such as at least once every ten seconds. The time between sampling of a publicly available information source and the public publication of the direct or derived output of the joint one-way function using the samples in question as direct or derived inputs can be at most one hour, such as at most ten minutes, such as at most one minute, such as at most ten seconds.

[0355] Therefore, for each main video stream or generated video stream, information relating to different portions or frames of the video stream in question can be iteratively used as direct or indirect inputs to the joint one-way function. The maximum time between the occurrence of such a portion or frame and its use as a direct or derived input to the joint one-way function can be at most ten minutes, such as at most one minute, such as at most ten seconds, such as at most one second.

[0356] As can be understood from the above, the joint one-way function OW serves as a mechanism for interweaving information along a timeline, within video streams PV and PR (via the video stream being fed into itself at a later point on its timeline); across video streams PV and PR (via the video stream being fed into another video stream); and / or together with publicly available information sources IS and / or publicly available channels PC (via the publicly available information source IS being fed into the video stream, or the video stream being fed into the publicly available channel PC), to form an interconnected information “interweaving” or “web” mathematically bound to the timeline. Verification of this “interweaving” or “web” involves retrospectively examining the corresponding information as needed, which may cover all input and output information of the one-way function OW used, and in some embodiments specifically covers the publicly available information sources IS and / or publicly available channel PC.

[0357] Such “interweaving” of the main video stream PV and / or the generated video stream PR can be designed so that one, more, or all of these video streams are interconnected in the sense that at least one, such as more or even all, of the associated output values ​​of the joint one-way function OW cannot be computed when at least one or more parts of each of the main video stream PV and / or the generated video stream PR are not accessible.

[0358] For example, since the first primary authentication video portion (AP) is "interleaved" in this manner, it is possible to retrospectively verify that the authentication of the first user 122 occurred, how the authentication was performed, and what the result of the authentication was; and if appropriate relevant information is "interleaved" from the first primary video stream (PV), the process of continuous monitoring can also be retrospectively verified. Continuous monitoring may also include, for example, retrospectively verifying, once or repeatedly over time, based on the joint one-way function (OW), that the authentication of the first user 122 was indeed "interleaved" and therefore verifiable.

[0359] In addition to the examples above, the joint one-way function OW can also use other information as direct or derived inputs, which will be "interwoven in" in a corresponding manner, and thus become traceable and verifiable in terms of information integrity and time.

[0360] Therefore, the joint one-way function OW can be computed using one or more of the following as direct or derived inputs: the first primary authentication video portion AP; the first authentication token; the updated continuity score; a one-way function computed using the generated video stream PR as direct or derived input (“third” one-way function); and a set of metadata that directly or indirectly but explicitly describes the complete set of generation steps for generating the generated video stream PR based on the first primary video stream PV and the second video stream PV.

[0361] exist Figure 8 The general method of this "interleaving" is illustrated in steps S813 to S817. In step S813, one or more publicly available information sources IS are sampled. In step S814, one or more inputs are identified, which are portions of one or more main video streams PV and / or generated video streams PR. Here, the term "input" refers to the direct or derived input of the joint one-way function OW. In step S815, the output of the joint one-way function OW is computed using the sampled information and the identified information as direct or derived inputs. It should be understood that different joint one-way functions OW may be used in different iterations. In step S816, the direct or derived output is embedded into one or more main video streams PV and / or generated video streams PR. In step S817, the direct or derived output is publicly released using a publicly available channel PC. This part of the method can be described as follows: Figure 8 The iteration is performed as shown. Furthermore, the iterations between steps S813-S817 can be performed independently of the iterations between steps S801-S812, even if the information generated in steps S801-S812 or information associated with steps S801-S812 can be used as input in step S814; and even if the embedding in step S816 can be related to any video stream used in or associated with steps S801-S812.

[0362] As described above, using the various described "interweaving" schemes, information related to any main video stream (PV), any generated video stream (PR), and historical or ongoing user authentication can be retrospectively verified. However, as also mentioned, such verification requires direct or derived access to any information used as input to the relevant one-way function (OW). Without such information, the accuracy of the corresponding output of the one-way function (OW) in question cannot be verified.

[0363] Figure 12 A method for verifying information using downsampled versions of the main video stream and the generated video stream is demonstrated, without requiring access to the original information. Specifically, Figure 12 The method shown allows for the avoidance of storing the potentially very large amount of information associated with high-resolution video material, which can be generated when using the solution according to the invention. Figure 8 The method shown is the same. Figure 12 This method is one for generating video streams, and it can be categorized as discussed above and used as an example. Figure 9 The system execution is shown in the figure.

[0364] In the first step S1200, the method begins.

[0365] In the subsequent step S1201, a first main video stream is captured, such as that captured by camera 401'', which visually shows the first user 122, 122''. In this case, a second main video stream is also captured, such as that captured by camera 401' or camera 401''', which visually shows the second user. As mentioned above, the second user can be user 122''' or user 122''''. Step S1201 may correspond to steps S802 and S804, and all the above descriptions regarding these steps apply accordingly.

[0366] In the subsequent step S1202, a generated video stream is automatically generated based on the first main video stream and the second main video stream. This automatic generation is based on an automatic generation decision, which in turn is based on at least one of the following: defined parameters; automatic image processing of the first main video stream and / or the second main video stream; and automatic audio processing of the first main video stream and / or the second main video stream. This step S1202 may correspond to step S806, and the above description applies accordingly.

[0367] In step S1203, which follows step S1201, the first main video stream is downsampled to obtain a downsampled version of the first main video stream, referred to herein as the first "shadow" main video stream. Furthermore, the second main video stream is downsampled to obtain the second shadow main video stream.

[0368] In the subsequent step S1204, a generated shadow video stream can be automatically generated based on the first shadow main video stream and the second shadow main video stream. Preferably, the automatic generation of the shadow video stream is based on a corresponding automatic generation decision used to generate the generated video stream, and the automatic generation decision is applied in the same manner, the only difference being that the generation uses downsampled material. Therefore, a generated shadow video stream can be generated to completely correspond to the generated video stream generated in step S1202, or at least completely correspond to a subset of the generated video stream generated in step S1202, but with lower quality, for example, in terms of image quality.

[0369] This can be described as follows: the automatic generation of the shadow video stream is based on the same automatic generation decisions used to generate the generated video stream. The significance is that the generation decisions themselves are identical, but they apply to the (downsampled) shadow video stream information. Therefore, the automatic generation of the generated video stream (PR) can be performed using a set of automatically applied generation decisions, and the same set (or corresponding sets) of automatically applied generation decisions can also be used to automatically generate the shadow video stream. In some embodiments, the same or corresponding complete set of automatic generation decisions can be used for both types of automatic generation, and in some embodiments, no other generation decisions are used besides this set of generation decisions to generate the shadow video stream.

[0370] The generated video stream can be provided to the receiving user in a manner corresponding to that described in step S807.

[0371] In the following text, the first and second main video streams are referred to as "raw quality" main video streams, and the generated video stream produced in step S1202 is referred to as the "raw quality" generated video stream, in order to distinguish them from the corresponding "shadow" video streams. It should be understood that everything described above regarding the various video streams, especially in conjunction with... Figures 8 to 11 Everything mentioned can still be found Figure 12 The method described is applicable and usable in the context of both the original quality video stream and, independently, the shadow video stream. Overall, the provisioning and use of the corresponding shadow video stream can be performed in parallel with other method steps.

[0372] In the subsequent step S1205, the first shadow master video stream, the second shadow master video stream, and the generated shadow video stream are stored, preferably in a persistent and permanent manner, such as on a conventional hard disk drive, flash drive, non-volatile memory storage device, etc. This storage is performed in a way that allows for complete retrospective copying of each shadow video stream; in other words, no lossy compression or cropping algorithms or similar algorithms are used. In addition to the shadow video streams, information identifying all automatic generation decisions used to generate the generated shadow video streams is also stored in a corresponding manner. Specifically, the stored information should be sufficient for the generation of the generated shadow video streams based on the retrospective copying of the shadow master video streams. In this way, the information content (images and / or audio) can be retrospectively verified for its content.

[0373] In the subsequent step S1208, the method ends. However, as... Figure 12 As shown, the method can be iterated so that as long as the capture of the main video stream and the generation of the generated video stream are in progress, the original quality video stream is continuously provided and downsampled to generate and store the shadow video stream.

[0374] Overall, combined Figure 8 and Figure 12 The methods described can be freely combined, and in particular, Figure 12 The method shown can be used in conjunction with the above. Figure 8 An additional scheme for any of the embodiments discussed ensures that a corresponding master shadow video stream is provided and stored for each master video stream and generated video stream occurring in such an embodiment, and that a corresponding shadow video stream is generated and stored.

[0375] In general, this additional approach may include downsampling at least one of the first main video stream and the second main video stream to obtain a downsampled first shadow main video stream and / or a downsampled second shadow main video stream; and then performing any or all of the method steps discussed in relation to the first main video stream, the second video stream, and the generated video stream in a corresponding manner but using the corresponding first shadow main video stream and / or the second shadow main video stream instead, thereby generating the corresponding generated shadow video stream.

[0376] Preferably, the retrospective visual and / or audible checks on the generated shadow video stream are configured such that they can be used to evaluate the information content of the generated shadow video stream, and the corresponding information content of the stored main shadow video stream, along with information about how the automatic generation was completed, can be used to verify that the generated shadow video stream was indeed generated based on the main shadow video stream in a specified manner. For example, it is typically required that humans be able to visually locate and potentially identify human participant user 122 visible in the generated shadow video stream.

[0377] The authenticity of the original quality generated video stream can then be verified by visually and / or audibly comparing it with the generated shadow stream, thereby confirming that the original quality generated video stream does indeed correspond to the generated shadow video stream.

[0378] To achieve this goal, the generated shadow video stream may contain embedded information, as long as the embedded information does not hinder or degrade this visual / auditory inspection or comparison. For example, any type of embedding described above, such as QR codes, alphanumeric additions, or watermarks, can typically be used without hindering or significantly degrading this inspection or comparison with the original quality generated video stream. In some embodiments, the generation step of embedding one or more pieces of information not embedded in the original quality generated video stream is specified in the generation information stored in step S1205.

[0379] An example of such information embedded in the generated shadow video stream is metadata about the automated generation steps performed to generate the generated shadow video stream; in other words, it is the generation information stored in step S1205, or a reference or pointer to such metadata or information. This provides a self-contained generated shadow video stream that can be evaluated on its own, even without independent or direct access to the generation information stored in step S1205. In some cases, this information may also include a reference or link to a stored master shadow video stream used to generate the generated shadow video stream, allowing a third party wishing to verify the authenticity of the generated shadow video stream to retrospectively access it.

[0380] In some embodiments, the generated shadow video stream is not provided to the first user or the second user 122, such as not being shown to the first user or the second user 122. In other words, step S807 is not performed for generating the shadow video stream. In some cases, the generated shadow video stream, or even all shadow video streams discussed herein, is not provided to or shown to the first user or the second user 122, or any human participant user of the video communication service 110, but is only stored for future reference. Therefore, if and when someone wishes to use the shadow video stream for verification purposes, the shadow video stream will certainly be provided to that person, but not as part of the ongoing video communication service 110, and in particular, not in real-time or near real-time as the original quality generated video stream.

[0381] As explained above, one purpose of generating shadow video streams is to enable visual and / or audible verification of the information content of the shadow video stream to determine whether it matches the corresponding information content of the original quality generated video stream. Such information content that one wishes to verify may include one or more of the following: the identity of the people shown in the shadow video stream; the language content of the discussions seen in the shadow video stream; the identity of the viewed objects; and so on. To achieve this goal, it is generally not necessary to provide the shadow video stream at a quality that meets modern video content standards. In other words, it is usually possible to compress the original quality video stream to a considerable extent without losing the possibility of retrospectively achieving this verification.

[0382] In any case, preferably, downsampling of the original quality main video stream is performed without removing too much information from it, so that the presence and identity of the first user 122, which is clearly visible and identifiable in the first main video stream, can also be visually identified in the resulting shadow video stream.

[0383] In some embodiments, downsampling of each primary shadow video stream is configured to reduce the byte size by at least 10 times, or even at least 100 times. In this and other cases, downsampling of the primary shadow video stream may include one or more of the following compression methods: • Compared to the original quality main video stream in question, the pixel resolution of the shadow video stream in question is reduced uniformly or selectively on the image plane and / or along the timeline, for example, reduced to at most half or even at most one-fifth of the corresponding original quality pixel resolution in any or every pixel map dimension and in at least one or more video frames. • Compared to the original quality main video stream in question, the pixel depth of the shadow video stream in question is reduced uniformly or selectively on the image plane and / or along the timeline, for example, reduced to at most half or even at most one-fifth of the pixel depth of the corresponding original quality for at least one or more image frames, and / or at least one or more image frames of the original quality color video stream are converted into grayscale video streams or even black / white video streams; • Compared to the original quality main video stream in question, the frame rate of the shadow video stream is reduced uniformly or variablely along the timeline of the shadow video stream in question, for example, to at most half, or even at most one-fifth, of the corresponding original quality frame rate for at least a portion of the shadow video stream; and • The shadow video stream is cropped and added to at least a portion of the shadow video stream in question, compared to the original quality main video stream in question, such as by cropping the background, for example, in the case where one or more users 122 and / or detected objects of interest constitute the foreground in the original quality video stream in question.

[0384] In some embodiments, downsampling of each or any original quality master video stream is dynamic, meaning it can vary along the timeline of the video stream and / or across the image planes of the video stream. For example, downsampling may take into account one or more defined parameter values ​​of the automatically generated decision stored in step S1205, such that, observed along the timeline of the master video stream in question, downsampling is applied differently over time, across the image planes of one or more original quality master video streams, and / or across different original quality master video streams.

[0385] For example, the automatic generation of the original quality video stream can be based on (as described above) the automatic detection of the currently speaking user 122; the position of one or more users 122 in the main video stream; the occurrence of one or more events and / or patterns; and so on. Downsampling can then be applied such that relatively less information is compressed from the image frame region containing one or more speaking or non-speaking users 122 compared to other image frame regions, and / or downsampling can be applied such that relatively less information is compressed from the temporal and / or image frame portions of the original quality main video stream containing a higher concentration of detected events and / or patterns. It should be understood that these are merely examples, and many different dynamic compression techniques can be applied. Of course, compression can be applied dynamically, alternatively or additionally, independent of the automatic generation decision, such as using conventional variable video compression techniques.

[0386] The generation of the shadow video stream can be performed by any suitable generation function 135 of the type described above, such as the same or different generation functions 135 that produce the original quality generated video stream.

[0387] In some embodiments, at least one, some, or all of the first original quality master video stream, the second original quality master video stream, and the original quality generated video stream are discarded after the generated video stream is provided to (as shown) the first user. Specifically, in some embodiments, all original quality master video streams may be discarded even if the original quality generated video streams are persistently stored. "Discarding" a video stream means not storing it persistently and permanently, or at least the system 100 does not perform such storage but deletes the video stream immediately or with a delay; the purpose of discarding may be to save available storage space.

[0388] In principle, steps S1203 and S1204 can occur at a later time than step S1202, such as much later, for example, several days later. However, to minimize the possibility of fraud, in some embodiments, step S1204, in other words, the generation of the shadow video stream, occurs with a time delay of up to one minute, such as up to ten seconds, relative to the generation of the original quality generated video stream. This is particularly true in the preferred case where the shadow video stream is "interleaved" in the general manner described above.

[0389] In general, all shadow video streams can be "interwoven" in a manner corresponding to the "interweaving" described above in conjunction with the original quality video streams. This involves using a joint one-way function OW, sampling publicly available information sources IS as input information, publicly publishing output information using publicly available distribution channels PC, and embedding the output information into EM. This "interweaving" of shadow video streams can be designed to interconnect one, more, or all of the main shadow video stream and / or generated shadow video streams; interconnect one, more, or all of the main original quality video stream and / or generated original quality video streams; and / or interconnect one, more, or all of the main shadow video stream and / or generated shadow video streams with one or more of the main original quality video stream and / or generated original quality video streams, where the term "interconnect" has the same meaning as described above regarding "interweaving" different video streams together.

[0390] Specifically, a piece of information, the direct or derived output of a one-way function computed using a previous frame of the shadow video stream as direct or derived input, is embedded into a later frame of the shadow video stream, veribly preserving the relative order between the previous and later frames. The direct or derived output of a one-way function computed using a common information source with direct or derived sampling as input is added to the frames of the shadow video stream, veribly placing the shadow video stream after the sampling time along the timeline. The direct or derived output of a one-way function computed using frames of the shadow video stream as direct or derived input is publicly released, veribly placing the shadow stream before the time of public release.

[0391] In some embodiments, the cryptographic fingerprint of the main video stream is embedded in the shadow video stream (such as in the shadow video stream corresponding to the main video stream) as the direct or derived output of a one-way function computed using frames of the main video stream as direct or derived inputs. The reverse can also be true (embedding the digital fingerprint of the shadow video stream into the main video stream). This verifiably locks the main video stream relative to the shadow video stream along the timeline.

[0392] Specifically, a first shadow one-way function can be computed to correspond to the first one-way function described above. Therefore, the first shadow one-way function can be computed using the first shadow main video stream as a direct input or a derived input. Correspondingly, a second shadow one-way function can be computed using the second shadow main video stream as a direct input or a derived input to correspond to the second one-way function described above. The first shadow one-way function can be computed using the first shadow authentication video portion of the first shadow main video stream as a direct input or a derived input, which corresponds to the first main authentication video portion and shows the authentication steps and / or the representation of information indicating the first authentication token in a manner corresponding to that described above, but within the corresponding downsampled video. Then, for each of the corresponding pieces of information representing the direct output or derived output of the first shadow one-way function and the second shadow one-way function, at least one of the following can be performed: embedding the visual and / or audible representation of the piece of information into the generated shadow video stream; storing the piece of information; and publicly publishing the piece of information. The content described above regarding the first authentication video portion and its subsequent use in the method can also be applied to the first shadow authentication video portion, but using the shadow video stream instead of the original quality video stream.

[0393] Furthermore, publicly available information sources IS that are the same as or different (but similar to) the publicly available information sources IS discussed above can be sampled, and these samples can be used as input to compute a second source one-way function, which corresponds to or is actually the aforementioned source one-way function (which can also be represented as the "first" source one-way function). Then, a second joint one-way function OW can be computed as input, which corresponds to or is actually the aforementioned joint one-way function OW (which can also be represented as the "first" joint one-way function): the direct or derived output of the second source one-way function; the direct or derived output of the first shadow one-way function; and the direct or derived output of the second shadow one-way function. Finally, the direct or derived output of the second joint one-way function OW can be publicly published using the public distribution channel PC in the manner described above. In the case that the first and second joint one-way functions OW are the same, only one public publication is required for that particular iteration.

[0394] To retrospectively verify that the original quality master video stream was indeed used to generate the corresponding master shadow video stream, the method may include one or more of the following: • The direct or derived output of a first or second shadow one-way function is embedded into one or more original quality main video streams, the first or second shadow one-way function having been computed based on the main shadow video stream corresponding to the original quality main video stream in question. This embedding then occurs after downsampling of the original quality main video stream in question and may also include explicit information on how the embedding is performed into the original quality main video stream, such that the embedding can be inverted to perform the verification (using the original quality main video stream prior to the embedding).

[0395] • Embed the direct or derived output of one or more of the first and second shadow one-way functions into one or more generated raw quality video streams.

[0396] • Embed the direct or derived output of the corresponding first or second one-way function into one or more shadow master video streams (the first or second one-way function is calculated based on the original quality master video stream corresponding to the master shadow video stream in question).

[0397] As described above, the shadow video stream, along with information about how the automatic generation was performed, can be used to verify the legitimacy of the available original-quality generated video stream, including that the first user 122 visible in the original-quality generated video stream has been properly authenticated. This is achieved, in particular, by visually / audibly examining the original-quality generated video stream and comparing it with the corresponding information content in the generated shadow video stream. Specifically, various techniques are used to “interweave” the various video streams, making the verification stronger, especially regarding the relative and absolute timing of the various parts and aspects of the original-quality generated video stream. One advantage of this approach is that the verification can be performed without storing potentially large original-quality master video streams.

[0398] However, in other scenarios, a master video stream may be required, such as for generating different generated video streams at later points in time. In cases where the original high-quality master video stream is not subsequently persisted, only the master shadow video stream is retained.

[0399] In this case (and others), the method may include using this master shadow video stream as a regular master video stream of the type discussed herein (in other words, as a raw quality master video stream), but using its converted version to improve its usefulness as such a master video stream.

[0400] In general, step S1205 may include storing at least one piece of context-related information about the first main video stream and / or the second main video stream, such as in the form of various types of metadata discussed above, which cannot be derived from the shadow main video stream corresponding to the main video stream to which it relates, and in particular cannot be derived from one or both of the first shadow main video and the second shadow main video.

[0401] The method may then include: at a later point in time, such as at least one hour, at least one day, or even at least one week later, using the stored context-related information to transform at least a portion of at least one of the first and second main video streams, thereby obtaining the first transformed main video stream and / or the second transformed main video stream.

[0402] In this context, the terms “conversion,” “enhancement,” and “upsampling” are used interchangeably, and correspondingly, “converted,” “enhanced,” and “upsampled” are also used interchangeably.

[0403] Then, a converted or enhanced generated video stream can be generated based on the first enhanced master video stream and / or the second enhanced master video stream. This generation can typically be performed using any of the automatic generation methods described herein.

[0404] The stored context-related information may include metadata describing one or more events (as defined above) or patterns (as defined above) that can be detected in the first main video stream and / or the second main video stream.

[0405] The stored context-related information may include metadata describing one or more things, people, or phenomena that are visually shown and / or audibly heard in the first and / or second main video streams, such as the identity of the user who is seen or speaking in the first and / or second main video streams, or the facial expressions of such users.

[0406] In other words, the method may include step S1206, in which the first shadow main video stream, the second shadow main video stream and / or the shadow video stream are automatically converted (enhanced / upsampled) and / or generated to obtain the corresponding converted (enhanced / upsampled) video stream.

[0407] In the subsequent step S1207, each such converted video stream may be provided to user 122, such as by showing it to user 122, or used as a primary video stream of original quality in any manner discussed herein. For example, a next-generation primary quality generated video stream may be generated based on the converted video streams discussed, possibly together with the additional primary video streams as described above.

[0408] In various embodiments, the transformation (enhancement / upsampling) may include one or more of the following: • Increase audio bitrate, pixel resolution, pixel depth, and / or frame rate; • Introduces de-cropped, image elements and / or audio elements, such as backgrounds, based on previously stored metadata; and • Partial or complete animation of participants shown in the converted video stream, which may be based on previously stored metadata.

[0409] All these transformations result in information being added to the main shadow video stream. This added information cannot be inferred from the main shadow video stream itself, necessitating at least one of the following: retrieving this added information from an available source of information (such as stored context-related information), and making an educated guess about the content of this added information. Generally, these tasks can be accomplished using appropriately trained neural networks, employing statistical analysis of the main shadow video stream to predict the natural appearance of filling in the gaps lost during the downsampling of the main shadow video stream. This can be done automatically, for example, using so-called generative AI tools such as large language models. In simpler cases, interpolation techniques can be used to insert additional pixels and / or frames into the video stream. This process can also be analogous to the upsampling of video signals used in modern television monitors. Corresponding processing can be performed on audio content, adding frequencies to achieve audio that sounds natural.

[0410] Furthermore, the conversion can utilize various types of available information.

[0411] For example, audio information from the master shadow video stream can be interpreted to better understand what is visually happening in the image content, such as understanding that the depicted user 122 is currently speaking or moving around. Accordingly, the images from the master shadow video stream can be used to artificially enhance the audio, for example, to make image processing lead to an understanding of the source of a particular sound, such as a visible object tipping over and falling to the floor.

[0412] Available information may include image and / or audio material related to user 122 and / or objects visible in the main shadow video stream. For example, a still image of the user's face may be used to improve the pixel resolution of the user 122's face in the converted video stream. This information may be of different types and may include, for example, information about the user 122's body size, normal posture, typical movement patterns, and voice attributes.

[0413] The aforementioned metadata can also be used as available information in the current context. In association with the downsampling of the original quality main video stream, information about what the video stream shows, what happens in the video stream, etc., can be automatically and selectively extracted using available digital audio and / or video processing techniques and / or defined parameter values, and stored as metadata in step S1205 (or as part of the metadata stored in step S808). This metadata may include, for example, the identity and / or personal attributes of user 122 visible in the original quality main video stream; textual or parameter descriptions of scenes visible in the original quality main video stream, particularly portions that have been cropped or significantly compressed due to downsampling; specifications of events; color information related to portions of the original quality video stream where pixel depth has been reduced or color information has been otherwise lost; and / or patterns occurring in the original quality main video stream, such as two users shaking hands, a particular user smiling or frowning, or a particular user looking at a particular other user. Using this information, the conversion can then be used to generate the best guess of artificial improvement for the main shadow video stream in order to obtain a converted main video stream that appears to be of higher quality. In all these examples, the available information can be combined with generative techniques such as interpolation and generative AI in various ways. For example, textual information about the scene can be fed into a large language model whose task is to graphically improve the shadow video stream in question.

[0414] In some embodiments, each or some shadow video streams, or the metadata associated with each or some shadow video streams, may be created to include certain image and / or audio information (pixel resolution, pixel depth, sampling rate, etc.) of higher quality. For example, intermittent frames (such as one frame out of every ten frames or similar frames) may have higher quality, such as full image quality, compared to the remaining frames. The conversion may then include manually rendering an improved quality version of the shadow video stream frames with lower quality using the higher quality portions. This may be performed, for example, using a properly trained neural network.

[0415] Note that the possibility of using the master shadow video stream to verify the original, high-quality master video stream is unaffected by this conversion, as the unconverted master shadow video stream is still stored for future reference. Also note that any converted master shadow video stream or generated video stream can be used as the master video stream in any of the methods described herein.

[0416] In general, the available information in the form of stored metadata can be determined or calculated based on at least one of the first original quality main video stream, the second original quality main video stream, and the original quality generated video stream. The metadata may also include externally provided information, such as personal information about one or more users 122 visible in the original quality main video stream.

[0417] In some embodiments, downsampling is irreversible and requires the addition of manually inferred information to obtain the transformed main video stream. The transformation itself can also be an irreversible process.

[0418] Figure 13 Another method is shown, which is similar to Figure 8 and Figure 12 The method shown is the same as that used to generate video streams, and can be seen from the discussion above as an example. Figure 9 The system 100 shown in the figure is executing.

[0419] In the first step S1300, the method begins.

[0420] In a subsequent step S1301, which corresponds to step S801, the first user 122 is authenticated.

[0421] In the subsequent concurrent step S1302 (which may correspond to steps S802, S804, and / or S1201), a first main video stream is captured to visually represent the first user 122. Furthermore, a second main video stream may be captured simultaneously, partially overlapping, or at different times to visually represent the second user. Similarly, the first user 122 may be... Figure 10 The user shown is 122'', and the second user can be either user 122''' or 122''''.

[0422] The first primary video stream may include a first primary authentication video portion as described above, which visually illustrates the authentication steps and / or includes information that visually and / or audibly represents the first authentication token generated by the authentication steps.

[0423] In step S1303, which may be performed partially or completely concurrently with step S1302, a fourth main video stream is captured. The fourth main video stream may be similar to the first main video stream, and the above description of the first main video stream also applies accordingly to the fourth main video stream. However, the fourth main video stream is captured to visually represent the first user 122 from a viewpoint different from the first main video stream. For example, the fourth main video stream may use... Figure 10 The fourth main video stream can be captured using a different physical camera 401' than the physical camera 401'' (sometimes referred to as the "first" camera) used to capture the first main video stream; this physical camera 401' may be referred to as the "fourth" camera.

[0424] The fourth primary video stream also includes a fourth primary authentication video portion, which may correspond to the first primary authentication video portion. Everything described above regarding the first primary authentication video portion applies correspondingly to the fourth primary authentication video portion.

[0425] Therefore, the fourth main authentication video portion visually shows the authentication steps of the first user 122 (but from a different viewpoint than the first main authentication video portion) and / or includes information that visually and / or audibly represents the first authentication token.

[0426] It should be understood that the authentication steps visually shown in the first and fourth main video streams are the same steps but viewed from different viewpoints, and / or the first authentication token visually represented in the first and fourth main video streams is the same token.

[0427] In a subsequent step S1304, which corresponds to step S803, the first one-way function is computed using the first main video stream as a direct or derived input. Specifically, the first main certified video portion is used as a direct or derived input to compute the first one-way function.

[0428] Furthermore, a fourth one-way function is computed. The fourth one-way function corresponds to the first one-way function in terms of the possible types and functions of the one-way function, and everything described above regarding the first one-way function applies accordingly to the fourth one-way function. Therefore, the fourth one-way function is computed using the fourth main video stream as a direct or derived input. Additionally, the fourth one-way function is computed using the fourth main certified video portion as a direct or derived input. As will be discussed below, the fourth one-way function and the first one-way function can be the same or different, and can also be identical in the sense that they compute only one one-way function using multiple inputs.

[0429] In a subsequent step S1305, which may correspond to step S806 or step S1204, a generated video stream may be automatically generated based on the first main video stream. The generated video may also be automatically generated based on a fourth main video stream, which may subsequently be used as any main video stream described herein. Therefore, the automatic generation of the generated video stream, in addition to the first main video stream, may, but does not necessarily, be based on a fourth main video stream.

[0430] In a subsequent step S1306, which may correspond to step S807, the generated video stream may be provided to a receiving user, which may be a first user 122'', a second user 122''' / 122'''', a different user who is a party to the same video communication service 110, and / or an external user 150.

[0431] In subsequent steps corresponding to step S808, for each of the corresponding messages representing the direct or derived output of the first one-way function and / or the fourth one-way function, at least one of the following is performed: • Embed the visual and / or audible representation of the information into at least one of the first main video stream, the second main video stream, and the generated video stream PR, such that the embedding depends on the content in a video stream different from the video stream into which the embedding is performed; • Store this information; and • Publicly release this information.

[0432] For example, the direct or derived output of the first one-way function can be embedded into the fourth main video stream (PV) and / or the generated video stream (PR); and / or the direct or derived output of the second one-way function can be embedded into the first main video stream (PV) and / or the generated video stream (PR). This embedding can be performed iteratively, with each iteration based on an updated version of the information in question.

[0433] Everything described in this article in connection with how a message output by the first one-way function is used and processed also applies to a message output by the fourth one-way function.

[0434] In some embodiments, the first one-way function and the fourth one-way function may be the same one-way function computed using both the first and fourth certified video portions as direct or derived inputs. In this embodiment, for each computation of the one-way function in question, only one piece of information will be embedded, stored, and / or publicly disclosed, representing the direct or derived output of the single one-way function corresponding to both the first and fourth one-way functions. This process can, of course, be repeated subsequently.

[0435] In other embodiments, there are two distinct pieces of information, both of which are embedded, stored, and / or publicly published. In this case, one of the two distinct pieces of information may be processed and manipulated in the same manner as the other, or at least each of them may be embedded, stored, and / or publicly published in the same manner.

[0436] exist Figure 13In this context, the method is described within the scenario where the first user 122 is authenticated. However, it should be understood that the corresponding method can be used for other purposes, such as recording and generating video streams of any meeting or event proceedings. In this more general case, the two cameras capturing the first and fourth main video streams can be positioned in the same physical space, or they can be positioned in different physical spaces, as long as their respective fields of view cover the same meeting or event proceedings that occur at a well-defined physical location, but the fields of view of the two cameras have no physical overlap or only a very small physical overlap, and thus depict different users and / or objects in practice. In a practical example, two different cameras may capture the corresponding main video streams of the same large meeting, concert, trade show, sporting event, or similar activity, and be positioned in different locations or with their fields of view facing away from each other, but such that the first and fourth main video streams depict events occurring within the same large context (meeting, concert, trade show, sporting event, etc.). For example, participating users may be shown in the first main video stream and subsequently in the fourth main video stream. It should be understood that in this embodiment, the method may not include the authentication step S1301, and one or both of the first main video stream and the fourth main video stream may not include the corresponding authentication video portion.

[0437] Figure 14 Corresponding to Figure 11 ,but Figure 14 Different subsets of possible information interactions are shown to illustrate examples of using the first and fourth main video streams as described in the current description. Figure 14 In the above, the topmost main video stream (PV) is the first main video stream, while the bottommost main video stream (PV) is the fourth main video stream. Even in some embodiments, the opposite can also be true.

[0438] like Figure 14 As shown, a publicly available information source IS is sampled SA, and the sampled information is used as direct or derived input to compute a third-source one-way function OW (such as...). Figure 14 (represented as a one-way function of D in the diagram). The direct or derived output of the third source one-way function OW can be embedded into the first main video stream PV and / or the fourth main video stream PV, such as during their respective authentication portion AP or not during their respective authentication portion AP.

[0439] The third joint one-way function OW can be computed using the direct or derived output of the third source one-way function OW, as well as the direct or derived output of the first and / or fourth one-way functions OW, as inputs. The direct or derived output of the third joint one-way function OW can be publicly released via PP to the publicly available channel PC. Figure 13In this context, the third joint one-way function OW can be, for example, composed of a one-way function OW labeled A. This one-way function A is calculated based on the output of the one-way function OW labeled C (here, the fourth one-way function), because this information is embedded in the first main video stream PV (EM), and thus affects the portion of the first main video stream PV used as the direct or derived input of one-way function A. One-way function C, in turn, is calculated using a portion of the fourth main video stream PV as its direct or derived input. Since the direct or derived output of one-way function D is embedded in the first main video stream (EM), one-way function A is also calculated using the output of one-way function D as its derived input, via the embedding of the EM and the extraction of a portion of the first main video stream PV affected by the embedded EM into one-way function A.

[0440] Note that since the direct or derived output of the one-way function OW (here, the first one-way function) labeled B is calculated using the first main video stream PV as the direct or derived input, the one-way function C and thus the one-way function A also use the output of the one-way function B as the derived input for calculation.

[0441] like Figure 13 As shown, the direct or derived output of the fourth one-way function C can be embedded into the first main video stream PV. Furthermore, the direct or derived output of the first one-way function B can be embedded into the fourth main video stream PV. Either or both of these embeddings into the EM can be repeated over time, effectively "intertwining" the first and fourth main video streams PV in a bidirectional manner. Along the timeline of the fourth main video stream PV and at least during the authentication video portion AP of the first main video stream PV, update information from the direct or derived output form of the one-way function OW calculated based on the first main video stream PV can be embedded into the fourth main video stream PV at least once every ten seconds, such as at least once per second, or even once every ten video frames, or even more frequently. The corresponding situation can be determined regarding how frequently update information from the direct or derived output form of the one-way function OW calculated based on the fourth main video stream PV can be embedded into the first main video stream PV along the timeline of the first main video stream PV.

[0442] exist Figure 11 and Figure 13In both cases, the embedding EM is shown as relating to a specific point in time in the video stream in question where the embedding EM is performed. Note that such an embedding EM can be performed such that it also affects the video stream in which the embedding EM is performed, at least such that subsequent portions of the same video stream used as direct or derived inputs to a subsequent one-way function OW will depend on the embedding EM, to achieve the cryptographic "linking" effect described above. Each embedding can also be more or less transient and can leave the video stream in question unaffected after a certain time period or number of frames. In any case, when a portion of the video stream is used as a direct or derived input to the one-way function OW in the next instance, the embedding EM and / or the extraction of such a portion of the video stream can be configured such that the embedding EM information affects the subsequent computation of the one-way function OW. This can be held independently for each such pair of embedding EM and subsequent one-way function OW computation.

[0443] Furthermore, the updated direct or derived output of the one-way function OW, using the corresponding publicly available information source IS as input, can be repeatedly embedded into the first main video stream PV, such as at least once every ten seconds, at least during the authentication video portion AP of the first main video stream PV; and / or such embedding can occur at least once before the authentication step during the authentication video portion AP of the first main video stream PV. The corresponding situation can be applied to the fourth main video stream PV and its authentication video portion. It should be understood that... Figure 13 This aspect, as well as any other compatibility aspect, also applies. Figure 11 The first main video stream PV, and vice versa.

[0444] Furthermore, information can be repeatedly extracted from the first main video stream (PV) (e.g., at least once every ten seconds), and the direct or derived output of a one-way function OW, using said extracted information as direct or derived input, can be publicly released to a public distribution channel (PC) via a PP. This extraction can be performed at least once after the certified video portion (AP), followed by the release of the PP. The corresponding scenario applies to the fourth main video stream (PV).

[0445] It is possible that the first and fourth main video streams are "interleaved" at certain times without using updated inputs from the publicly available information source IS and / or without public release PP. Updated inputs and public releases may be used less frequently than the "interleaving" step, and / or during periods of slow or absent internet connectivity. This can be true for any single "interleaved" video stream and any pair of "interleaved" video streams. In this way, the security enhancement process, particularly authentication continuity, can be implemented using local computing on one or more locally deployed devices, such as on one or more locally interconnected clients 121, each with a corresponding digital camera for capturing the main video streams involved, without sending or receiving any information over the internet.

[0446] Figure 11 and Figure 13 The extraction, embedding (EM), sampling (SA), and publishing (PP) shown do not need to be synchronized in any particular way, as long as they are performed in such a way that they depend on each other in the manner generally described above to achieve the “interweaving” effect discussed.

[0447] Also note that all extraction, embedding EM, sampling SA, and publishing PP can occur in real time, such that extracting information from the video stream uses the video stream in its current state (e.g., using the most recent, such as the most recently captured video frame); embedding EM occurs relative to the most recent (e.g., the most recently captured) video frame currently being considered; sampling SA is a sampling of the most recent accumulated state of the publicly available information source; and / or publishing PP occurs immediately.

[0448] Therefore, the first main video stream PV and the fourth main video stream PV are closely "intertwined", and they are also "intertwined" with both the public information source IS and the public release channel PC that evolve over time.

[0449] In this way, the verification of the authenticity of the first user 122, who is filmed from two different concurrently used viewpoints, is improved using a video that is “interwoven” together. Similarly, any identification of individuals, objects, events, etc., can be made with greater confidence because the viewer who generates the video stream can believe (and verify) that the information content of the video stream is accurate.

[0450] Accordingly, when using two such “intertwined” main video streams PV from the same location but not showing the first user 122 being authenticated, a very reliable and verifiable record of events at that location will be provided, even if the main video streams PV are not captured to depict the same thing from different viewpoints, but rather they can depict different things.

[0451] In some embodiments, the first main video stream and the fourth main video stream are recorded not only from different viewpoints, but also from one or more different angles; using different hardware, such as camera hardware; using different software, such as camera software; and / or operated by different entities, such as the first main video stream PV being captured by a mobile phone camera, while the fourth main video stream PV is captured by a corporate conference camera. This further increases the reliability of the information shown in the generated video streams.

[0452] It should be understood that there may also be at least three or even more cameras, each showing the same user 122, location, venue, etc. from a different viewpoint in a manner corresponding to that described above for the first and second main video streams.

[0453] As described above, the present invention also relates to system 100 itself. As described above, one or more central servers 130 and one or more clients 121 are configured to perform the above-described method in order to generate a video stream.

[0454] Furthermore, the present invention also relates to a computer program product for generating a video stream. This computer program product includes instructions configured to perform the steps of the method described herein when executed in system 100. This execution can be performed on one or more central servers 130 and / or one or more clients 121 as described herein.

[0455] The current authentication status, such as information about the authentication type and result of the first user 122, combined with the currently updated continuity score and / or credibility metric discussed above, or a piece of information representing an updated authentication status calculated based on the continuity score and / or credibility metric, can be associated with the first user 122 and stored and processed, for example, in the form of metadata as described above. This current authentication status can then be used by two or more users 122 of the video communication service 110 to reach an agreement on a particular piece of information occurring in the main video stream or the generated video stream. In one example, system 100 includes user controls, such as user controls in the user interface of client 121, allowing one or more users to select the current status of the video communication service 110 to constitute a certain type of "agreement." System 100 can be configured to store metadata about the currently updated authentication status of the involved user 122 along with the fact that such an "agreement" has been created for secure retrospective verification by relevant parties. In other embodiments, system 100 is configured to disallow the creation of such an "agreement" unless one or more relevant users 122 have a current authentication status within a predetermined range. In some embodiments, creating such an "agreement" will cause system 100 to trigger a re-authentication of the type described above as a prerequisite for allowing the creation of the "agreement." The "agreement" itself may be stored in the form of an original quality video stream and / or a shadow video stream of the type described above.

[0456] The general principles of authenticating the first user 122 in the authentication video portion of the main video can be applied to other scenarios different from those provided in the examples above. For example, a user wishing to prove he or she is the author of a message (such as an email) or document can record a main video stream in which the email and / or document is composed and / or sent, wherein the composition or sending is shown in the same authentication video portion as the user's authentication, which is of the type described in the general description above. The main video can then be used to retrospectively demonstrate the legitimacy of the message or document, particularly where the main video stream is "interwoven" as described above, and more particularly, where two "interwoven" main video streams from different cameras are used simultaneously to capture the composition or sending, also as described above. It should be understood that, for this embodiment, the main video stream or the generated video stream does not necessarily need to be provided to any user, and in particular, does not need to be provided in association with the generation or sending of emails, etc. Instead, an automated verification process can be used to verify the accuracy of the authentication of the composing and / or sending user.

[0457] In general, the various types of authentication for the first user 122 described herein can be performed and implemented in parallel for two or more concurrent users within the same video communication service 110.

[0458] It should be understood that references to “first,” “second,” “third,” “fourth,” etc., are merely labels unless any particular order is clear from the context.

[0459] Preferred embodiments have been described above. However, it will be apparent to those skilled in the art that many modifications can be made to the disclosed embodiments without departing from the basic concept of the invention.

[0460] For example, many additional functionalities may be provided as part of the system 100 described herein, and these additional functionalities are not described herein. Overall, the solution currently described provides a framework upon which detailed functionalities and features can be built to meet a variety of different specific applications that use video data streams for communication.

[0461] In general, everything described herein applies to the methods, systems, and computer program products, even if not explicitly mentioned.

[0462] For example, intentional delays can be introduced into one or more main video streams to create time for performing high-security one-way function calculations of the aforementioned general description type, and / or for calculating and processing metadata of the aforementioned general description type, and / or for enhancing the video quality of shadow video streams used as main video streams as described above.

[0463] Embedded EMs, sampled SAs, and / or publicly released PPs can be used in combination to retrospectively calculate the latency or delay used by a particular video stream relative to other video streams in the same video communication service 110 by examining the associated stored timestamps. For example, each embedded EM is associated with a specific portion of the video stream in question, which inherently has a timestamp along the video stream timeline. Each sampled SA is performed at a specific time, and the timestamp of the sampled SA is preferably recorded and stored as part of the sampled SA itself and / or as metadata. Each publicly released PP will inherently have a timestamp as part of the publicly released channel PH used.

[0464] This specification is based on the term "video" as referring to 2D video that includes 2D image frames. However, it should be understood that the same or corresponding principles can be applied to 3D video that includes 3D image frames.

[0465] In general, metadata about one or more main video streams and / or one or more generated video streams can be identified, stored, and / or “interleaved” into one or more of the streams, as generally described above. This metadata can include many different types of information relating to the video stream itself, the context in which the video stream occurs, the attributes of the users participating in the video stream, and so on. In some examples, the identity of the camera or client device capturing a particular main video stream, such as its IP address, MAC address, hardware identity or fingerprint, and updated geolocation information (such as GPS coordinates) of the camera or client device, can be stored / used / interleaved as metadata about the main video. Accordingly, metadata information about the identity of the central server or device that performs the automatic generation of the generated video streams can be stored / used / interleaved in the general manner described above.

[0466] Therefore, the present invention is not limited to the described embodiments, but may vary within the scope of the appended claims.

Claims

1. A method for generating a generated video stream (PR), the method comprising the following steps: a) Capture the first and second main video streams (PVs) visually representing the first user (122, 122''); b) The generated video stream (PR) is automatically generated based on the first main video stream (PV) and the second main video stream (PV). The automatic generation is based on an automatic generation decision, which in turn is based on at least one of the following: defined parameters, automatic image processing of the first main video stream (PV) and / or the second main video stream (PV), and automatic audio processing of the first main video stream (PV) and / or the second main video stream (PV). c) Downsample the first main video stream (PV) and the second video stream (PV) to obtain the first shadow main video stream and the second shadow main video stream; d) Automatically generate a shadow video stream based on the first shadow master video stream and the second shadow master video stream, automatically generating a shadow video stream based on the corresponding automatic generation decision used to generate the generated video stream (PR); as well as e) Store the first shadow main video stream, the second shadow main video stream, the generated shadow video stream, and information identifying the automatic generation decision.

2. The method according to claim 1, further comprising: The output of the second one-way function (OW) is computed using the second primary video stream (PV) as either direct or derived input. For a message representing the direct or derived output of the second one-way function (OW), perform at least one of the following: The visual and / or audible representation of the information is embedded into the generated video stream (PR); Store the aforementioned message; and The information described in the public release (PP) is as follows.

3. The method according to claim 1 or 2, further comprising: Provide at least one of the following to the first user (122, 122''): The generated video stream (PR); or Based on the first main video stream (PV) and the second main video stream (PV), different video streams are automatically generated. The automatic generation is based on an automatic generation decision, which in turn is based on at least one of the following: defined parameters, automatic image processing of the first main video stream (PV) and / or the second main video stream (PV), and automatic audio processing of the first main video stream (PV) and / or the second main video stream (PV).

4. The method according to claim 3, wherein, Providing the corresponding generated video streams (PRs) to the first user (122, 122'') and the second user (122; 122''', 122'''') is carried out in the context of an interactive video communication with the first user (122, 122'') and the second user (122; 122''', 122'''') as participants, and forms part of the interactive video communication.

5. The method according to any one of the preceding claims further comprises: Authentication is performed on the first user (122, 122''), where, The first primary video stream (PV) includes a first primary authentication video portion (AP), which visually illustrates the authentication steps and / or includes information that visually and / or audibly represents the first authentication token generated by the authentication steps, and wherein, The method further includes using the first primary certified video portion (AP) as a direct or derived input to compute the output of the first one-way function (OW).

6. The method according to claim 5, wherein, The authentication process includes capturing images of both the first user (122, 122'') and the camera used to capture the first main video stream (PV).

7. The method according to claim 5 or 6, wherein, Authentication includes providing a one-time password to the first user (122, 122''), and the first user (122, 122'') provides the one-time password audibly and / or visually, such that it forms part of the first master authentication video portion (AP).

8. The method according to any one of claims 5 to 7, wherein, The first primary authentication video portion (AP) is a part of the first primary video stream (PV), and it is located before the portion of the primary video stream (PV) used to automatically generate the generated video stream (PR).

9. The method according to any one of claims 5 to 8, further comprising: A set of metadata is associated with and stored in relation to the first primary video stream (PV) and / or the generated video stream (PR), the set of metadata including information of at least one of the following: The first authentication token; and The current authentication status of the first user (122, 122'').

10. The method according to any one of claims 5 to 9, further comprising: Continuity monitoring is performed at least based on automatic image and / or audio processing of the first main video stream (PV) to determine the extent to which the first user (122, 122'') has been continuously visible and / or audible in the first main video stream (PV) since the authentication step.

11. The method according to claim 10, wherein, The continuous monitoring is performed in a central server based on the first primary video stream (PV), which is provided to the central server from a hardware entity in its raw or pre-processed format. The hardware entity is decoupled from the central server and configured to continuously capture the first primary video stream (PV) using a camera; and / or The continuous monitoring is performed by the hardware entity.

12. The method of claim 11, further comprising: Detect interruptions in the communication connection between the central server and the hardware entity; As a result of the detection, the continuous monitoring was switched from being performed by the central server to being performed by the hardware entity. The communication connection was re-established; as well as As a result of regaining the communication connection, the continuous monitoring is switched from being performed by the hardware entity back to being performed by the central server.

13. The method of claim 12, further comprising: Detect interruptions in the communication connection between the central server and the hardware entity; As a result of the detection of the interruption, the continuous monitoring performed by the central server is suspended. The detection process confirms the re-establishment of the communication connection. As a result of the re-acquired detection, the central server catches up with the reception of the main video stream (PV); The central server retrospectively performs the continuous monitoring on the caught-up portion of the first main video stream (PV); and The central server continues to perform the continuous monitoring of the main video stream provided to the central server.

14. The method according to any one of claims 10 to 13, further comprising: A continuity score is calculated based at least on the automatic image processing and / or audio processing of the first main video stream (PV) to determine the continuity score as a credibility measure of whether the first user (122, 122'') has been continuously visible in the first main video stream (PV) since the self-authentication step, or to determine the continuity score based on the credibility measure. as well as If the continuity score falls within a predetermined range, repeat steps a and b to generate an updated first master authentication video portion (AP), and then use the updated first master authentication video portion (AP) as the first master authentication video portion (AP) to repeat step c.

15. The method according to claim 14, wherein, The continuity score is determined based on at least one of the following: The visual appearance of the first user (122, 122''); The voice fingerprint of the first user (122, 122''); The activity level of the first user (122, 122''); The current meeting role of the first user (122, 122''); The interaction between the first user (122, 122'') and the second or third user (122; 122''', 122''''), particularly the confirmation of the authenticity of the first user (122, 122'') by the second or third user (122; 122''', 122''''), and the second or third user (122; 122''', 122'''') is also visible in the first main video stream (PV), the second main video stream (PV), or the third main video stream (PV); The video quality and / or continuity of the first main video stream (PV); This shows the change in the main video stream (PV) of the first user (122, 122''); as well as The time elapsed since the first certified video portion (AP) is measured in terms of the timeline of the first main video stream (PV).

16. The method according to claim 14 or 15, further comprising: The first user (122, 122'') and the third user (122; 122''', 122'''') were both visible in the first mainstream; and As a result of detecting that both the first user and the third user (122) are visible, and optionally as a result of detecting that the third user (122; 122''', 122'''') actively confirms that the first user (122, 122'') is real, the continuity score for the first user (122, 122'') is increased.

17. The method according to any one of claims 14 to 16, further comprising: The authentication performed when step a is repeated is a different type of authentication than the authentication performed in the previous iterations of step a.

18. The method according to any one of claims 14 to 17, further comprising: The continuity score is represented in the generated video stream (PR).

19. The method according to any one of claims 5 to 18, further comprising: The publicly available information source (IS) is sampled (SA), and the sampled information is used as input to compute the output of the first source one-way function (OW); The output of the first joint one-way function (OW) is calculated using the direct or derived output of the first source one-way function (OW) and the direct or derived output of the first one-way function (OW) as inputs. as well as Publicly release (PP) the direct or derived output of the first joint one-way function (OW).

20. The method according to claim 19, wherein, The output of the first joint one-way function (OW) is iteratively calculated using the direct update output or derived update output of the second one-way function (OW) as input. The second one-way function (OW) is calculated using the second main video stream (PV) as direct input or derived input.

21. The method according to claim 19 or 20, wherein, The output of the first joint one-way function (OW) is iteratively calculated using the direct or derived update output of the first source one-way function, the direct or derived update output of the first one-way function (OW), and the direct or derived output of the first joint one-way function (OW) of the previous iteration as input.

22. The method according to any one of claims 19 to 21, wherein, The output of the first joint one-way function (OW) is also computed using at least one of the following as direct or derived inputs: The first primary authentication video portion (AP); The first authentication token; Updated continuity score; The output of a third one-way function (OW), which is computed using the generated video stream (PR) as a direct or derived input; and A set of metadata describing the complete set of generation steps for generating the generated video stream (PR) based on the first primary video stream (PV) and the second video stream.

23. The method according to any one of claims 19 to 22, further comprising: A piece of information representing the direct or derived output of the first joint one-way function (OW) is visually and / or audibly embedded into at least one of the following: The first main video stream (PV) prior to being used to generate the generated video stream (PR); The second main video stream (PV) prior to its use in generating the generated video stream (PR); and The generated video stream (PR).

24. The method according to claim 23, wherein, For different portions or frames of the first main video stream (PV), the second main video stream (PV), and the generated video stream (PR), the calculation of the output of the first joint one-way function (OW) is repeatedly performed over time, and the information is embedded into at least one of the first main video stream (PV), the second main video stream (PV), and the generated video stream (PR).

25. The method according to any one of the preceding claims, further comprising: Metadata about the automated generation steps performed to generate the generated shadow video stream is embedded into the generated shadow video stream.

26. The method according to any one of the preceding claims, wherein, The generated shadow video stream is not shown to the first user (122, 122'') or the second user (122; 122''', 122'''').

27. The method according to any one of the preceding claims, wherein, The downsampling is configured to reduce the byte size by at least 10 times.

28. The method according to any one of the preceding claims, wherein, The downsampling includes at least one of the following: Compared to the main video stream (PV) in question, the pixel resolution of the shadow video stream in question is reduced uniformly or selectively on the image plane. Compared to the main video stream (PV) in question, the pixel depth of the shadow video stream in question is reduced uniformly or selectively on the image plane. The frame rate of the shadow video stream in question is reduced compared to the main video stream (PV) in question; as well as Compared to the main video stream (PV) in question, a cropping of the shadow video stream in question is added.

29. The method according to claim 28, wherein, The downsampling is performed without removing more than the amount of information from the main video stream (PV) such that the presence and identity of the user (122) visible in the main video stream (PV) can also be visually identified in the shadow video stream.

30. The method according to any one of the preceding claims, wherein, The downsampling of the first main video stream (PV) and / or the second main video stream (PV) is dynamic in the sense that it takes into account one or more parameter values ​​of the automatically generated decision, such that the downsampling is applied differently over time when viewed along the timeline of the main video stream (PV) in question.

31. The method according to any one of the preceding claims further comprises: At least a portion of at least one of the first main video stream (PV), the second main video stream (PV), and the generated video stream (PR) is discarded, the discarding occurring after the generated video stream (PR) is provided to the first user (122, 122'').

32. The method according to any one of the preceding claims, wherein, The generation of the corresponding generated shadow video stream occurs with a time delay of up to 10 seconds relative to the generation of the generated video stream (PR).

33. The method according to any one of the preceding claims further comprises: The output of the first shadow one-way function (OW) is calculated using the first shadow master video stream as a direct or derived input, and the output of the second shadow one-way function (OW) is calculated using the second shadow master video stream as a direct or derived input. The output of the first shadow one-way function (OW) is calculated using the first shadow authentication video portion (AP) of the first shadow master video stream as a direct or derived input. The first shadow authentication video portion (AP) corresponds to the first master authentication video portion (AP), which shows the authentication steps and / or represents the information of the token. as well as For each corresponding piece of information representing the direct or derived output of the first shadow one-way function (OW) and the second shadow one-way function (OW), perform at least one of the following: The visual and / or audible representation of the aforementioned information is embedded into the generated shadow video stream; Store the aforementioned message; as well as The information described in the public release (PP) is as follows.

34. The method of claim 33, further comprising: The publicly available information source (IS) is sampled (SA), and the sampled information is used as input to compute the output of the second source one-way function (OW); The output of the second joint one-way function (OW) is calculated using the direct or derived output of the second source one-way function (OW), the direct or derived output of the first shadow one-way function (OW), and the direct or derived output of the second shadow one-way function (OW) as inputs. as well as Publicly release (PP) the direct or derived output of the second joint one-way function (OW).

35. The method according to any one of the preceding claims, further comprising: Embed the direct or derived output of the first shadow master one-way function (OW) into the first master video stream (PV); and / or The direct or derived output of the first one-way function (OW) is embedded into the first shadow main video stream.

36. The method according to any one of the preceding claims further comprises: Automatically convert the first shadow main video stream, the second shadow main video stream, and / or the generated shadow video stream to obtain a converted video stream; as well as The converted video stream is provided to the user (122, 150), wherein... The transformation includes at least one of the following: Increase audio bitrate, pixel resolution, pixel depth, and / or frame rate; Introducing de-cropping, image elements, and / or audio elements; and Animate the participants shown in the converted video stream, either partially or completely. The transformation is based on previously stored metadata.

37. The method of claim 36, further comprising: The metadata is determined based on at least one of the first primary video stream (PV), the second primary video stream (PV), and the generated video stream (PR).

38. The method according to claim 36 or 37, wherein, The metadata includes information provided externally.

39. The method according to any one of claims 36 to 38, wherein, The downsampling is irreversible.

40. The method according to any one of the preceding claims further comprises: Capture a fourth main video stream (PV) visually representing the first user (122, 122'') from a viewpoint different from the first main video stream (PV), wherein the fourth main video stream (PV) includes a fourth main authentication video portion (AP) that visually illustrates the authentication steps and / or includes information visually and / or audibly representing the token; and The output of the fourth one-way function (OW) is computed using the fourth primary certified video portion (AP) as a direct or derived input.

41. A system for generating a generated video stream (PR), the system comprising a central server and a client, the system being configured to perform the method according to any one of the preceding claims.

42. A computer program product for generating a generated video stream (PR), the computer program product comprising instructions configured to perform the method according to any one of claims 1 to 40 when executed.

Images