Computer-implemented method for transmitting and displaying at least video data

Abstract

The invention relates to a computer-implemented method for transmitting and displaying at least video data using streaming technology, comprising: - by means of at least one AI program (21-24, 32; 25), detecting and evaluating real events that in data streams (11-17) including at least video data streams by a plurality of data sources (1-7); - by means of at least one AI program (21-24, 32; 25), generating and assigning instructions (A) to one or more of the data streams (11-17) generated by the data sources (1-7); - combining at least a plurality of the data streams (11-17) to form a final individual stream (39) by means of an A / V mixer (38), implemented as software, of a conference server (30) taking into account instructions (A) assigned to the respective data streams (11-17); - sending the final individual stream (39) from the conference server (30) to a plurality of subscriber terminals (52-54).

Description

[0001] Computer-implemented method for transmitting and displaying at least video data

[0002] The invention relates to a computer-implemented method for transmitting and displaying at least video data using a streaming technology and a corresponding system.

[0003] Various methods are known for displaying multiple data streams, particularly video data streams, but also audio data streams and other data streams, in a bundled format on playback devices. For example, live images recorded by various cameras can be displayed clearly on a single screen, e.g., in a surveillance service control center or a fire department control center. If the operator notices something unusual, they can select the camera of interest and have its transmitted images displayed enlarged on their screen.

[0004] However, this procedure is very much tailored to the individual operator and therefore requires experience and increased attention.

[0005] It is an object of the present invention to provide a method and a corresponding system with which the transmission and in particular the representation of data streams can be realized in a more variable, useful and effective manner.

[0006] This problem is solved in the present case by the subject-matter of the independent claims.

[0007] The invention utilizes single-stream technology, as described in particular in EP 2 498 491 A1 by the applicant. Here, all video, audio, and other data streams (e.g., measurement data) from a software-implemented conference server or from an A / V mixer in the conference server are combined into a single stream and transmitted in this form to all participant terminals. After combining the individual video, audio, and other data streams into the final single stream, which is then transmitted to the participant terminals, the display of this single stream on the screens (and the playback of its audio components via the audio outputs) of the participant terminals is unchangeable and is therefore displayed in the same way to all participants.All participants therefore receive the same information, with all data being transmitted in real time or near real time (delay of a maximum of a few seconds).

[0008] The invention enables event-driven playback of data streams on subscriber terminals, whereby the data streams from the individual data sources are analyzed before assembling the final individual stream of AI programs. AI programs are software programs based on artificial intelligence that recognizes and classifies information from input data (in this case, these are the data streams). The artificial intelligence can be based on programmed processes and / or generated through machine learning.

[0009] When reference is made to the AI ​​program below, this term also includes the associated interfaces, which are also implemented as software. Data streams are therefore routed via a corresponding software interface to the AI ​​software for the purpose of detecting and evaluating events. The resulting instructions are assigned, preferably again via this interface, to the corresponding data streams according to the event analysis and transmitted to the conference server to combine this data stream with the other data streams to form the final single stream.

[0010] The term "detection of real events" involves the analysis of the corresponding data stream by at least one AI program, particularly for predetermined deviations, i.e., anomalies compared to the usual processes, e.g., fire outbreaks. However, data streams can also be analyzed continuously without deviations from the usual occurring, e.g., the detection of traffic volume, including the detection that no special traffic events (accidents, traffic jams, etc.) are occurring. The continuous measurement of temperatures with a sensor, for example, or the examination of social media feeds, is therefore also considered "detection of real events."With regard to data sources designed as audio sources, “real-world event detection” is, for example, the detection and analysis of an unusual audio event, such as a scream, a car horn, or a detonation, but also the regular or continuous detection of common sounds in an audio stream.

[0011] The term "assigning instructions to one or more data streams" can be used in a variety of ways. Depending on the analysis of the corresponding AI program, specific instructions can be assigned to each data stream. These instructions can be embedded as digital information in the respective data stream and then sent with it to the conference server, which uses the instructions to determine how the corresponding data stream is to be integrated into the final individual stream. Alternatively (or additionally), the instructions can be sent separately, i.e., independently of the individual streams, to the conference server, which then assigns the instructions to the corresponding data streams based on specific characteristics (e.g., numbering the data sources with corresponding numbering of the instructions).

[0012] The instructions can be generated in a variety of forms, for example, as pure commands, but also as metadata, i.e., structured data with information about the characteristics of other data. The generation and assignment of instructions can be performed by the same AI program that also detects and evaluates the event; alternatively, these tasks can be performed by one or more other downstream AI programs.

[0013] According to the invention, real events recorded by means of a plurality of data sources in the form of data streams, including at least video data streams and preferably also audio data streams and / or other data streams, are recognized by at least one AI program. If, for example, an unusual event is recorded by a camera, such as a camera-monitored gate being broken open or a fire breaking out in a camera-monitored area, the AI ​​program recognizes this event based on its analysis and evaluates it as relevant. Accordingly, an instruction relating to the event is assigned to this video data stream from the relevant data source, i.e. the relevant camera. One such instruction is, for example, that the relevant data stream be displayed larger and / or in a central position and / or, for example, with a conspicuous border on the screens of the subscriber terminals.The conference server or an A / V mixer within the conference server then combines the various data streams, taking into account the instructions from these and any other data sources, to create the final individual stream, which is then transmitted to the participant terminals. The inventive playback on the participant terminals draws the participants' attention directly to the event in question. Information and / or commands can then be passed on to other people more quickly and precisely, for example to firefighters on site in the event of a fire. Decision-making and the entire decision-making process can thus be significantly improved and accelerated. It is also advantageous if the final individual stream includes optical and / or acoustic signals to notify specific people (e.g. firefighters) via their participant terminals.Alternatively or additionally, the final individual current can trigger actions such as vibration of appropriately equipped subscriber terminals or the triggering of external alarm devices that receive the final individual current.

[0014] Said events may, in particular, include visual events and / or acoustic events and / or other real events based on other data. The data sources are preferably configured to collect and transmit one or more of the following data:

[0015] - Camera recordings, for example from stationary and / or mobile cameras;

[0016] - surveillance videos;

[0017] - Live drone footage;

[0018] - Real-time bodycam footage;

[0019] - Sensor data, such as temperature data, speed data, direction data, etc.;

[0020] - Satellite communication data, such as location data;

[0021] - Tracking data;

[0022] - tactical communications data;

[0023] - geographical information data, for example elevation profiles - consumption data, for example energy consumption data;

[0024] - Data resulting from electronic games;

[0025] - Platform streaming data;

[0026] - Social media feeds.

[0027] Data sources designed as sensors can be of various types, for example, IoT sensors. Sensors, also known as detectors or measuring transducers, can detect certain physical or chemical properties and / or the material properties of the environment qualitatively or quantitatively as a measurand and, in particular, convert them into a further processable electrical signal.

[0028] The said instructions, which are generated by the at least one Kl program from the data stream analysis and assigned to one or more data streams, preferably originate from the following group, which is not to be understood as exhaustive:

[0029] Instruction that a relevant event has occurred which should be highlighted on the screens;

[0030] Instruction that there is no relevant event or only a slightly relevant event and therefore should not be highlighted on the screens;

[0031] Instruction that additional elements in at least one data stream are to be displayed on the screens when the final individual stream is displayed;

[0032] Instruction that individual elements from at least one data stream are to be hidden when the final individual stream is displayed on the screens;

[0033] Instruction that no relevant or only a slightly relevant event is present in at least one data stream and that this at least one data stream is not to be incorporated into the final individual stream;

[0034] Instruction that one or more audio streams from the subscriber terminals are output louder, more attenuated, or muted compared to other audio streams.

[0035] Through instructions of the aforementioned type, the user of the subscriber terminals is presented with a weighted representation of the detected event(s). If the event is a forest fire, for example, the at least one AI program can prominently display the data streams from all cameras capturing the fire on the screens, particularly preferably in different sizes and / or with intelligent placement, depending on the weighting regarding the significance of the images recorded by the respective camera and evaluated by the AI. A temporal component can also be included here, for example, if certain fire sources change significantly within a certain period of time. In these cases, it is possible, for example, to highlight the camera recordings from the corresponding camera data source(s) more clearly than those from camera data sources that tend to record an unchanged fire progression at other locations.

[0036] The above example also demonstrates that the invention is particularly suitable for dynamically adapting the playback of data streams. Continuously adapting the playback over time ensures maximum timeliness and, if necessary, enables rapid adjustment of decision-making.

[0037] According to an advantageous embodiment, the data streams from some or all data sources are first received by the at least one Kl program before these data streams and the instructions generated by the at least one Kl program are combined into a single stream by the conference server or by an A / V mixer as part of the conference server.

[0038] Each data source can be assigned a single AI program that analyzes the data stream from that data source with respect to real-world events, evaluates it, and, if necessary, generates one or more instructions. However, it is also possible for a single AI program to analyze multiple data streams from different data sources, evaluate them, and, if necessary, generate one or more instructions.

[0039] According to a similarly advantageous alternative, the conference server receives data streams from the majority of data sources without the interposition of at least one AI program and combines them into a first individual stream using an A / V mixer. Only then is this first individual stream sent to the at least one AI program for the aforementioned detection and evaluation of events and the generation and assignment of the aforementioned instructions. Based on these instructions, the final individual stream is then assembled and sent to the participant terminals.

[0040] In a particularly preferred embodiment, the conference server is instructed to continue receiving the data streams from the data sources and combine them into the final single stream, taking the assigned instructions into account. This way, the final single stream always contains the most up-to-date data streams.

[0041] The at least one AI program can be integrated into the conference server, separated from it, or run independently. The AI ​​program(s) and conference server are implemented as software.

[0042] The two aforementioned embodiments—on the one hand, the evaluation of data streams using AI analysis before combining them into a single stream, and on the other hand, combining data streams into a first single stream followed by AI analysis—can also be implemented in a single embodiment. In this case, one or more data streams are first analyzed by one or more AI programs, i.e., examined for real events and evaluated, while the information in other data streams is analyzed and evaluated only after being integrated into a first single stream.

[0043] Preferably, at least one AI program is provided that evaluates and assesses events in different data streams in relation to one another. This at least one AI program can be connected downstream of other AI programs that detect the events in individual data streams. If, for example, two cameras record the same event and both events are detected or registered by a respective AI program, the corresponding information can be sent to a downstream AI program, which then decides whether one or the other camera is currently providing the more important information. The corresponding weighting analysis as well as the evaluation and assignment of instructions for the purpose of appropriately adapted screen display is therefore carried out by this downstream AI program.

[0044] As already described above, it is advantageous if – depending on the assigned instructions – the display of the data streams combined in the final individual stream on the screens of the participant devices, initiated by the conference server, changes over time, i.e., a dynamic adaptation of the playback occurs. For example, if at least one audio program detects temporal changes in the recorded events in one or more data streams, one or more corresponding data streams can be displayed more clearly (larger and / or more centrally positioned and / or acoustically and / or visually highlighted differently), while one or more other data streams can be displayed less conspicuously or not integrated into the final individual stream at all.

[0045] According to a further development of the invention, it is advantageous if the conference server is designed to receive command inputs from authorized persons regarding the display of individual streams on the screens of the participant terminals and to process them accordingly in order to combine the final individual stream from the data streams accordingly. This gives operators the option of individually influencing the composition and thus the playback of the final individual stream. If, for example, an operator wishes to display a specific data stream more prominently in order to show other participants information in this data stream (e.g. recordings from a specific camera) in more detail or to draw their attention to it, they can then select the corresponding data source and influence the size and / or location and / or type of playback of the associated data stream by entering commands (e.g. via a keyboard or voice input).

[0046] Preferably, the at least one Kl program performs one or more of the following actions based on the detection of real events:

[0047] - Initiating an event-related message on one or more subscriber terminals outside of the final individual stream. For example, the at least one class program can send information about the event to one or more subscriber terminals that is only relevant to this limited group of subscribers and, if played back via the final individual stream, would only take up screen space that would then be unavailable for other information.

[0048] - Determining to which of a plurality of participant terminals the conference server will send the final individual stream. For example, the at least one KL program can cause a specific group of participant terminals to be excluded from receiving the event, for which the specific event is not of interest or which should not receive the corresponding information for other reasons. In this case, two or more individual streams with different content (i.e., from non-identical data streams) are generated, which are then sent to different groups of participant terminals.

[0049] - Initiating the control of at least one data source to modify the recording of the data stream. For example, the at least one AI program can detect that the event would be recorded more accurately by a stationary camera with a different camera angle. The AI ​​program can then initiate a control command to control this camera accordingly in order to pan the camera to a more suitable position.

[0050] - Sending said instructions to another application, implemented as software, for processing outside of the final individual stream. The instructions can be evaluated, for example, for statistical purposes or for training programs. They can also be evaluated for future improved data acquisition, for example, to supplement existing data sources with additional data sources for the purpose of improved event recording. The equipping of new objects or areas with cameras and other data sources can also be optimized using the wealth of experience gained from the instructions.

[0051] Preferably, the final individual stream is compiled in such a way that the relative spatial arrangement of the data sources is displayed in an overview on one section of the screens of the subscriber terminals, advantageously identifiable in the form of a symbol or a number. Clear identification of the individual data sources is particularly advantageous in this case. In the aforementioned overview display and the reproduction of the individual data streams (within the final individual stream) on other screen areas, it is particularly advantageous if the viewer or operator can immediately recognize which data stream belongs to which data source shown in the overview display.

[0052] Particularly preferred is the display of information from one or more data sources as text output in at least one area of ​​the screens. Sensor data, such as wind directions and wind speeds in the case of fires, GPS and / or altitude positions of drones, vehicle and / or drone speeds, etc., are particularly suitable for this purpose.

[0053] The at least one AI program can further be configured to generate and / or extract information from one or more of the data streams, which information is output on the playback devices and / or further processed by other software. Such information can also be referred to as secondary information, while the data in the data streams then constitutes the primary information. A relevant example is the analysis of a fire over time and the findings resulting from the analysis, such as fire spread forecasts, the issuance of corresponding warnings, and the provision of advice for optimal firefighting.

[0054] The invention also encompasses a system with at least one computing unit configured to carry out a computer-implemented method as described above. The system is configured for information transmission and presentation using streaming technology and has at least said computing unit, on which the conference server is implemented.

[0055] Furthermore, the invention proposes a computer program which, when executed by a computing unit, causes the computing unit to carry out the method steps of a computer-implemented video conferencing method according to one or more of the appended method claims.

[0056] The invention also proposes a computer-readable storage medium comprising a computer program which, when executed by a computing unit, causes the computing unit to carry out the method steps of a computer-implemented video conferencing method according to one or more of the appended method claims.

[0057] Advantageous developments of the invention are characterized by the features of the subclaims.

[0058] The invention is explained in more detail below with reference to the accompanying figures. Figure 1 shows a schematic overview of the devices involved in transmitting and displaying information according to a computer-implemented method;

[0059] Fig. 2 the overall system of Fig. 1 with additional information transmission options;

[0060] Fig. 3 is a schematic overview of an alternative embodiment of the devices for transmitting and displaying information in a computer-implemented method;

[0061] Fig. 4a, b a screen of a subscriber terminal with different arrangements of data streams from different data sources, and

[0062] Fig. 5 shows the screen of a subscriber terminal with another alternative arrangement of data streams from different data sources.

[0063] In the figures described below, the use of identical reference symbols indicates identical or comparable features or functionalities. If reference symbols are not explicitly mentioned or explained for a figure, reference is hereby made to their description and explanation in other figures.

[0064] Fig. 1 shows a schematic overview of the basic idea of ​​the invention. Several data sources 1 - 6 (also designated by "Q" in Fig. 1 for easier orientation) each generate a data stream 11 - 16. In Fig. 1, only six sources 1 - 6 with their generated data streams 11 - 16 are shown as examples; considerably more data sources (indicated by the dots between data source 6 and the lowest data source Q), but also fewer, can easily be included. The data sources 1 - 6 can be of very different nature. Preferably, at least some of the data sources 1 - 6 are designed as cameras, which can be stationary cameras such as surveillance cameras, cameras installed on drones or other flying objects or mobile bodies, body cameras (so-called bodycams), etc.Other preferred data sources are microphones and sensors for recording a wide variety of parameters such as speed, GPS locations, temperatures, brightness, geometric dimensions, etc. Sensors typically provide data in the form of numbers or symbols (such as colors, arrows, etc.) or even words. Microphones generate audio data, also known as voice data. Cameras produce video data, i.e., moving image data, or still images.

[0065] The data sources, i.e. data sources 1-6 in Fig. 1, are designed to record real events, such as fires, criminal activities, anomalies of any kind, noises, voices, sensor data, etc.

[0066] According to the embodiment shown in Fig. 1, the data streams 11-16 are sent to several AI programs 21-24, i.e., software programs based on artificial intelligence. These programs recognize, sort, and process information from the data streams 11-16 of the data sources 1-6. Typically, the AI ​​programs are self-optimizing software routines that evolve through machine learning.

[0067] The Kl programs 21-24 each contain a correspondingly designed Kl interface (not separately shown here). The respective Kl interface receives the data stream, forwards it to the actual Kl processing software, and transmits the results of the Kl processing. The Kl interface and the actual Kl processing software are presently components of a respective Kl program 21-24. Fig. 1 shows by way of example that not all data sources 1-6 send their data streams 11-16 to an associated Kl program. For example, multiple data streams, here the data streams 14-16 from the data sources 4-6, can be transmitted to a Kl program 24, which processes these three data streams 14-16.

[0068] According to the invention, the Kl programs 21-24 are designed to examine the data streams 11-16 and to recognize real events recorded by the data sources 1-6 in the data streams 11-16. In many cases, this is understood to mean, in particular, the identification of certain real occurrences that deviate from a usual course of events. For example, if a forest area is monitored for fire outbreaks using data sources designed as cameras, e.g. data sources 1-4, and the various cameras detect abnormal brightnesses that also change over a short period of time, this can indicate a fire. The Kl programs 21-24 recognize these changes in brightness and evaluate them. As a result of such an evaluation, e.g. "fire identified" or "fire spreading to the north", the corresponding Kl programs 21-24 generate instructions A, i.e. digital commands, such as the playback of the respective data stream 1-4 on screens or.generally on output devices - in relation to the playback of other data streams (see below).

[0069] For illustrative purposes, in Fig. 1 the Kl programs 21-24 are marked as rectangles with the abbreviation “Kl: +A” to clarify that the Kl programs 21-24 generate the instructions A and are assigned to the data streams 1-6 according to the embodiment of Fig. 1.

[0070] The aforementioned instructions A are, for example, appended to the respective data stream 11-16 by the associated interfaces in the Kl programs 21-24 or forwarded independently of the respective data stream 11-16, but assignable to it. Figure 1 shows the former case, with these data streams 11-16 and the instructions A assigned to them being designated in Figure 1 by the reference symbols 11+A to 16+A.

[0071] Figure 1 shows another data source 7, which is designed, for example, as a sensor and generates a data stream 17 consisting of measurement data from the sensor. A KL evaluation is not performed in this exemplary embodiment.

[0072] After analyzing and evaluating data streams 1-6 and generating and assigning the corresponding instructions A, data streams 11-16 with the associated instructions A are transmitted to a conference server 30. Data stream 17 from data source 7 is also sent to conference server 30. In the present embodiment, an input-side AI program 32 of conference server 30, including a corresponding AI interface, receives data streams 11-17 with, if applicable, the respective associated instructions A, and processes these instructions A. In addition, it is preferred that AI program 32 also evaluates data streams 11-17 and / or instructions A relative to one another and generates further instructions A, for example to establish a weighting of data streams 11-17 relative to one another.For example, the Kl program 32 can determine that the data stream 11 contains more significant information than the data stream 12 and should therefore be displayed accordingly prominently in terms of size, etc., on output devices.

[0073] Based on all of the aforementioned instructions A, an A / V mixer 38 (audio / video mixer) implemented as software in the conference server 30 combines the data streams 11-17 into a single stream 39. Such a single stream 39 is described, for example, in the applicant's EP 2 498 491 A1. All video, audio, and other data streams, e.g., measurement data from sensors, are combined by the A / V mixer 38 into a single stream 39 and transmitted as such to the participant terminals 52-54. Therefore, all participants receive and see the same individual streams 39 on their screens 42-44 in the same display, which is based solely on the combination of the data streams 11-16 in the conference server 30.

[0074] The invention now provides that the instructions A resulting from the AI ​​analyses and evaluations determine the location and / or size and / or type of playback of the data streams relating to the event. The individual stream 39 adapted in this way is accordingly displayed on the screens 42-44 of the subscriber terminals 52-54 shown as examples. The number of subscriber terminals including their screens is, in principle, not subject to any restrictions. If necessary, the bandwidth of the transmission of the individual stream 39 can also be adjusted by the conference server 30, preferably dynamically. For example, analysis programs can be used for this purpose which also take into account the bandwidths of possible transmission paths to the subscriber terminals and / or the corresponding reception and processing capabilities of the subscriber terminals themselves.

[0075] An instruction A generated by one of the Kl programs 21-24 may, for example, state that the associated data source 1-6 has not recorded any relevant event or has only a slightly relevant event and therefore the corresponding data stream 11-16 should not be incorporated into the final individual stream 39.

[0076] A further instruction A may, for example, consist in one or more audio streams generated by corresponding data sources 1-6 being output by the subscriber terminals 52-54 at a higher volume or at a lower volume or muted compared to other audio streams.

[0077] Examples of different display formats for various data streams on screens are discussed below. Screens 42-44 can, for example, be installed in mobile devices or in command centers or other monitoring facilities.

[0078] According to a further development of the invention, one or more or all data streams 11-16 are already processed by the respective AI program 21-24 itself and sent in this processed state to the conference server 30. For example, the AI ​​program already makes a selection as to which section of a video data stream will be forwarded to the conference server 30.

[0079] The Kl program 32 can also be provided outside the server device 30.

[0080] Fig. 2 shows the same overview as Fig. 1. In addition, further functionalities are shown, some of which will be explained below.

[0081] According to one example, a generated instruction A may include incorporating further elements into a data stream 11-16, in addition to the size, location, and type of display, which are then displayed on screens 42-44. These may, for example, be indications in the form of symbols regarding the temporal change of the event, such as the progression of the fire size over the past hour.

[0082] For example, an event-related message can be output to one or more subscriber terminals 52-54 via the Kl program 32—outside of the final individual stream 39. This is schematically illustrated in Fig. 2 by the transmission 35 to the subscriber terminal 52. Such a message can, for example, be individual information (message, notice, command, etc.) to the operator of the subscriber terminal 52. The transmission of such separate messages to the subscriber terminals 52-54 can, in principle, also be implemented by one or more of the Kl programs 21-24.

[0083] Furthermore, for example, one of the Kl programs 21-24 can determine to which subscriber terminals 52-54 the conference server 30 should send the final individual stream 39. In such a case, the conference server 30 or the A / V mixer 38 implemented therein preferably generates at least two different final individual streams 39, wherein, for example, a privileged subgroup of subscriber terminals 52-54 receives a different final individual stream 39 with partially different, preferably additional, data streams 11-17 than another subgroup of subscriber terminals 52-54, which does not require the information from these other, preferably additional, data streams 11-17 or should not receive it. The composition of the corresponding individual streams 39 and the determination of which group of subscriber terminals 52-54 receives which final individual stream 39 is determined by at least one of the Kl programs 21-24.

[0084] According to an advantageous embodiment, at least one AI program can issue instruction A to instruct at least one data source to modify the recording of the data stream. For example, the instruction can be "pan camera to the side" or "pan camera down" or "zoom in on the camera image." Other possibilities include instructions from the at least one AI program to a drone equipped with a camera to fly closer to the relevant event or to climb higher for a better overview. Another possibility is an audio output to a mobile emergency worker wearing a bodycam or a smartphone, telling them to turn to the side to capture a more suitable image. These scenarios are indicated by the transmission 25 from the AI ​​program 24 to the data source 5. The AI ​​program 32 also initiates a command for a modified recording by the data source 3.

[0085] Figure 2 also shows that the Kl program 32 communicates directly with the subscriber terminal 42 via transmission 35 and transmits, for example, a message intended only for the operator of this device. This message is sent to the subscriber terminal 42 independently of the individual stream 39.

[0086] Furthermore, Fig. 2 shows that the Kl programs 21-24 can communicate with each other (indicated by the dashed lines 29) in order to determine a weighting for the subsequent display or playback of the data streams 11-16 on the subscriber terminals 52-54 from a comparison of the data streams 11-16. The results of this comparison can then be embedded in the instructions A and transmitted to the conference server 30. For the sake of clarity, the communication 29 between Kl programs 21-24 is shown in Fig. 2 only between two Kl programs at a time; however, there are no restrictions here. All Kl programs 21-24 can also communicate with each other, for example.

[0087] According to an advantageous embodiment, the communication 29 between the Kl programs 21-24 can also replace the Kl program 32. In this case, the Kl programs 21-24 coordinate among themselves which instructions A are assigned to the data streams 11-16, which are then sent to the conference server 30 and, in particular, to the AA / mixer 38.

[0088] Alternatively, the Kl program 32 takes over all tasks of the Kl analysis, i.e., the detection of real events in the data streams 11-17, the corresponding evaluation, and the assignment of the instructions A created thereon for the display of the data streams 11-17 using the subscriber terminals 52-54. The Kl program 32 therefore takes over both the analysis of the individual data streams 11-17 and the analysis of the data streams 11-17 among themselves in order to realize an optimal, event-adapted display of the data streams 11-17.

[0089] Figure 3 shows an alternative embodiment of the invention. Here, the individual data streams are not analyzed and evaluated by at least one AI program. Rather, the data streams 11-16 are routed directly to the AA / mixer 38 of a conference server 30, which combines them into a single stream 39'. The data sources 1-6 can, in particular, be configured as cameras, microphones, sensors, etc.

[0090] This individual stream 39' is now sent to at least one AI program 25 (which in turn contains a corresponding AI interface), which detects and evaluates events in the data streams 11-16 that are integrated into the individual stream 39'. The AI ​​program 25 then passes the resulting instructions A back to the AA / mixer 38, which, based on the instructions A, compiles the final individual stream 39 from the currently incoming data streams 11-16 in order to send it to the subscriber terminals 52-54 for playback on their output devices, in particular the screens 42-44. The individual streams 39' and 39 are thus generated with a slight time delay, since the individual stream 39' is evaluated and evaluated first, and only then, with the aid of the instructions generated by the AI ​​program 25, is the individual stream 39 to be sent to the subscriber terminals 52-54 generated.

[0091] In Fig. 3, it is also provided, for example, that a data source 7 sends data or a data stream directly to the Kl program 25 without first being integrated into the individual stream 39'. The Kl program 25 can evaluate the data from this data source 7 so that it is incorporated into the creation of instructions A.

[0092] Furthermore, according to the embodiment of Fig. 3, it is provided that the Kl program 25 controls the data sources 5 and 1 by transmissions 33 in order to modify their recordings, as was also described above for Fig. 2.

[0093] Furthermore, Figure 3 shows that the Kl program communicates directly with the subscriber terminal 44 via transmission 35, for example, transmitting a message that is of interest only to the operator of this device or intended exclusively for him. This message is sent to the subscriber terminal 44 independently of the individual stream 39.

[0094] Mixed forms of Figs. 1-3 are readily possible, i.e., a Kl analysis before combining the data streams into a single stream (Figs. 1, 2) as well as an additional Kl analysis after the individual stream has been created (Fig. 3). For example, if the embodiment of Fig. 3 is used, it is advantageous to provide a Kl interface, as shown in Figs. 1 and 2 as Kl program 32, which, for example, evaluates the incoming data streams 11-16 and incorporates them into the final individual stream 39 using corresponding instructions A (see also the explanations for Figs. 1 and 2).

[0095] In Figs. 4a and 4b, different possible representations of the data streams 11 - 17 analyzed by at least one Kl program 21 - 25, which are summarized in the individual stream 39, are shown on a screen 42 of a subscriber terminal 52.

[0096] The at least one Kl program 21-24 and possibly also the Kl program 32 have detected, analyzed, and evaluated real events in the data streams 11-16 in order to then assign instructions A to some or all of the data streams for displaying the data streams 11-16 on the screens 42-44 (generally: the display devices) of the subscriber terminals 52-54. These instructions A are taken into account when incorporating the data streams 11-16 into the final individual stream 39 and relate in particular to the location and / or size and / or type of display of the data streams 11-16. For example, according to Fig. 4a, individual stream 11 is displayed significantly larger than individual stream 14, which in turn is larger than individual streams 12, 13, 15, 16.Individual stream 11 was classified as most important by the Classification Program 21 and / or the Classification Program 32 and is therefore shown in an enlarged format; to a lesser extent, this also applies to individual stream 14, which was evaluated accordingly by the Classification Program 24 and / or the Classification Program 32.

[0097] Additionally or alternatively, instructions A can also include highlighting certain data streams in color (e.g., with a colored border) or displaying them at a different location on the screen. Instructions A regarding acoustic playback (louder, softer, mute) are also possible to emphasize or fade more relevant speech or other acoustic information into the background.

[0098] For example, screen area 19 displays written information, which may include general information as well as continuously updated data from sensors such as sensor 7. This information may include temperatures, wind directions and speeds, planned next actions (introduced by a dispatcher, for example), unplanned outages of data sources, transmission channels to be used, etc.

[0099] Fig. 4b shows another possible way of displaying the individual streams 11-16 in real time. Here, the data streams 11-15 are displayed on screen 42 with essentially the same size. The data stream 16 is, for example, a continuous camera recording of a dispatcher organizing an operation, for example, for firefighting, and is visible the entire time to all participants on their screens and audible via the loudspeakers of the subscriber terminals, in particular for the transmission of operational information and commands. The screen area 19, in turn, is an information field, while the data stream 17 supplies the data from one or more sensors 7, preferably continuously and updated.

[0100] Fig. 5 shows another display option for the data streams. Its special feature is that a schematic overview field 18 is provided, on which the positions of some or all of the data sources 1-7 can be seen at a glance. Additionally, the field of view or viewing direction of individual data sources 1, 3, 4 configured as cameras is indicated by a cone radiating from the respective data source 1, 3, 4. Mobile cameras, which are carried by emergency personnel such as firefighters, are marked with squares as data sources 2, 5. Their positions change accordingly over time on the overview field 18.

[0101] In the display fields for data streams 11-15, the number of the respective data source 1-5 is displayed in the lower left corner, allowing an operator to quickly assign the corresponding data source 1-5 in the overview field 18. The stationary data sources 1, 3, and 4 are represented by circles, while the mobile data sources 2 and 5 are represented by rectangles.

[0102] Reference numeral 17, in turn, symbolizes the data stream consisting of sensor data, which is displayed in writing in the form of words, letters, numbers and / or symbols on the screens 42-44. Reference numeral 16, in the present case, again indicates the data stream showing a dispatcher who, for example, is located in a control center and coordinates, for example, a fire-fighting operation. The dispatcher can advantageously also determine the size, location and type of display (including the exclusion of one or more data streams from the individual stream) by sending corresponding commands to the conference server (not shown in the figures). The display or playback of the data streams 11-17 on the screens 42-44 orgenerally the output devices of the subscriber terminals 52-54 preferably changes over time and is dependent on the repeated, preferably continuously running, Kl analysis and evaluation of the data streams 11-17 by the Kl programs 21-25, 32.

[0103] The present invention has been explained using firefighting as an example. In general, it has extremely diverse applications, particularly in real-time monitoring and decision-making. Other exemplary areas of application include traffic monitoring and general status control of an area, such as a city, incorporating, for example, air and weather data, traffic jam forecasts, police and / or ambulance availability, operational coordination in the event of an accident, etc.

[0104] In general, a wide variety of data sources from a wide variety of areas can be used for the invention, for example IoT sensors (e.g. for traffic registration, weather, air quality), HD video cameras, audio sensors, security systems (e.g. access controls, surveillance cameras), data sources from energy management systems (e.g. energy consumption data), traffic systems (e.g. passenger counts), emergency response systems (e.g. accident vehicle tracking, dispatch systems), monitoring and inspections in the healthcare sector (ambulance telematics, control of medication intake or administration, medical imaging).

[0105] The invention can also be used in industry and manufacturing to increase operational efficiency and improve safety measures. The invention enables a wide range of potential applications, from optimizing real-time monitoring of complex machines to analyzing complex supply chains and interactions. Data sources can include machine sensors, factory surveillance cameras, environmental monitoring sensors, quality control systems, human-machine interfaces, surveillance drones, noise sensors, etc.

[0106] Another area is gaming and e-sports (eSports). By integrating a variety of data sources—from game consoles and wearable devices to team performance data and social media feeds—real-time insights can be enabled and the overall gaming experience improved. Data sources include game consoles, wearable devices, social media feeds, streaming platforms, team performance data, player ranking data, etc.

[0107] Furthermore, the invention can be used to improve communication and collaboration across various service platforms, including CCaaS (Contact Center as a Service), UCaaS (Unified Communications as a Service) and CPaaS (Communications Platform as a Service).

[0108] Integrating diverse data sources into real-time video streams enables immediate contextual understanding, improved decision-making, and enhanced customer service.

[0109] The invention can also be used in the FinTech and insurance sectors. Data sources can come from customer interactions, transactions, risk assessments, regulatory compliance, and advanced analytics. The invention can contribute to decision-making, fraud detection, customer service, and policy management.

[0110] The variety of possible applications of the invention is virtually unlimited and may include, for example, the education sector, e-commerce, agriculture, logistics, healthcare, tourism, real estate, etc. The invention has been explained using an exemplary embodiment. Modifications within the scope of the claims are readily possible.

Claims

Patent claims 1 . A computer-implemented method for transmitting and displaying at least video data using a streaming technology, comprising the following steps: Detecting real events recorded by means of a plurality of data sources (1-7) in data streams (11-17), including at least video data streams and preferably also audio data streams and / or other data streams, by at least one Kl program (21-24, 32; 25); Evaluation of the events by at least one class program (21-24, 32; 25); Generating and assigning instructions (A) to one or more of the data streams (11 - 17) generated by the data sources (1-7) by at least one Kl program (21-24, 32; 25) based on the aforementioned detection and evaluation, wherein the assigned instructions (A) relate to the location and / or size and / or type of reproduction of data streams relating to the event on subscriber terminals (52-54), including their screens (42-44); Combining at least several of the data streams (11-17) into a final single stream (39) by an A / V mixer (38) of a conference server (30) implemented as software, taking into account at least some of the instructions (A) assigned to the respective data streams (11-17); Sending the final individual stream (39) from the conference server (30) to a plurality of participant terminals (52-54) for reproducing at least some of the data streams (11-17).

2. Method according to claim 1, characterized in that the events comprise visual events and / or acoustic events and / or other real events based on other data, wherein the data sources (1-7) collect and send one or more of the following data: - Camera recordings; - surveillance videos; - Live drone footage; - Real-time bodycam footage; - sensor data; - satellite communications data; - Tracking data; - tactical communications data; - geographical information data; - Consumption data, e.g. energy consumption data; - Data resulting from electronic games; - Platform streaming sources; - Social media feeds.

3. Method according to at least one of the preceding claims, characterized in that the instructions (A) assigned to one or more data streams (11-17) comprise those from the following group: Instruction (A) that a relevant event has occurred which is to be highlighted on the screens (42-44); Instruction (A) that there is no relevant event or only a slightly relevant event and therefore should not be highlighted on the screens (42-44); Instruction (A) that additional elements in at least one data stream (11-17) are to be displayed on the screens (42-44) when the final individual stream (39) is displayed; Instruction (A) that individual elements from at least one data stream (11-17) are to be hidden when displaying the final individual stream (39) on the screens (42-44); Instruction (A) that no relevant event or only a slightly relevant event is present in at least one data stream (11-17) and that this at least one data stream (11-17) is not to be incorporated into the final individual stream (39); Instruction (A) that one or more audio streams from the subscriber terminals (52-54) are output louder or more attenuated or muted compared to other audio streams.

4. Method according to at least one of the preceding claims, characterized in that the at least one Kl program (21-24, 32; 25) receives data streams (11-17) from the plurality of data sources (1-7) for said detection of events before the AA / mixer (38) of the conference server (30) combines data streams into a single stream.

5. Method according to at least one of the preceding claims, characterized in that several Kl programs (21-24, 32; 25) are provided, each of which recognizes and evaluates events in one or more data streams (11-17).

6. Method according to at least one of the preceding claims except for the two immediately preceding ones, characterized in that the conference server (30) receives data streams (11-17) from the plurality of data sources (1-7), the AA / mixer (38) combines these into a first individual stream (39') and the conference server (30) sends this first individual stream (39') to the at least one Kl program (25) for said detection of events.

7. Method according to the preceding claim, characterized in that after said assignment of the instructions (A) by the at least one Kl program (25) of the conference server (30) is caused to continue to receive the data streams (11-17) from the data sources (1-7) and to combine them, taking into account the associated instructions (A), to form the final individual stream (39), in order to then send this to the subscriber terminals (52-54) for corresponding display on their screens (42-44).

8. Method according to at least one of the preceding claims, characterized in that the at least one Kl program (32) is integrated in the conference server (30).

9. Method according to at least one of the preceding claims, characterized in that the at least one Kl program (21 - 24, 32; 25) also evaluates events in different data streams (11 - 17) in relation to one another.

10. Method according to at least one of the preceding claims, characterized in that, depending on the assigned instructions (A), the display of the data streams (11-17) combined in the final individual stream (39) on the screens (42-44) of the subscriber terminals (52-54) initiated by the conference server (30) is changed dynamically over time.

11. Method according to at least one of the preceding claims, characterized in that the conference server (30) is designed such that it can receive and process commands from authorized persons in order to change the display of the data streams (11-17) combined in the final individual stream (39) on the screens (42-44) of the subscriber terminals (52-54).

12. Method according to at least one of the preceding claims, characterized in that the at least one Kl program (21-24, 32; 25) performs one or more of the following actions based on the recognition of real events: - Initiating an event-related message on one or more subscriber terminals (52-54) outside the final individual stream (39); - Determining to which subscriber terminals (52-54) of a plurality of subscriber terminals (52-54) the conference server (30) sends the final individual stream (39), wherein preferably said or another A / V mixer (38) combines at least one further final individual stream (39) from partially different data streams (11-17), which is sent to other subscriber terminals (52-54); - causing at least one data source (1-7) to be controlled to modify the recording of the data stream (11-17); - sending said instructions (A) to another application implemented as software for processing outside the final single stream (39).

13. Method according to at least one of the preceding claims, characterized in that the final individual stream (39) is composed by the AA / - mixer (38) of the conference server (30) in such a way that the relative spatial arrangement of several data sources (1-6) is displayed on an area (18) of the screens (42-44) of the subscriber terminals (52-54).

14. Method according to at least one of the preceding claims, characterized in that information from data sources (1-7) is displayed as text output in at least one area of ​​the screens (42-44).

15. Method according to at least one of the preceding claims, characterized in that the at least one Kl program from at least one data stream (11 - 17) comprises secondary, not the primary derives information relating to data in the data streams, which information can preferably be output via the subscriber terminals (52-54).

16. System (100) for information transmission and presentation using a streaming technology with at least one computing unit on which the conference server (30) is implemented and which is designed to carry out a computer-implemented method according to one or more of the preceding claims.

17. Computer program which, when executed by a computing unit, causes the computing unit to carry out the method steps of a computer-implemented method according to one or more of the preceding claims.

18. A computer-readable storage medium comprising a computer program which, when executed by a computing unit, causes the computing unit to carry out the method steps of a computer-implemented method according to one or more of the preceding claims.

Images