A system, method, and storage medium for transmitting a plurality of data streams via a communication network for remote monitoring

Abstract

Disclosed are a system, a method, and a storage medium for transmitting a plurality of data streams via a communication network for remote monitoring. In an exemplary embodiment, a plurality of video streams may be pre-processed into a single synchronized frame, the single synchronized frame may be encoded, packetized as multiplexed packets using the multiplexed packets via the communication network, the multiplexed packets received via the communication network may be depacketized to generate a single synchronized frame, and the synchronized frame may be decoded into a plurality of video streams for remote monitoring.

Description

Technical Field

[0001] The present disclosure relates to a system, method, and storage medium for transmitting a plurality of data streams via a communication network for remote monitoring.

Background Art

[0002] Transmitting audio and / or video via a communication network for output at a remote location is now common both for entertainment and commercial use. Transmission of video and audio has become more important in the case of work at a remote location, but will become even more important in the future as automation, i.e., "robots," progresses.

[0003] As an example, operations performed on an offshore vessel, including everything from a research submarine to operations on the deck or bridge of a mother ship, can be automated by "robots." The robots are not fully automated and still require monitoring and at least partial control by personnel. Even so, the number of personnel on board performing monitoring and / or control of the robots can be significantly reduced, and in cases where the personnel are located on land and can remotely monitor / control the on-board operations via video and / or audio returned to a ground control headquarters or control room, the number of personnel can in some cases be completely eliminated.

Summary of the Invention

[0004] One aspect of the present disclosure relates to a system configured to transmit a plurality of data streams via a communication network for remote monitoring. The system may include one or more hardware processors set by machine-readable instructions. The one or more processors may be configured to preprocess a plurality of video streams into a single synchronized frame. The one or more processors may be configured to encode the synchronized single frame. The one or more processors may be configured to packetize the encoded synchronized single frame as multiplexed packets for transmission using the multiplexed packets via a communication network. The one or more processors may be configured to depacketize the multiplexed packets received via the communication network to generate a synchronized single frame. The one or more processors may be configured to decode the synchronized frame into a plurality of video streams for remote monitoring.

[0005] In some embodiments of the system, the one or more processors may be configured to buffer the multiplexed plurality of packets received via the communication network.

[0006] In some embodiments of the system, the one or more processors may be configured to post-process the buffered multiplexed plurality of packets to generate a plurality of synchronized single frames for remote monitoring.

[0007] In some embodiments of the system, post-processing may further include counting the frames.

[0008] In some embodiments of the system, post-processing may further include synchronizing the frames.

[0009] In some embodiments of the system, the one or more processors may be configured to simultaneously display a plurality of channels on a single output device.

[0010] In some embodiments of the system, one or more processors may be configured to encrypt multiplexed packets before they are transmitted via a communication network. In some embodiments of the system, one or more processors may be configured to decrypt multiplexed packets received via a communication network.

[0011] Another aspect of the present disclosure relates to a method of transmitting a plurality of data streams via a communication network for remote monitoring. The method may include preprocessing, at an operating location, a plurality of video streams obtained from individual live video feeds, the plurality of video streams being preprocessed into a single synchronized frame of video data and individual audio streams of audio data. The method may also include encoding the single synchronized frame. The method may also include packetizing the encoded single synchronized frame as multiplexed packets for transmission from the operating location to a monitoring location physically remote using the plurality of multiplexed packets via the communication network. The method may also include depacketizing the multiplexed packets received via the communication network to generate a single synchronized frame. The method may also include decoding the synchronized frame into a plurality of video streams and displaying the plurality of channels on at least one output device for remote monitoring at the monitoring location.

[0012] In some embodiments of the method, the method may further include buffering the plurality of multiplexed packets received via the communication network.

[0013] In some embodiments of the method, the method may further include postprocessing the buffered plurality of multiplexed packets to generate a plurality of single synchronized frames for remote monitoring.

[0014] In some embodiments of the present method, the post - processing step may further include a step of counting frames.

[0015] In some embodiments of the present method, the post - processing step may further include a step of synchronizing frames.

[0016] In some embodiments of the present method, the present method may further include a step of simultaneously displaying a plurality of channels on a single output device.

[0017] In some embodiments of the present method, the present method may further include a step of encrypting multiplexed packets before being transmitted via a communication network. In some embodiments of the present method, the present method may include a step of decrypting multiplexed packets received via a communication network.

[0018] Yet another aspect of the present disclosure is a non - transitory computer - readable storage medium having instructions incorporated therein, the instructions being executable by one or more processors to perform a method of transmitting a plurality of data streams via a communication network for remote monitoring. The method may include pre - processing, at an offshore operating location, a plurality of video streams obtained from individual live video feeds, the plurality of video streams being pre - processed into audio data and video data. The method may also include encoding the video data into a synchronized single frame. The method may also include packetizing the encoded synchronized single frame as multiplexed packets for transmission to an on - shore monitoring location physically remote from the operating location using a plurality of multiplexed packets via a satellite communication network. The method may also include transmitting the audio data separately from the video data as an audio stream. The method may also include depacketizing the multiplexed packets received via the satellite communication network to generate a synchronized single frame. The method may also include decoding the synchronized frame into a plurality of video streams and simultaneously displaying a plurality of channels on at least one output device for remote monitoring at the on - shore monitoring location.

[0019] In some embodiments of the computer - readable storage medium, the method may further include buffering a plurality of multiplexed packets received via the communication network.

[0020] In some embodiments of the computer - readable storage medium, the method may further include post - processing the buffered multiplexed packets to generate a plurality of synchronized single frames for remote monitoring.

[0021] In some embodiments of the computer-readable storage medium, the post-processing step may further include a step of counting frames.

[0022] In some embodiments of the computer-readable storage medium, the post-processing step may further include a step of synchronizing frames.

[0023] In some embodiments of the computer-readable storage medium, the method may further include a step of simultaneously displaying a plurality of channels on a single output device.

[0024] The above and other features and characteristics of the present technology, the operating methods and functions of the elements related to the configuration, the combinations of parts, and the economics of manufacturing will become more apparent by considering the following description and the appended claims with reference to the accompanying drawings. The entire accompanying drawings form a part of this specification, and in the drawings, like reference numerals indicate corresponding parts in various figures. However, it should be clearly understood that the drawings are only used for illustrative and explanatory purposes and are not intended to define the boundaries of the present invention. It should be clearly understood that the singular forms "a", "an", and "the" used in this specification and the claims include plural referents unless the context clearly dictates otherwise.

Brief Description of the Drawings

[0025]

Figure 1

[0026]

Figure 2

[0027]

Figure 3A

Figure 3B

[0028]

Figure 4A

Figure 4B

[0029]

Figure 5

[0030]

Figure 6

[0031] The present disclosure relates to transmitting a plurality of data streams via a communication network for remote monitoring. In existing video and / or audio transmissions, multiple data streams are combined or "multiplexed" into packets for transmission via a communication network. For example, in a four-camera system, four data streams are encoded into timestamped packets for transmission. The decoder decodes the received timestamped packets into individual video streams and / or audio streams, but multiplexing introduces overhead that degrades the transmission efficiency and quality of video and / or audio.

[0032] The embodiments described in the present disclosure address the above and other drawbacks by providing a remote inspection system (RIS). The RIS includes a video processor that encodes a plurality of video streams and / or audio streams into a single synchronized frame and then decodes the frame for remote monitoring. For example, in a four-camera system, four buffers can be read to form a single synchronized quad frame having four elements, eliminating the need to attach timestamps. The overhead of a frame having four elements in a set is smaller than that of a conventional timestamped packet. By being able to transmit the frame in a single cycle, the transmission efficiency and quality of video and / or audio used for remote monitoring are improved.

[0033] FIG. 1 shows an example of video and / or audio transmission via a communication network between a ship and a coastal or onshore headquarters according to one or more embodiments. The ship and the coastal or onshore headquarters are merely illustrative of the operating environment and are not intended to limit beyond the scope explicitly set forth in the claims.

[0034] In this figure, the method of transmitting video and / or audio may be implemented via any suitable communication network 100 such as a satellite, a mobile network (e.g., 3G, 4G, 5G, etc.) and / or the Internet (but not limited thereto). The offshore remote operation side may include any operation on one or more vessels 110 (e.g., a research vessel), and / or any operation related to the vessel (e.g., an operation by a research submarine). The offshore remote operation side may include obtaining data (e.g., obtaining data by cameras 112a, 112b). The data may include video data 114, audio data 116 and / or sensor data 118, but is not limited thereto, and these may be processed by the processing electronic device 120 and transmitted to the coastal operation side 102 via the communication network 100.

[0035] The coastal operation side 102 may include post-processing the data transmitted from the offshore remote operation side 101 via the communication network 100. The post-processing may be by the post-processing electronic device 130 that obtains data (e.g., video data 132, audio data 134 and / or sensor data 136) used for analysis at the remote operation center 140. The data may be analyzed, for example, by a person who checks the audio feed and / or video feed of the work performed by the robot on the vessel 110, and this is schematically shown as operation 01 and operation 02 in FIG. 1.

[0036] FIG. 2 is a high-level block diagram of a remote video and / or audio configuration 200 used for transmission via a communication network according to one or more embodiments. In FIG. 2, the conventional offshore configuration 210 may include receiving video data from a camera feed 212, processing the corresponding video data 214, and subsequent human confirmation and / or reporting 216.

[0037] The remote video and / or audio configuration 200 for transmission via a communication network is exemplified by a real-time remote configuration 220 and a retrospective remote configuration 230. The real-time remote configuration 220 may include receiving video data from a camera feed 222, processing the corresponding video data according to the techniques described in this disclosure 224, and subsequent human confirmation and / or reporting 226. The processing of the data may be performed in real time, for example, in real time when a video feed is acquired on a ship. In this example, processing the corresponding video data 224 may include preprocessing the data (e.g., preprocessing the data on the ship for transmission via a communication network) and then postprocessing the data (e.g., postprocessing the data at a ground headquarters), thereby reducing the need for human intervention on the ship.

[0038] The retrospective time remote configuration 230 may include receiving video data from a camera feed 232, processing the corresponding video data according to the techniques described in this disclosure 234, and subsequent human confirmation and / or reporting 236. The processing of the data may be performed afterwards, for example, at the end of the day or following the operation on the ship. In this example, processing the corresponding video data 234 may include preprocessing the data (e.g., preprocessing the data on the ship for transmission via a communication network) and then postprocessing the data (e.g., postprocessing the data at a ground headquarters), thereby reducing the need for human intervention on the ship.

[0039] Figures 3A and 3B show flowcharts of examples of operations 300a and 300b used for encoding video at a remote location, transmission via a communication network, and decoding the transmitted video, according to one or more embodiments. Note that Figures 3A and 3B form a combined flowchart and are not independent of each other.

[0040] Figure 3A shows data 310 generated at a remote location (e.g., the offshore work location in FIG. 1, i.e., a ship). Examples of data 310 may include, but are not limited to, telemetry data, audio data, and raw video. The processing of audio data will be described later with reference to FIGS. 4A and 4B. In FIG. 3A, the raw video is shown as four streams of video data, i.e., four channels. The data is pre-processed by a video processor 314. For example, a frame 314 having a set of four synchronized elements is generated, and then encoded (316) and packetized (318) as data packets 320, 321. The operation continues in FIG. 3B.

[0041] In FIG. 3B, the data packets 320, 321 are transmitted via a network 330. The data packets 320, 321 are depacketized (340) and decoded (342). In one example, the decoded data may be buffered. The video data is post-processed (344) and output as a frame 350 having a set of four synchronized elements. In one example, a frame having a set of four synchronized elements may be output as a user-configurable channel (e.g., four channels can be simultaneously displayed as a single channel) by an output device 355. The corresponding telemetry data and audio data may also be output for the end user.

[0042] A typical embodiment of encoding and transmitting video at a remote location according to one or more embodiments includes efficiently compressing a predetermined number (one, two, three, or four or more) of input video streams by combining individual images into a single frame, and then separating the single frame into the original number (e.g., four) of images after transport. This approach enables efficient communication using a data transport protocol where the bandwidth may be limited. As an example, bandwidth-limited communication may exist even in satellite-based communication, which is often carried out between an offshore headquarters (e.g., a research vessel) and an onshore headquarters on the ground (e.g., a data processing center). Note that this example is not intended to impose any limitations in any way.

[0043] In a typical embodiment, encoding and transmitting video at a remote location includes a data transmission protocol that combines a predetermined number (e.g., four) of video channels, and manipulates the video data issued over the network by combining and compressing data packets for transport. In one example, the video channels include high-definition video (e.g., 1080p), and a plurality of video channels are formatted into a single frame (e.g., 4k) for transmission using a bandwidth-limited connection with normal node stream encoding / decoding techniques.

[0044] Telemetry data suitable for transmission may also be provided. For example, all data from depth sensors, inertial sensors, orientation, direction, cameras, sensors for performing tests may be introduced in a continuous character string for navigation together with the camera feed. Audio data may also be included. For example, the audio may include a live narration by a monitor regarding the object being monitored / commented on. The audio will be described in detail later with reference to FIGS. 4A and 4B.

[0045] The navigation string is converted into a video overlay and encoded for transport and reconstruction. As an example, the reconstruction may typically be done ashore for a surveillance technician based ashore. The data is passed to and input into a streaming decoder (sent ashore, split / decoded), and then the ashore workflow is executed to manage the operation ashore. This procedure can be said to "reflect" the conventional operation where everyone is on board. On the other hand, without having to have everyone on board, personnel can be located at a location away from the ship (e.g., ashore) by the procedure of encoding and transmitting the video at a remote location.

[0046] Implementing the method of encoding and transmitting video at a remote location described in this disclosure, returning instructions to the ship (e.g., returning instructions to the crew on board the ship and / or machines or other devices (e.g., devices operating cameras)), and further examining what the surveillance technician sees through the video feed may also be done. This may be particularly important when this method is implemented on a fully autonomous ship. [[ID=⑥]]

[0047] [[ID=⑦]] Conventionally, video from multiple cameras, along with audio data and telemetry data, can be timestamped, multiplexed, and packetized. This typically involves combining video data, audio data, and telemetry data, as well as packet overhead and timestamps, and then converting this into packets of a new RTP format or packets of another format for transport over a network, which includes adding yet another layer (e.g., an RTP layer) to the packets. As an example, a preprocessor having four encoders needs to operate its own hardware, and often the four encoders are temporally separated from each other, i.e., not linked, and each acquires individual frames from individual cameras. Subsequently, the packetized data is received on the other side and processed in reverse order (demultiplexer). That is, video is extracted from the packetized data, synchronized, and sent to each display. Each video is a single channel. Each encoder necessarily has its own demultiplexer, and this process necessarily needs to be repeated each time (e.g., four times in the case of four video feeds). The overhead increases with each step. This may be acceptable when bandwidth is not a problem on a closed local network (e.g., in the case of a closed caption TV or CCTV system), but this method requires an overhead that is overly large for transmission via a communication network (e.g., a satellite) for transport over a remote location.

[0048] In contrast to the above, in the method described in this disclosure, video is preprocessed. All frames are obtained in one cycle at a time, and the timing with the corresponding frames is aligned. Since only one encoder is needed, all the time between single-threaded videos occurs during the same one or more CPU cycles. This is referred to as synchronous data capture in this disclosure.

[0049] Note that when the term "synchronous" is used in this disclosure, the frame synchronization data means the data that arrives within the limited time. As an example, all transmission systems (RSS, RS232 or UDP) are asynchronous by definition. That is, the chipset is asynchronous. On the other hand, the term "synchronous" used in this disclosure means that any data that arrives within the 16-millisecond limited time will be sent using the corresponding frame. The frame in which the video is output is the same frame as the one that received the data. This enables the system described in this disclosure to simultaneously acquire all multiple telemetry data from different sources and output all of them together with the corresponding video.

[0050] Paying attention again to the preprocessing of the video, in one example, four 1080p frames may be combined into a single 4k frame. The four frames are copied to the areas corresponding to each in the 4k frame. By putting the four frames into one encoder, in this method, the overhead is removed four times compared to the conventional system.

[0051] In addition to the above, the entire frame may be used as a reference by taking advantage of the way the encoder works. That is, when one 1080 video is input to the encoder, a part of the video may be used and other parts of the video may be referred to in subsequent frames. As an example, if an orange circle disappears in frame 30 and reappears again in frame 35, there is no need to resend the orange circle. Instead of resending, it can be referred to. In a single frame, a frame that can be used in subsequent frames may be referred to. Up to four frames may be referred to for any other frame. For example, if there is an orange dot in camera 1 and the same orange dot is in frame 35 in camera 4, there is no need to send the video from camera 4.

[0052] An algorithm for whether to send or reference a frame can be executed by the encoder hardware. In one example, since data can be decompressed with the same encoder for all frames, multiple frames from multiple other cameras can be represented as one picture. A portion of the camera video (e.g., camera 4) can be used in a previous frame to construct data for another camera (e.g., camera 1).

[0053] In a typical embodiment, encoding and transmitting video at a remote location involves video processing. During video processing, scaling of one or more video streams is performed on the way out of the camera and into the pre-processor. The decoder references each frame when receiving the frame. The encoder tuner may also perform motion evaluation, reference to previous frames, etc., to determine the output optimal for the minimum bitrate and optimize the video based on the input feed. Note that this process is more efficient than four separate feeds. Unlike four separate feeds, the data is packetized using only one descriptor (not four as in the prior art).

[0054] To further improve transmission efficiency, the control system may allow the user to focus on any one or all of the video streams at most. When the user is browsing only one channel (or two channels, etc.), the system sends only the channel being browsed.

[0055] In a typical embodiment, encoding and transmitting video at a remote location involves a new packetizer. A multiplexer is not required to pack all formats into one packet. Instead, the video packets are separate from the audio packets, and the audio packets are separate from the data packets. Only two headers exist.

[0056] When receiving video packets (e.g., receiving them along the shore), since they can be split and regenerated into individual streams, with only one-time decoding, the user can view any of the video streams on any output device. The method of encoding and transmitting video at a remote location includes the system recognizing the number of incoming frames and the number of frames present that are referenced in the packet when receiving packets out of sync by counting frames on both sides. The packets are read into a synchronization buffer and timed with the frames. Knowing the cause of any time shift when the payload of the video data is full and when it is empty, there is sufficient time to send telemetry data, thus preventing 90% overhead of the packets. Of course, since all packets have overhead, it is more efficient when all packets are as full as possible.

[0057] In one example, telemetry and video are sent separately. This also has the advantage of maintaining a constant (or nearly constant) bitrate over time. Packets of the same (or substantially the same) size are being sent, as opposed to being bursty. Sending telemetry data during troughs is useful for maintaining a constant baseline bitrate, which is preferred in satellite communication, for example, where variable bitrates do not function effectively.

[0058] In a typical embodiment, encoding and transmitting video at a remote location includes processing the packets on one side (e.g., along the shore) at the same rate as the packets arrive. In one example, the user can increase or decrease the output speed. The system may buffer the packets for smooth playback.

[0059] Note that the encoder may also include temporal analysis (e.g., regions of interest within a video segment). In one example, the temporal analysis is extended to the edges of the frame such that the region of interest is the entire frame. Thereafter, the encoder can rather evaluate the entire frame and render the outer edges. The temporal portion of the encoder can also significantly reduce the bitrate and further reduce the packet size that needs to be transmitted.

[0060] Figures 4A and 4B illustrate flowcharts of examples of the operation of encoding audio at a remote location, transmitting via a communication network, and decoding the transmitted audio, according to one or more embodiments. In the remote monitoring industry, attention has shifted to two groups. Group 1 requires communication with the lowest possible latency, and Group 2 requires the highest quality and most reliable video transmission, which is used for synchronizing telemetry data and video along the coast. Based on this, the synchronization of audio data and visual data is separated to provide a voice communication system at the technical limit and to provide a highly reliable video using a synchronization data distribution system. The audio system is different from other communication systems. In a conventional IP telephony network, all participants perform both the transmission of data and the reception of data from all other participants. Therefore, the bandwidth requirements quickly become complex, and this approach is not practical for large groups transmitting via satellite. In the approach disclosed in this disclosure, audio is mixed at each group point, so that only what corresponds to one participant is transmitted to the group at the downstream end.

[0061] In a typical embodiment, encoding and transmitting video at a remote location involves separating the audio. That is, there is no need to align the timing of the audio with the video for transmission. The audio portion may be processed by an encoder separate from the encoder that processes the video. As an example, there may be individual audio devices for all of the audio on a bridge, inside a submarine, inside a monitoring room, etc. If these are implemented as individual VOIP servers, there will be audio traffic from each of them and everyone will get the data of everyone else (known as polycasting). To reduce the transmission size (e.g., reduce the transmission size for a satellite link), the individual audio devices on an offshore ship can be linked to one device on the same ship. The system downlinks all the audio to a single stream or a single audio packet. The reverse process is done on the shore. The audio may be incoming audio using a daisy chain or multiplexed incoming audio.

[0062] In the example shown in FIG. 4A, a two-way audio network 400 transmits / receives audio corresponding to one or more video streams. For example, two-way audio I / O 410a and 410b originating from one location (e.g., a ship) may be encoded (412a, 412b) and packetized (414a, 414b) before being transmitted via a communication network 450.

[0063] The two-way audio I / O 410a and 410b from the first location (e.g., a ship) received at the other location (e.g., an onshore analysis location) may be depacketized (426a, 426b) and decoded (428a, 428b) after being received via the communication network 450.

[0064] [[ID=Eleven]] Two-way audio I / O 420a and 420b originating from another location (e.g., an onshore analysis location) may be encoded (422a, 422b) and packetized (424a, 424b) at the onshore analysis location before being transmitted via the communication network 450.

[0065] Two-way voice IOs 420a and 420b received at one location (e.g., a ship) from another location (e.g., an onshore analysis location) may be depacketized (416a, 416b) and decoded (418a, 418b) after being received via communication network 450.

[0066] Figure 4B shows the combination of voices from multiple voice devices 460, 461, 462 of the onshore network 465 by a single voice device 463 and the transmission of the combined voice via communication network 470. The voice is received by a voice device 483 of the ship network 485 and separated into corresponding voice devices 480, 481, and 482.

[0067] Figure 4B also shows two-way voice communication including combining voices from multiple voice devices 480, 481, 482 of the ship network 485 by a single voice device 483 and transmitting the combined voice via communication network 470. The voice is received by a voice device 463 of the onshore network 465 and separated into corresponding voice devices 460, 461, and 462. Figure 4B also shows the daisy-chain connection of voice device 490.

[0068] Figure 5 shows a system 500 configured to transmit a plurality of data streams via a communication network for remote monitoring, according to one or more embodiments. In some embodiments, system 500 may include one or more computing platforms 502. The one or more computing platforms 502 may be configured to communicate with one or more remote platforms 504 according to a client / server architecture, a peer-to-peer architecture, and / or other architectures. The one or more remote platforms 504 may communicate with other remote platforms via the one or more computing platforms 502 and / or may be configured to communicate with other remote platforms according to a client / server architecture, a peer-to-peer architecture, and / or other architectures. A user may access system 500 via the one or more remote platforms 504.

[0069] The one or more computing platforms 502 may be set by machine-readable instructions 506. The machine-readable instructions 506 may include one or more instruction modules. The instruction modules may include computer program modules. The instruction modules may include one or more of a stream preprocessing module 508, a frame encoding module 510, a frame packetization module 512, a packet depacketization module 514, a frame decoding module 516, a packet buffering module 518, a packet postprocessing module 520, a channel display module 522, a packet encryption module 524, a packet decryption module 526, a packet transmission module 528, a video data telemetry data transmission module 530, a video feed reference module 532, a data separation module 534, and / or other instruction modules.

[0070] The stream preprocessing module 508 may be configured to preprocess a plurality of video streams into a single synchronized frame. As used in this disclosure, the term "video stream" refers to video images electronically acquired via a video camera for continuous delivery and consumption. A "synchronized frame" refers to a data structure that includes video data obtained from a plurality of video streams that are temporally synchronized with each other. In other words, video images captured at the same time T from a plurality of sources can be configured in the data structure to correspond to each other with respect to time T. Preprocessing a plurality of video streams may include reading four buffers of four video channels and forming a single 4K frame having a set of four elements. One video feed may occupy all four channels of a single 4K frame having a set of four elements.

[0071] The frame encoding module 510 may be configured to encode the single synchronized frame.

[0072] The frame packetization module 512 may be configured to packetize the encoded single synchronized frame as multiplexed packets for transmission using a plurality of packets multiplexed over a communication network. As used in this disclosure, the term "multiplexed packets" refers to a plurality of data packets or other electronic signals representing data transmitted substantially simultaneously with each other over a single communication channel. As used in this disclosure, the term "communication network" refers to any electronic communication network for transmitting / receiving data in electronic form. Examples of communication networks include, but are not limited to, telephones, 3G, 4G, and 5G data networks (and future mobile data networks), satellite communication networks, and the Internet.

[0073] The packet depacketization module 514 may be configured to depacketize the multiplexed packets received over the communication network to generate a single synchronized frame.

[0074] The frame decoding module 516 may be configured to decode the synchronized frame into a plurality of video streams for remote monitoring. As used in this disclosure, the term "remote monitoring" refers to viewing and / or listening to video data and / or audio data at a location physically remote from the location where the video data and / or audio data was acquired. The term "monitoring" may refer to human monitoring in some cases, but this term is not limited to human monitoring and may also include monitoring methods assisted by humans and / or fully automated monitoring methods.

[0075] The packet buffering module 518 may be configured to buffer a plurality of multiplexed packets received via a communication network.

[0076] The packet post-processing module 520 may be configured to post-process a plurality of buffered multiplexed packets so as to generate a plurality of synchronized single frames for remote monitoring.

[0077] The channel display module 522 may be configured to simultaneously display a plurality of channels on a single output device. Note that any output device, such as a computer monitor and / or other display and / or audio device for rendering video data and / or audio data, may be implemented. The term "single" used in the reference to an output device in this disclosure means one device, but one device may be divided into a plurality of components such as individual video display areas on a single computer monitor.

[0078] The packet encryption module 524 may be configured to encrypt the multiplexed packets before transmission via a communication network.

[0079] The packet decoding module 526 may be configured to decode multiplexed packets received via a communication network.

[0080] The packet transmission module 528 may be configured to transmit all of the multiplexed packets via a communication network in a single cycle. Data is typically transmitted through a computer network along a series of nodes. After a plurality of data packets that belong together are transmitted across any number of nodes (i.e., network paths) within the communication network, they may be reconstructed at the destination of the plurality of data packets regardless of the network path. Data that is transmitted together across the same node can be said to be transmitted in the same cycle. Or, when the term "single cycle" is used in this disclosure, it can be said to be transmitted during the "single cycle".

[0081] The video data telemetry data transmission module 530 may be configured to transmit video data and telemetry data separately via a communication network. The term "telemetry data" refers to data that is collected at multiple locations and automatically transmitted to one or more receiving devices for monitoring. The term "video data" as used in this disclosure refers to data (e.g., image data) corresponding to a video stream, as the term has already been defined in this disclosure. The video data may represent a part or the whole of the video stream.

[0082] The video feed reference module 532 may be configured to reference one video feed in a decoder and may be configured to reference the feed without retransmitting it. The term "decoder" as used in this disclosure refers to program code and / or electronic equipment configured to convert encoded data (e.g., video data) into a form readable by an endpoint (e.g., a form for output to a display device).

[0083] The data separation module 534 may be configured to separate the audio data from the video data for individual transmission via a communication network. The term "audio data" as used in the present disclosure represents sound waves and refers to data corresponding to an audio stream. The audio data may represent a part or the whole of the audio stream. It should be noted that the audio data may be separately packetized and / or pre-processed for transmission separately from the video data. For example, the video data may be transmitted at a peak, and the audio data may be transmitted during a trough. The audio data from all of the audio input devices may be transmitted via the communication network as individual 64k blocks of audio data. For transmission via the communication network, any suitable audio input device such as a microphone or other device for receiving sound waves and digitizing the sound waves (e.g., digitizing as an analog signal and / or a digital signal) may be implemented. It should be noted that the audio data may be transmitted in data blocks of any suitable size and is not limited to 64k blocks.

[0084] In some embodiments, post - processing may further include counting frames. In some embodiments, post - processing may further include synchronizing frames. In some embodiments, telemetry data may be separately transmitted in troughs so as to maintain a substantially constant bitrate during transmission over a communication network. It is well known that the term "trough" in electronic communication means one or more locations in a cycle or signal having the minimum amplitude. The term "bitrate" as used in this disclosure refers to the number of bits per second that can be transmitted over a communication network. Transmission is not limited to a predetermined bitrate. In some embodiments, all voice input devices may be linked through a single voice device to transmit voice data from all of the voice input devices as a single stream over a communication network. The term "single" as used in this description means one part or a distinct unique part. In some embodiments, at least some of the voice input devices may be daisy - chained and input into a single voice device.

[0085] In exemplary embodiments, encoding and transmitting video at a remote location involves using different protocols. For example, an encoder may implement its own packetization protocol that determines how data is sent and may select from standard transmission protocols. Standard protocols may include, but are not limited to, UDT, UDP, SRT, SRT UDP / FEC.

[0086] In another example, the system may implement its own protocol to remove overhead. Note that the term "protocol" as used in this disclosure refers to various layers for transport, such as various layers from the physical layer upwards and the presentation layer. This new protocol includes latency techniques and other parameters defined at the transport layer. This enables flexibility in sending files, data, video, audio, etc. In addition to the above, data integrity is important in the protocol, but packets may be sent in real-time mode when the importance decreases over time (e.g., eventually reaching zero). The protocol may also set the importance (e.g., set to zero). This promotes removing as much packet overhead as possible and enables communication between endpoints regarding the status of data equivalence with minimal messages.

[0087] In an exemplary embodiment, encoding and transmitting video at a remote location includes an appliance application or "app" (e.g., an app for a tablet device). The app may display or show the user various encoders, decoders, video streams, audio systems, etc. available for a particular facility. The user may draw lines to connect the device to other components (e.g., a video stream). For example, the user may draw lines to connect an encoder and a decoder and the audio that will necessarily pass through. The app may display or show the user where the audio can go and where it cannot go. This allows the user to distribute the audio to relevant personnel higher up in the chain of command.

[0088] In a general usage example, when starting a job, the user may draw a connection to the facility. Without moving to and setting up the actual physical device, the user can adjust parameters such as bitrate and latency based on network quality, thereby providing the user with centralized control. This may be particularly useful, for example, when the device is located in a rack (especially useful even if the device is in a rack behind or other hard-to-reach location).

[0089]

[0090] In some embodiments, one or more computing platforms 502, one or more remote platforms 504, and / or external resources 536 may be functionally linked via one or more electronic communication links. By way of example, such electronic communication links may be at least partially established via a network such as the Internet and / or other networks. This is not intended to be limiting, and it will be understood that the scope of the present disclosure includes embodiments in which one or more computing platforms 502, one or more remote platforms 504, and / or external resources 536 may be functionally linked via any other communication medium.

[0091] A given remote platform 504 may include one or more processors configured to execute computer program modules. The computer program modules may be configured to enable a person or user associated with the given remote platform 504 to interface with the system 500 and / or external resources 536 and / or to perform other functions attributable to one or more remote platforms 504 in the present disclosure. By way of non-limiting example, the given remote platform 504 and / or the given computing platform 502 may include one or more of a server, a desktop computer, a laptop computer, a handheld computer, a tablet computing platform, a NetBook, a smartphone, a gaming console, and / or other computing platforms.

[0092] External resources 536 may include information sources external to the system 500, external entities involved with the system 500, and / or other resources. In some embodiments, some or all of the functions attributable to the external resources 536 in the present disclosure may be provided by resources included in the system 500.

[0093] One or more computing platforms 502 may include electronic storage 538, one or more processors 540, and / or other components. One or more computing platforms 502 may include communication lines, or ports that enable the exchange of information with a network and / or other computing platforms. The illustrated examples of one or more computing platforms 502 in FIG. 5 are not intended to be limiting. One or more computing platforms 502 may include a plurality of hardware components, software components, and / or firmware components that cooperate to provide the functionality attributed to one or more computing platforms 502 in this disclosure. For example, one or more computing platforms 502 may be implemented by a cloud of computing platforms that cooperate as one or more computing platforms 502.

[0094] The electronic storage 538 may comprise a non-transitory storage medium for electronically storing information. The electronic storage medium of the electronic storage 538 may include one or both of system storage that is provided integrally (i.e., substantially non-removable) with one or more computing platforms 502, and / or removable storage that is removably connectable to one or more computing platforms 502 via, for example, a port (e.g., a USB port, a FireWire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storage 538 may include one or more of an optically readable storage medium (e.g., an optical disk, etc.), a magnetically readable storage medium (e.g., a magnetic tape, a magnetic hard drive, a floppy drive, etc.), a charge-based storage medium (e.g., an EEPROM, a RAM, etc.), a solid state storage medium (e.g., a flash drive, etc.) and / or other electronically readable storage media. The electronic storage 538 may include one or more virtual storage resources (e.g., cloud storage, a virtual private network and / or other virtual storage resources). The electronic storage 538 may store software algorithms, information determined by one or more processors 540, information received from one or more computing platforms 502, information received from one or more remote platforms 504, and / or other information that enables one or more computing platforms 502 to function as described in the present disclosure.

[0095] One or more processors 540 may be configured to provide information processing capabilities in one or more computing platforms 502. Accordingly, one or more processors 540 may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and / or other mechanisms for electronically processing information. In FIG. 5, one or more processors 540 are shown as a single entity, but this is for illustrative purposes only. In some embodiments, one or more processors 540 may include multiple processing sites. These processing sites may be physically located within the same device, or one or more processors 540 may represent the processing functions of multiple devices operating in cooperation. One or more processors 540 may be configured to execute modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532 and / or 534 and / or other modules. One or more processors 540 may be configured to execute modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532 and / or 534 and / or other modules by software, by hardware, by firmware, by a combination of software, hardware and / or firmware, and / or by other mechanisms for setting processing capabilities in one or more processors 540. As used in this disclosure, the term "module" may refer to any component or set of components that performs the functions attributed to the module. This may include one or more physical processors executing processor-readable instructions, processor-readable instructions, circuits, hardware, storage media, or other components.

[0096] In FIG. 5, modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534 are shown as being implemented within a single processing site. However, in embodiments where one or more processors 540 include multiple processing sites, it will be appreciated that one or more of modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534 may be implemented at locations remote from other modules. The descriptions provided below of the functions provided by the various modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534 are for purposes of illustration only and are not intended to be limiting. Any of modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534 may provide more or fewer functions than those described. By way of example, one or more of modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534 may be deleted and some or all of their functions may be provided by other modules of modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534. As another example, one or more processors 540 may be configured to execute one or more additional modules that can perform some or all of the functions attributed to one of modules 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, and / or 534 as described below.

[0097] FIG. 6 shows a method 600 for transmitting a plurality of data streams via a communication network for remote monitoring according to one or more embodiments. The operations of method 600 shown below are intended to be illustrative. In some embodiments, method 600 may be performed using one or more additional operations not described and / or may be performed without using one or more of the operations described. In addition to the above, the order in which the operations of method 600 are shown in FIG. 6 and the order in which the operations of method 600 are described below are not intended to be limiting.

[0098] In some embodiments, method 600 may be implemented on one or more processing devices (e.g., digital processors, analog processors, digital circuits designed to process information, analog circuits designed to process information, state machines and / or other mechanisms for electronically processing information). The one or more processing devices may include one or more devices that execute some or all of the operations of method 600 in response to instructions electronically stored on an electronic storage medium. The one or more processing devices may include one or more devices configured by hardware, firmware, and / or software specialized to perform one or more of the operations of method 600.

[0099] Operation 602 may include preprocessing and synchronizing a plurality of video streams into a single frame. Operation 602 may be performed by one or more hardware processors set by machine-readable instructions including the same or a similar module as stream preprocessing module 508 according to one or more embodiments.

[0100] [[ID=!2]]Operation 604 may include encoding the synchronized single frame. Operation 604 may be performed by one or more hardware processors set by machine-readable instructions including the same or a similar module as frame encoding module 510 according to one or more embodiments.

[0101] Operation 606 may include packetizing an encoded and synchronized single frame as multiplexed packets for transmission using a plurality of multiplexed packets via a communication network. Operation 606 may be performed by one or more hardware processors configured by machine-readable instructions including the same or similar module as frame packetization module 512 according to one or more embodiments.

[0102] Operation 608 may include depacketizing multiplexed packets received via a communication network to generate a synchronized single frame. Operation 608 may be performed by one or more hardware processors configured by machine-readable instructions including the same or similar module as packet depacketization module 514 according to one or more embodiments.

[0103] Operation 610 may include decoding a synchronized frame into a plurality of video streams for remote monitoring. Operation 610 may be performed by one or more hardware processors configured by machine-readable instructions including the same or similar module as frame decoding module 516 according to one or more embodiments.

[0104] The technology has been described in detail for illustrative purposes based on what is currently considered to be the most practical and preferred embodiments. However, such details are for the purposes used only, and the technology is not limited to the disclosed embodiments, but rather is intended to cover modifications and equivalent configurations within the spirit and scope of the appended claims. For example, in this technology, it should be understood that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

Claims

**Claim 1** A system configured to transmit a plurality of data streams via a communication network for remote monitoring, preprocessing a plurality of video streams into a single synchronized frame, encoding the synchronized single frame, packetizing the encoded synchronized single frame as multiplexed packets for transmission using a plurality of multiplexed packets via the communication network, depacketizing the multiplexed packets received via the communication network to generate the synchronized single frame, decoding the synchronized single frame into the plurality of video streams for remote monitoring A system comprising one or more hardware processors set by machine-readable instructions. **Claim 2** The system according to claim 1, wherein the one or more hardware processors are further set by machine-readable instructions to buffer the multiplexed plurality of packets received via the communication network. **Claim 3** The system according to claim 2, wherein the one or more hardware processors are further set by machine-readable instructions to post-process the buffered multiplexed plurality of packets to generate a plurality of synchronized single frames for remote monitoring. **Claim 4** The system according to claim 3, wherein the post-processing further comprises counting frames. **Claim 5** The system according to claim 3, wherein the post-processing further comprises synchronizing frames. **Claim 6** The system according to claim 1, wherein the one or more hardware processors are further set by machine-readable instructions to simultaneously display a plurality of channels on a single output device. **Claim 7** The system according to claim 1, wherein the one or more hardware processors are further set by machine-readable instructions to encrypt the multiplexed packets before transmission via the communication network, and the one or more hardware processors are further set by machine-readable instructions to decrypt the multiplexed packets received via the communication network. The system according to claim 1. **Claim 8** A method for transmitting a plurality of data streams via a communication network for remote monitoring, preprocessing, at an operating location, a plurality of video streams obtained from individual live video feeds, the plurality of video streams being preprocessed into a single synchronized frame of video data and individual audio streams of audio data, encoding the synchronized single frame, packetizing the encoded synchronized single frame as multiplexed packets for transmission from the operating location to a monitoring location physically remote therefrom using a plurality of multiplexed packets via the communication network, depacketizing the multiplexed packets received via the communication network to generate the synchronized single frame, decoding the synchronized frame into the plurality of video streams and displaying a plurality of channels on at least one output device for remote monitoring at the monitoring location, A method comprising the steps of: **Claim 9** The method according to claim 8, further comprising buffering the multiplexed plurality of packets received via the communication network. **Claim 10** The method according to claim 9, further comprising post-processing the buffered multiplexed plurality of packets to generate a plurality of synchronized single frames for remote monitoring. **Claim 11** The method according to claim 10, wherein the step of post-processing further comprises counting frames. **Claim 12** The method according to claim 10, wherein the step of post-processing further comprises synchronizing frames. **Claim 13** The method according to claim 8, further comprising simultaneously displaying a plurality of channels on the single output device. **Claim 14** The method according to claim 8, further comprising encrypting the multiplexed packets before transmission via the communication network and decrypting the multiplexed packets received via the communication network. The method according to claim 8. **Claim 15** A non-transitory computer-readable storage medium having instructions incorporated therein, the instructions being executable by one or more processors to perform a method of transmitting a plurality of data streams via a satellite communication network for remote monitoring, the method comprising: Preprocessing, at an offshore operating location, a plurality of video streams obtained from individual live video feeds, the plurality of video streams being preprocessed into audio data and video data; Encoding the video data into a synchronized single frame; Packetizing the encoded synchronized single frame as multiplexed packets for transmission via the satellite communication network to a terrestrial monitoring location physically remote from the operating location using a plurality of multiplexed packets; Transmitting the audio data as an audio stream separately from the video data; Depacketizing the multiplexed packets received via the satellite communication network to generate the synchronized single frame; Decoding the synchronized frame into the plurality of video streams and simultaneously displaying a plurality of channels on at least one output device for remote monitoring at the terrestrial monitoring location; A computer-readable storage medium comprising: **Claim 16** The computer-readable storage medium according to claim 15, wherein the method further comprises buffering the multiplexed plurality of packets received via the satellite communication network. **Claim 17** The computer-readable storage medium according to claim 16, wherein the method further comprises post-processing the buffered multiplexed plurality of packets to generate a plurality of synchronized single frames for remote monitoring. **Claim 18** The computer-readable storage medium according to claim 17, wherein the post-processing step further comprises counting frames. **Claim 19** The computer-readable storage medium according to claim 17, wherein the post-processing step further comprises synchronizing frames. **Claim 20** The computer-readable storage medium according to claim 15, wherein the method further comprises the step of simultaneously displaying a plurality of channels on a single output device.

21. Encoding and transmitting video at a remote location by a data transmission protocol that combines a predetermined number of video channels, and combining and compressing data packets to transport as a single frame for transmission by a normal node stream encoding / decoding method via a connection with a limited bandwidth, the computer-readable storage medium according to claim 15, further comprising encoding and transmitting video at a remote location that manipulates video data issued via a network.

22. The computer-readable storage medium according to claim 15, further comprising the steps of converting navigation strings into video overlays, encoding, transmitting, and decoding the video overlays for transport and reconstruction by passing the data to a streaming decoder for input.

23. The computer-readable storage medium according to claim 15, further comprising the step of preprocessing a video stream, obtaining all frames in one cycle at a time using only one encoder, and aligning the timing with corresponding frames, so that the time between single-threaded videos all occurs during the same one or more CPU cycles as synchronous data capture.

24. The computer-readable storage medium according to claim 15, further comprising the step of evaluating motion, referring to previous frames, determining an output optimal for the minimum bitrate, and optimizing the video based on the input feed, wherein the data is packetized using only one descriptor.

25. The computer-readable storage medium according to claim 15, further comprising the step of splitting the video packets into individual streams for playback when the video packets are received, and splitting and playing back so that the user can view any of the video streams on an arbitrary output device with only one-time decoding.

Images