System and method for arrangements of visual information in user-configurable format

Abstract

Presentation and monitoring systems and methods are provided for a user-end unit in wireless communication with one or more remote units. The user-end unit wirelessly receives data, including 360 degree video data and global positioning data, from a remote unit. The user-end unit renders a 360 degree video and a user interface (Ul), including multiple windows, is displayed such that one or more windows, the sizes and positions of which may be modified, may be displayed as overlaid over a window displaying the 360 degree video, the user end unit may additionally monitor the status of one or more elements of the remote unit, including the 360 degree camera, and of the wireless communication between the user-end unit and the remote unit, and information of the status, and interactive options related thereto may be displayed on one or more of the windows of the UI.

Description

SYSTEM AND METHOD FOR ARRANGEMENTS OF VISUAL INFORMATION INUSER-CONFIGURABLE FORMATCROSS-REFERENCE TO RELATED APPLICATION^)

[0001] This Application claims the benefit of U.S. Provisional Application 63 / 563,068, filed March 8, 2024, and is a Continuation-sn-Part of U.S. Application 18 / 140,045, filed April 27, 2023, which Application claims priority to Patent Cooperation Treaty (PCT) Application PCT / US2021 / 0654117, filed December 28, 2021 , which Application claims the benefit of U.S. Provisional Application 63 / 106,964, filed October 29, 2020, the disclosures of which are hereby incorporated herein by reference in their entireties.BACKGROUND

[0002] Field

[0003] Apparatuses and methods consistent with example embodiments relate to virtual reality (VR) command centers and, in particular to systems and methods enabling operatorfs) of remote vehicle teams to monitor those teams from a VR or extended reality head-mounted display (HMD).

[0004] Related Art

[0005] Command center operators are required to maintain situational awareness of a large array of media sources simultaneously. Traditionally command centers have utilized large arrays of displays to allow the simultaneous viewing of information across different media sources including but not limited to dashboards, video feeds, camera feeds, sensor visualizations, and data visualizations. Effective performance of the tasks required of command center opera tors has depended to a large extent upon the operator’s physical presence within the command center to allow viewing of a large array of data sources simultaneously.SUMMARY

[0006] Example embodiments may address at least the above problems and / or disadvantages and other disadvantages not described above. Also, example embodiments are not required to overcome the disadvantages described above, and may not overcome any of the problems described above.

[0007] According to an implementation of an example embodiment, a presentation system comprises: a wireless receiver configured to wirelessly receive, from a remote twit, 360 degree video data and global positioning data; a display unit; and a user-end unit operatively connected to the wireless receiver and to the display unit, the user-end unit comprising a non-transitory storage medium storing instructions and a processor configured to execute the instructions and thereby, render a 360 degree video from a point of view based on the 360 degree video data; create a user interface (UI) comprising a plurality of windows based on the 360 degree video data and the global positioning data, the plurality of windows comprising: a video window displaying the 360 degree video from the point of view, and at. least one overlay window comprising a menu window displaying an interactive settings menu comprising a user- controllable indication of the point of view, and the global positioning data; control the display unit to display the UI; receive user input comprising a change of the indication of the point of view; render an updated 360 degree video from an updated point. of view based on the 360 degree video data and the user input; and control the display unit to display an updated video window comprising the updated 360 degree view.

[0008] According to an example implementation the processor may be configured to create the UI such that each of the at least one overlay window is an overlay displayed over the video window.

[0009] According to an example implementation the at least one overlay window may further comprises a map window displaying a map and an indication on the map of a location of the remote unit; and control of the display unit to display the UI may comprise controlling the display unit to display, in the map window, an automatically updated map and an automatically updated indication based on the global positioning data,

[0010] According to an example implementation the processor may be further configured to: receive user input comprising an instruction to change one of a size and a position of one of the at least one overlay windows to one of a new size and a new position; and control the display unit to d isplay an updated UI disp laying the one of the at least one overlay windows in the one of the new size and the new position,

[0011] According to an example implementation the processor may be further configured to receive user input comprising an instruction to change display of the map in the map window; and control the display unit to display an updated UI displaying a changed display of the map in the map window.[00'12] According to an implementation of an example embodiment, a method of data presentation is provided, the method comprising: wirelessly receiving 360 degree video data and global positioning data from a remote unit: rendering a 360 degree video from a point of view based on the 360 degree video data; creating a user interface (UI) comprising a plurality of windows 'based on the 360 degree video data and the global positioning data, the plurality of windows comprising: a video window displaying the 360 degree video from the point of view;and at least one overlay window comprising a menu windo w displaying an interactive settings menu comprising a user-controllable indication of the point of view, and the global positioning data: controlling a display unit to display the Ul; receiving user input comprising a change of the indication of the point of view; rendering an updated 360 degree video from an updated point of view based on the 360 degree video data and the user input; controlling the display unit to display an updated video window comprising the updated 360 degree view.

[0013] According to an example implementation the creating the Ul may comprise creating the 01 such that each of the at least one overlay window is an. overlay displayed over the video window.

[0014] According to an example implementation the at least one overlay window may further comprise a map window displaying a map and an indication on the map of a location of the remote unit; and the controlling the display unit to display the UI may comprise controlling the display unit to display, in the map window, an automatically updated map and an automatically updated indication based on the global positioning data.

[0015] According to an example implementation, the method may further comprise receiving user input comprising an instruction to change one of a size and a position of one of the at least one overlay windows to one of a new size and a new position ; and controlling the display unit to display an updated IJI displaying the one of the at least one overlay windows in the one of the new size and the new position,

[0016] According to an example implementation, the method may further comprise receiving user input comprising an instruction to change display of the map in the map window; and controlling the display unit to display an updated UI displaying a changed display of the map in the map window.

[0017] According to an implementation of an example embodiment, a presentati on and monitoring system is provided comprising: a wireless transceiver configured to wirelessly communicate with a remote unit comprising a camera; a display unit; and a user -end unit operatively connected to the wireless receiver and to the display unit, the user-end unit comprising a non-transitory storage medium storing instructions and a processor configured to execute the instructions and thereby: recei ve, via the wireless transceiver, a data stream of 360 degree video data from the camera; render a 360 degree video based on the 360 degree video data; create a user interface (HI) comprising a plurality of windows based on the 360 degree video data and the global positioning data, the plurality of windows comprising; a video window displaying the 360 degree video from the point of view, and at least one overlay window comprising a status menu displaying an internet protocol (IP) address of the camera and a status of one of the camera and a wireless connection between the wireless transcei ver and the camera; control the display to display the UI; transmit to the camera, via the wireless transceiver, a status request; determine whether a response to the status request is received; determine an updated status based on one of a received response to the status request and a determination of no response received; create an updated UI based on the updated status; and control the display to display the updated UL[00'18] According to an example implementation, the processor may be configured to create the UI such that each of the at least one overlay window is an overlay displayed over the video window.

[0019] According to an example implemeniation, the status may comprise one of a ping status and a latency of a wireless transmi ssion.

[0020] According to an example implementation, the processor may be further configured to, in response to a change of the status: perform a response operation comprising at least one of: initiating a reconnection with the camera; transmiting an instruction for reconfiguration to the camera; throtling, buffering, pausing, or disabling rendering of the 360 degree video; switching to a mixed reality mode; increasing a bitrate; changing a transmission format.

[0021] According to an implementation of an example embodiment, a presentation and monitoring method is provided comprising: wirelessly communicating with a remote unit comprising a camera; wirelessly receiving a data stream of 360 degree video data from the camera; rendering a 360 degree video based on the 360 degree video data; creating a user interface (VI) comprising a plurality of windows based on the 360 degree video data and the global positioning data, the plurality of windows comprising: a video window displaying the 360 degree video from the point of view, and at least one overlay window comprising a status menu displaying an internet protoco l (IP) address of th e camera and a status of one of the camera and a wireless connection between the wireless transceiver and the camera; controlling a display unit to display the VI; transmiting to the camera, via the wireless transceiver, a Status request; determining whether a response to the status request is recei ved; determining: an updated status based on one of a received response to the status request and a determination of no response received; creating an updated III based on the updated status; and controlling the display to display the updated VI.

[0022] According to an example implementation, the creating the UI may comprise creating the 111 such that each of the at least one overlay window is an overlay displayed over the video window.

[0023] According to an exampl e implementation, the status may comprise one of a ping status and a latency of a w ireless transmiss ion.

[0024] According to an example implementation, the method may further comprise: in response to a change of the status., performing a response operation comprising at least one of: initiating a reconnection with the camera; transmitting an instruction for reconfiguration to the camera; throttling, buffering, pausing, or disabling rendering of the 360 degree video; switching to a mixed reality mode; increasing a bitrate; changing a transmission format.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or other aspects will 'become apparent and more readily appreciated from the following description of example embodimen ts, taken in conjunction with the accompanying drawings in which:

[0026] FIG. 1 is a block diagram of an exemplary system for a command center with the required additions to the system to enable the architecture for remote viewing and control of the video wall via a head-mounted display;

[0027] FIG. 2 is a block diagram of the required communication flows for the control of audiovisual hardware utilizing software capable of rendering a video wall within a head-mounted display;

[0028] FIGS. 3A, 3B, and 3C, and 3 A 1, 3 A2S3B1 , 3B2, 3C lsand 3C2 are diagrammatic block and flow diagram illustrations of various components according to exemplary uses of the disclosed systems and methods;

[0029] FIG. 4 is a diagrammatic illustration of a head- mounted display, host computer, control devices and sensors typical of Virtual Reality Systems that are capable of being utilized with exemplary implementations of exemplary embodiments of disclosed systems and methods;

[0030] FIGS. 5A, 5B, 5C, and-5D are diagrammatic illustration of elements of examples of user interface according to exemplary implementations of exemplary embodiments of disclosed systems and methods;

[0031] FIG. 6A is a diagrammatic illustration of a VR or MR display, tracking and input system capable of being utilized with, or deploying, exemplary implementations of exemplary embodiments of disclosed systems and methods;

[0032] FIGS. 6B and 6C are illustrative examples of various components capable of being utilized in exemplary implementations of exemplary embodiments of disclosed systems and methods;

[0033] FIGs, 7 and 8 show examples of screen captures of a display unit of a Ul trait according to one or more example embodiments;

[0034] FIG. 9A is a schematic illustration of a system according to an example embodiment;

[0035] FIG. 9B is a schematic illustration of another system according to an example embodiment;

[0036] FIG. 10 illustrates a scene representing a system according to one or more example embodiments;

[0037] FIGs. 11, 12, 13, and 14 show examples of screen captures of a display unit of a UI unit according to one or more example embodiments;

[0038] FIG. 15 is a flow diagram of a presentation method according to one or more example embodiments;

[0039] FIG. 16 is a Sow diagram of a presentation and monitoring method according to one or more example embodiments; and

[0040] FIG. 17 shows an example of a screen captures of a display unit of a UI uni t according to one or more example embodiments.DETAILED DESCRIPTION

[0041] Reference will now be made in detail to example embodiments which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the example embodiments may have different forms and may not be construed as being limited to the descriptions set forth herein.

[0042] It will be understood that the terms “include,” “including,” “comprise,” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0043] It will be further understood that, although the terms “'first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections may not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section.

[0044] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.In addition, the terms such as “unit,’’1“-er (-or),” and “-module” described in the specification refer to an element for performing at least one function or operation, and may be implemented in hardware, software, or the combination of hardware and software.

[0045] Expressions of relational orientation, such as “upper;' “lower.” “inside,” “outside,” and / or other expressions, which are used for explaining the structural positions of various components as described herein, are not absolute but relative. The orientation expressions are appropriate when the various components are arranged as shown in the figures, but should change accordingly when the -positions of the various components in the figures change.

[0046] As would be readily appreciated one of skill in the art, while descriptive terms such as “configuration;5“headwall;1“visual,” “virtual,” “integrated;1“screen ,” “headset,” “wearable;1“3d party;1“control ” “encoder,” “decoder;1“hardware;1“software,” “heads-up display;1and others are used herein, these terms are not intended to limit any components that can be used in combinations or individually to implement any of various aspects of one or more example embodiments. Also, example implementations and / or example embodiments described herein may be references as “Topaz” or “TOPAZ” to facilitate understanding Or as a concise point of reference, these terms are not intended to be limiting of any structural and / or functional features of any one or more example embodiments described herein,

[0047] Various terms are used to refer to particular system components. Different companies may refer to a component by different names - this document does not intend to distinguish between components that differ in name but not function .

[0048] Matters of these example embodiments that are obvious to those of ordinary skill in the technical field to which these example embodiments pertain may not be described here in detail.

[0049] The systems and methods described 'herein with respect to example embodiments are related to those described in U.S. Patent Application 18 / 140,045, filed January 2, 2025, published as U.S. Publication 2025 / 0001298, the full disclosure of which is reproduced in foil, below:

[0050] System and method for arrangements of visual information in user-configurable format

[0051] The embodiments described herein relate to utilization of a head mounted display and virtual reality engine to decode and render video streams, transmit control messages to enable control of remote headend hardware and store / recall preset layouts either in synchronization with a physical video wall or as an extension of a physical video wall.

[0052] Current virtual reality systems typically utilize software on the host computer to store, process and play out video content i n the 3D engine and HMD. An example of this architecture is the display of a locally stored MP4 video on a video streaming texture within the 3d environment. Under this architecture the 3d engine accesses a file on the host computer and renders it in the 3d engine to be viewed in the HMD. Another method for the display of video within a 3d engine from a remotely hosted conten t source, requires the transfer of data from a server to the host where it is locally buffered and played out in the 3d engine. In each scenarios a host computer, and in some implementations a server, must be loaded with software that is allowed access to the display drivers and file structures of a computer which might contain sensitive information. Such implementations are not practical for secure command center operations. Typically display devices used for ths viewing of secure content must not have software packages installed that enable them to access, transmit or store sensitive data. Further display devices must be physically separated from networks with connected devices that host sensitive infonnatian. Current VR. implementations that allow screen sharing and video playout.tail to provide a secure method to enable integration with traditional command center infrastructure based on technical requirements for access to files on the host or access to networks containing sensitive information. The current industry accepted secure command center video display implementation relies on hardware architectures consistent with the exemplary diagram shown in FIG. 1 inclusive only of devices 101, 102, 103, 104, 105, 106, 107, 108, 109 and 115. Such architectures by design, mitigate the possibility of data spillage. Any VR / AR solution implemented in a secure environment must act as a display device that maintains logical and physical separation from networks or hosts containing sensitive information and must not store the video images it displays.

[0053] The embodiments described herein approach these problems from a different perspective. Instead of utilizing the host computer and HMD as repository for content or a. host client that accesses data from a server, the system utilizes the system exclusively for content playout of real time streams of video in the same manner that a stateless display device displays content, but neither stores nor accesses the source of the content.

[0054] Moreover, exemplary embodiments of the disclosed system and methods allo w users of the system to receive and view video streams from computers and other video sources while being physically disconnected from the network that the source computers and devices generating that video are connected to, providing an enhanced level of segmen tation and security.

[0055] The method used to allow operators of the disclosed system to route and switch video sources as well as control hardware devices is handled via an integrated API which is compatible with existing hardware control system processors such as the C'restron CP4, Extron IPCP, AMXNetlinks, Control 4 or any other control system processor that allows communication via IP protocols.

[0056] In this way; access to routing, switching. Camera PTZ, VTC device control and other hardware control capabilities of the system are subject to the permissions granted by die control system processor that manages control messages for the command center hardware. In an exemplary implementation, the disclosed system communicates only to the control system processor and never directly to a device, again providing an enhanced layer of compartmentalization and security,

[0057] Exemplary embodiments described herein provide technologies and techniques for using a 3D engine and VR / AR capable HMD to reproduce and arrange video images and process control messages to and from the hardware devices typically controlled by an AV Control System processor such as Video Teleconferencing Hardware, Pan-Tilt-Zootn robotic camera systems. Video Switching infrastructure. Video Processing hardware. Lighting systems, Building management systems, displays and any other device with an API, relay control, GPIO and Logic IO.

[0058] In other disclosed exemplary embodiments, systems and methods are provided for the implementation of various configurations and use eases which may be implemented utilizing the disclosed methods and system. Examples of such implementations include the provisioning of multiple VR / AR HMD systems that share the same unicast streams providing mirroring of content across all media streaming textures within the 3D engine, multiple VR / AR HMD systems that each receive their own unique unicast video streams allowing indi vidual unique content to be displayed in each HMD and variations of the system where the VR / AR HMD system is either collocated in the command center where the video sw itching and streamingencoders installed or where the VR / AR HMD is remotely located and connected via a secure encrypted VPN, GRE tunnel or other TCP / IP protocol.

[0059] Exemplary embodiments described herein further can include systems and methods for generating a virtual reality environment, wherein the virtual reality environment contains one or more three-dimensional virtual controls; providing the virtual reality environment for display, through a virtual reality device; obtaining input forough manipulation of the one or more virtual controls, wherein the manipulation is made using the virtual reality device and the manipulation follows at least one movement pattern associated with the one or more virtual controls; determining, based on the input, content of foe one or more virtual controls, and the movement pattern associated with the manipulation, changes to the virtual environment wherein the changes are reflective of the manipulation of the one or more virtual controls; and providing the changes to the virtual reality device for display,

[0060] FIG. 1 is a block diagram of an exemplary system according to exemplary embodiments including devices / components / niodules / functions 101 , 102, 103, 104, 105, 106, 107, 10S, 109, 110 and 115 representative of those included in a command center AV architecture to which the disclosed system can be attached to enable extension of foe display wall 115 to a be viewable in a VR headset. The architecture can be scaled to support an unlimited number of input devices (101, 102, 103, 104, 105, 106) and an unlimited number of output de vices (115) depend ing on the requirements of the system, where modules 1.01. are COTS personal computers with graphics cards, installed (typically HDM1 or USB outputs would be available), module 102 is a COTS USB KVM transmitter that connects to a matrix switching headend. Module 101 would typically be connected to module 102 using an HDM1 or Display Port cable and a USB cable depending on manufacturer. Module 103 is a non~KVM video sourcesuch as a COTS CATV receiver 104 is an HDMI transmitter which is connected to a video mafrix switching headend, Module 103 and receiver 104 would typically be connected via an HDMI, Display Port or SDI cable. Module 102 and recei ver 104 are connected to COTS Network Switch or Video .Matrix Switcher 107 using multimode or single mode fiber or CATx cabling dependent on the application.[00611 System 100 inclusive of devices 101, 102, 103, 104, 105, 106 and 115 comprises a standard architecture for a command center. According to exemplary implementation, for the enablement of the disclosed VR / AR and HMD extension of the system the following systems and methods can be implemented,

[0062] Additional outputs of 110, video wall processor, should be appropriately configured to enable routing of one or many sources to each of the outputs connected to 112 IP Encoder, Connection between 1 If) and 1 12 can be any video format that shares compatibility between 1 10 and I 12, Common implementations will include HDMI, 12G SDI, Display Port and DVI for connection between 110 and 112, In an exemplary implementation, it is preferable that the output of 110 and the input of 112 be configured such that the maximum resolution per ip encoder be delivered from the video wall processor 110 to the IP Encoder 1 12. This can enable the maximum amount of video information to be transmitted to the VR / AR engine per stream.Multiple outputs of 1 10 may be connected to multiple separate instances of 112 within the same system. The process of video encoding from a baseband or HDMI signal type to and IP encoded video stream occurs utilizing the COTS IP video encoder device 112.

[0063] IP Video Encoder devices 112 will encode video signal to an IP streaming protocol which may be either Unicast or Multicast depending on the appl ication. A wide variety of codecs may be utilized with the system, compatibility with decoding software module, as described inthe exemplary implantation of Appendix C (set forth bel ow), may be required. As new video streaming codecs are established in the market, updates to the encoding capabilities of the software module, for example as described in appendix C, may be required, however exemplary methods of implementation of the system may remain unchanged. According to an exemplary implementation, the only modification that may be required to support any new codecs would be the expansion of the decoding module, such as on described in examp le of Appendix C , to include said codec,

[0064] IP Video Encoders 112 can be connected to a COTS network switch device 114 using standard Ethernet protocol Device 114 can be either a single switch or a LAN composed of multiple switches, routers, servers and hosts as required. Upon egress from the LAN for transmission across the public internet or other wide area transmission that may be intercepted by an adversary of IP Encoders 112, device 113 hardware encryption device or VPN appliance may be inserted. Under this architecture a decryption device or VPN device 1 13 would be inserted at the point of ingress back to a physically and logically secured LAN environment. Connection from 113 at the ingress point would then be typically connected to .1 14 network switch lor distribution of data inside the LAN. According to an exemplary embodiment, the principal of this portion of the system is that hardware encryption devices 113 can be implemented as a part of the system where the signals encoded by 1 12 require encryption and transmission in a secure manner.

[0065] According to exemplary embodiments, COTS PC with VICAR Peripherals, device 116 can be connected to network switch 114 via standard Ethernet .fiber or copper cabling. Ip network streams sent from devices 112 are received, processed, and played out. in the 3d Engine hosted on device 116. Playout within the 3D engine can be handled using the method such asthose described in Appendix C or by any other means sufficient to render the video to a texture within the 3D engine. Ip streaming video is processed, decoded, and recomposed into a video image. The video image is rendered and displayed on a 2-dimensional surface texture- witbin the 3d environment. The texture is commonly referred to in COTS 3d engines as a media streaming texture. An example of media streaming texture is sho wn in FIG. 5C, display 301.

[0066] According to exemplary embodiments, arrangement and positioning of individual source video content from within the pixel space of the video encoding device 112 can be managed by the video processing external hardware device 110. For example individual inputs sources 101, 103, 105 and 106 can be arranged within a single video stream based on the windowing configuration applied in device 110. The result of the arrangement of input sources by device 110 can be observed in FIG. 5B where individual media streaming textures can contain one image or many images.

[0067] According to an exemplary implementation of disclosed embodiments, a critical component of the disclosed system is control of the external video switching and routing hardware .from within the 3d engine using COTS control devices such as device 385 depicted in a non-limiting example of FIG. 6C. In traditional command center designs, routing, switching, video processing and device control is managed by a control system processor, device 108. An example of an API described in Appendix A provides a method for the communication of control commands between the COTS PC .1 16 and control processor 108. By utilizing a. virtual pointer to select user interface components within the 3D engine as depicted in the examples of FIGS. S A, 5B, 5C, and 5D it is possible to control video source-destination routing, video wall layouts and arrangements, USB-KVM .routing, audio routing, robotic camera control as well as preset arrangement storage and recall.

[0068] FIG. 4 shows an example of a screen capture of the rotary control UI utilized to enable various modes of control and content viewing within the VR environment, according to exemplary implementations f disclosed embodiments. A rotary menu 401 can be operated from a typical COTS VR controller. A buton 402 can be provide to allow the user io toggle between VR and Augmented reality modes using video pass through cameras to superimpose all menu and video features over a live video feed from cameras mounted on the headset. This can be typically referred to as Augmented Reality or Extended Reality. A preset window launch buttonv403 can also be provided such that, for example, pressing control button 403 causes the preset control menu 505 to show and hide within the environment. A magnification window launch buttonv404 can also be provided such that pressing the button 404 will pop up magnification window 501. A source selection page popup button 405, can also be provided such that pressing button 405 will pop up source selection control menu 505. A rotary selector 406 can be controlled for example by placing the users thumb over the rotary' selection button on a COTS VR controller.

[0069] FIGS. 5A-5D are screen captures showing exemplary implementations of embodimen ts of the disclosed system user interface as viewed in a VR headset as device 375 depicted in a nonlimiting example of FIG. 6B. In an exemplary implementation, a magnification window can be conceptualized as a display in the virtual environment. The magnification window 501 is a virtual object in the 3d environment with a media streaming texture applied to the object. The media streaming texture is connected logically in software such as to ffmpeg software plugin as set forth in the example of Appendix C. In the example of Appendix C, the plugin has a stream url field that can be user defined. When a stream URL is present at the defined address the magnification window will then display video on the media streaming texture.

[0070] According to exemplary implementations of various embodiments of the present disclosure, there can be provided a positioning control 502 for the magnification window. The magnification window can be positioned in the 3d environment by selecting the positioning control bar 502 and dragging the window in the 3d space. Controls have been enabled for X,Y,Z positioning of the magnification window m the coordinate plane of the 3d environment.

[0071] According to exemplary implementations of various embodiments of the present disclosure, there can be provided a close window button 503 which allows the magnification window to be closed, hiding it from view in the 3d environment.

[0072] According to exemplary implementations of various embodiments of the present disclosure, there can be provided a source control menu 504. This menu contains sources configured via a web application hosted on COTS control processor 108. The sources shown on this UI element are representative of physical video inputs to the system as shown in 101, 103, 105 and 106, The naming and configuration of sources is executed via a web browser. In the V R application, the user is able to select a source as shown, in 505 and route it to a destination as shown in 507, In so doing, a command is sent from COTS PC 116 to processor 108 utilizing API such as in the example of Appendix A, a physical video source 102 is switched at the video matrix switcher 107 to an input of the COTS video wall processor 110, video outputs of the processor 1 10, video wall processor are physically connected to the IP video encoders 112 al the stream URLs defined by the user during system setup. IP video streams are decoded and displayed in in the 3d environment on the media streaming texture.

[0073] According to exemplary implementations of various embodiments of the present disclosure, there can be provided a source selection button 505. Source selection buttons are representative of physical video sources connected to the system. By selecting a 505 source andthen selecting a virtual display 507', API' commands are sent to the hardware devices as shown inFIG, 2 such that physical video routes are executed resulting in the video being displayed in the 3d environment

[0074] According to exemplary implementations of various embodiments of the present disclosure, there can be -provided a preset button 506. Presets are storable and recallable configuration methods by which a user can store all information related to source / destinafion routing, video wall processor setings and x,y,z positioning of windows in the 3d environment. A preset is stored by clicking and holding for 3 seconds on a typical 506 type buton, The software will prompt with a message confirming t he user wishes to overwrite the particular preset after the button is held for 3 seconds. When the preset button is clicked and released in under 3 seconds, all parameters stored in that preset are recalled and displayed in the 3d environment. All parameters related to presets are stored on the COTS Control processor 10S to prevent any data related to source / destination routing or windowing from being stored on the PC 1 16.

[0075] According to exemplary implementations of various embodiments of the present disclosure, there can be provided a virtual display 507 with media streaming texture applied. The virtual display is an object in the 3d environment, the media streaming texture is a software component that enables playout of video as a texture applied to an object,

[0076] FIG. 5D, illustrates an exemplary image 508 of the application of a video wall processor for windowing of display sources within a single stream according to exemplary implementati ons of vari ous embodiments of the present disclos ure.

[0077] According to exemplary implementations of various embodiments of the present disclosure, there can be provided a windowing control button 509 that allows control of the 110, video wall processor, associated with the display 507, virtual display with media streamingtexture applied. By selecting this control button in the application, a command can be transmitted as depicted in FIG. 2 which results in a modification of the tiling of video sources at the output of the 110 video wall processor. The result is a change in the arrangement of sources shown on a virtual dispIay507.

[0078] FIG. 5B illustrates an exemplary image 510 of the specialty controls available lor sources defined as compatible with robotic pan tilt zoom control parameters for cameras according to exemplary implementations of various embodiments of the present disclosure. When a camera source is routed to a virtual displaySO?, controls are displayed to enable the user the ability to send ptz control messages to a type 105 device capable of robotic or cropping based ptz control.

[0079] FIG. 5B further illustrates an exemplary image 511 of the specialty control open / close parameter to enable ptz control buttons to be displayed or hidden over the virtual display 507 media streaming texture of a compatible routed source according to exemplary implementations of various embodiments of the present disclosure.

[0080] FIG, 6A is a diagrammatic illustration of an example of virtual reality (VR) of mixed reality (MR) display, tracking and input system including a PC 116 driving AR VR HMD, and COTS VR / MR HMD 1 17 and controller 385, with an external interface emitter 9999 in communication therewith.

[0081] While the present disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made dierein without departing from the spirit and scope of the embodiments of the present disclosure.

[0082] For example, US patent application Pub. No. US 2018 / 0082477 Al dated Mar. 22,2018, the entire disclosure of which is incorporated herein by reference, contains examples of conventional VR, systems, where for example, FIGS. 3 A, 3B and 3C, and 3AI, 3A2, 3B 1 , 3B2, 3C1 , and 3C2 of Pub. No. US 2018 / 0082477 Al .illustrate components of a VR system of the type that can be used complimentarily with, or improved by, exemplary embodiments described in this disclosure.

[0083] The compon ents of the illustrative devices, systems and methods employed in accordance with the illustrated embodiments can be implemented, at least in. part, in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or in combinati ons of them. These components can be implemented, for example, as a computer program product such as a computer program, program code or computer instructions tangibly embodied in an information carrier, or in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus such as a programmable processor, a computer, or multiple computers.

[0084] Exemplary non-limiting implementations of embodiment s of the present disclosure are further described in the enclosed Appendices A-C, which are included in, and made part of, the present disclosure, to aid still further in the description of exemplary technology associated therewith, where:

[0085] APPENDIX A provides an exemplary API Reference document demonstrative of the required control command set and possible syntax protocols between the VR / AR Command center application and the connected hardware control server.

[0086] APPENDIX B provides an exemplary source code for control system communication with VR / AR Command Center application as well as external hardware that can be controlled via the VR / AR Command Center user from within the AR / VR Command Center application

[0087] APPENDIX C provides exemplary Diagrams and descriptions of FFMPEG Optimization for VR / AR utilization in the VR / AR Command Center application.

[0088] Related art VR systems typically utilize software on a host computer to store, process, and play out video content in a 3D engine and HMD. An example of this architecture is the display of a locally stored MP4 video on a video streaming texture within the 3D environment. Under this architecture the 3D engine accesses a file on the host computer and renders it in the 3D engine to be viewed in the HMD. Another method for the display of video within a 3D engine from a remotely hosted content source, requires the transfer o f data from a server to the host where it is locally buffered and played out in the 3D engine. In each of these scenarios, a host computer, and, in some implementations a server, must be loaded with software that is allowed access to the display drivers and file structures of a computer which might contain sensitive information. Such implementations are not practical for secure command center operations. Typically display devices used for the viewing of secure content must not have software packages installed that enable them to access, transmit or store sensitive data. Further display devices must be physically separated from networks with connected devices that host sensitive informaton. Current VR implementations that allow screen sharing and video playout, fail to provide a secure method to enable integration with traditional command center infrastructure based on technical requirements for access to files on the host or access to networks containing sensitive infonuaiion.

[0089] One or more example embodiments described herein may provide technologies and techniques for using a 3D engine and VR.'AR capable HMD to reproduce and arrange video images and process control messages to and from the hardware devices typically control led by an audio visual (AV) control system processor such as video teleconferencing hardware, pan-tilt- zoom robotic camera systems, video switching infrastructure, video processing hardware, lighting systems, building management systems, displays and any other device with an API, relay control, GFIO and logic input / output One or more example embodiments maybe designed to preserve awareness of a physical workspace by displaying on one or more extended reality HMDs that display a user’s surroundings via optical or video see through camera devices. Thus, one or more example embodiments may enable blending of a physical display with remote information from 360 degree cameras and / or Internet of Things (loT) sensors,

[0090] One or more example embodiments may enable one or more of various features and / or functions on a remove vehicle (i.e, unmanned craft), including, but not limited to: high fidelity, low latency 360 degree video; wireless video reliability and monitoring tools; Global Positioning System (GPS) and / or other location formats referenced to maps and / or other reference points; bearing, heading, elevation, and / or other positional data of the unmanned craft displayed as a heads-up display (HUD) overlay; and loT data, such as battery and / or fuel level or other internal conditions of the unmanned craft, motor speed, etc.

[0091] One or more example embodiments described herein may bring one or more of these informational elements into a three dimensional (3D) VR environment One or more example embodiments may include a HMD enabling a user to ha ve a. small logistic and spatial footprint while the 3D nature of the data and display con vey to the user an understanding of spatial relationships and situational awareness from the perspective of an unmanned craft.

[0092] One or more example embodiments described herein may provide a 360 degree video enabling a user to have awareness of the immediate vantage point of an unmanned craft. For example, one or more example embodiments may enable a user to observe aft of the craft and to thereby verify, for example, deployment of a sensor or ordinance that is ejected from the craft, and to then, immediately view in a forward direction to thereby ascertain threats and / or obstacles that may be in the way of the craft. According to one or more ex ampl e embodiments these functions may be achieved without moving mechanical parts or die latency associated with joystick or other mechanical maneuvering of cameras.

[0093] According to one or more example embodiments described herein, the reliability of transmission of 360 degree video over wireless networks) may be addressed. For example, rendering architecture may be configured to function with any 360 degree camera thereby enabling operation in conjunction with lightweight and / or ruggedized camera platforms,

[0094] According to one or more example embodiments described herein, a system may enable a user to configure and control input(s) in one or more concealable menu panels. For example, according to an example embodiment, and as shown in FIG. 7, a system may include features of monitoring the health of streamed data via text and / or color indicators displayed to a user as “reachable” in. conjunction with an indicator, for example, a green diamond, indicating the stream health of both video and sensor feeds. One or more display options may include, but are not limited to, an ability to pan, tilt, and yaw a spherical image to provide an optimal viewing perspecti ve from the point of view of an unmanned craft. Ln other words, a default perspective may provide fore, aft, starboard, and port perspectives or some blend thereof. According to an example implementation, a display option may additionally include a bearing or heading compass to thereby enable a user to maintain cardinal direction awareness.

[0095] According to one or more example embodiments described herein, a system may enable vehicle locations in the format of GPS coordinates to be referenced to third party mapping systems. As shown in FIG. 8, a current location of a vehicle can be shown while a 360 degree video is being displayed, and the map location of the vehicle .may be displayed in an overlay that is capable of being manipulated by a user. According to an example implementation, the overlay may been delivered as a Bing Map (Microsoft) or ArcGlS Map (ESRI), but may alternately be con.ured for any map or space including user supplied point cloud data, for example.

[0096] According to one or more example embodiments described herein, a system may provide a display of vehicles on a three-axis map, including coordinates and elevation, thereby enabling a user to see vehicles in relation to one another and the landscape. One or more example implementations may facilitate a user’s understanding of relationships between and among vehicles and topography in a manner that is both quick and intuitive.

[0097] One or more example embodiments described herein provide methods and systems for facilitating large scale and arrangements of visual information in a user-configurable format within a head mounted display (HMD). An HMD may be referred to herein by the descriptive, non-limiting term “Headwall” for purposes clarity and conciseness.

[0098] One or more example embodiments described herein may relate to utilization of an HMD and VR engine to decode and render video streams, transmit control messages to enable control of remote headend hardware and store / recall preset layouts either in synchronization with a physical video wall or as an extension of a physical video wall.

[0099] Example System

[0100] FIG. 9A is a schematic illustration of a system according to an example embodiment.The system 200 includes a remote unit 200 in wireless communication with a user-end unit 300connected to a user interface (UI) unit 450. FIG. A illustrates a single remote unit 200, a single user-end unit 300, and a single U1 unit 540, However, as would be understood by one of skill in the art, the archi tecture of the system 500 may be scaled to support multiple remote units 200, multiple user-end units 300, and multiple III units 450, and may include any one or more of devices 101, 102, 103, 104, 105, 106, 107, 108, 109, and 115, as discussed with respect to FIG. 1.

[0101] FIG. 9B is a schematic illustration of another system according to an example embodiment.

[0102] Remote unit - The remote unit 200 may be, for example, but is not limited to, a remote vehicle or other craft and may include one or more of a camera 210, a GPS uni t 220, one or more other sensors 230, and a wireless transmitter 240. The camera 210, GPS unit 220, and one or more other sensors 230 are operatively connected to the wireless transmitter 240 configured to transmit data to the user-end unit 300.

[0103] Camera ~ The camera 210 of the remote unit 200 may be any commercially available camera configured to capture images and therefrom create a 360 degree field of view. The camera 2'10 may have 4k resolution in an equirectangular frame, for example, and may obtain video images from about 15 to about 60 frames per second. However, these features of the camera 210 are merely examples and the camera may have any Configuration as would be understood by one of skill in the art. An encoder (not shown) included, for example, within the camera 210 or as an external unit hardware device may be configured to encode video from the camera XX into a staudards-based format such as, but not limited to RTSP, RTP, RT.MP, and may output a 4k stream using, for example, UDP, TCP, or HLP depending on system requirements. It should be noted that certain 360 degree cameras may not support RTSP. RTMPmay be used, for example, between about 10-25 MPBS and may provide the least latency for certain cameras, for example, the Instapro camera. The 4k stream may be wirelessly output from the wireless transmitter 240,

[0104] GPS ~ The GPS unit 220 of the remote unit 200 may be a GPS beacon,

[0105] Wireless recei ver - A wireless receiver 550 is operatively connected to a user-end unit 300 and is configured to recei ve transmission from the wireless transmitter 240. The wireless receiver 550 is configured to receive the 4k stream transmitted from the remote unit 200 and to transmit the stream to the user-end unit 300, for example to a decoder therewithal, With respect to GPS data, the wireless receiver 550 may receive data directly, via a GPS unit 220 of the remote unit 200 or may receive data via an intermediary server using, for example, textual requests and responses over a socket. For example, a request for GPS data can first be written to a designated socket on a service daemon on a GPS device. Time-Position- Velocity reports (TPV) may then be read from the GPS deviee / server at every measurement epoch, Responses can be in the format of, for example, but not limited to GPS built on top of JavaScript Object Notation (JSON). With respect to data from any one or more sensors 230, the wireless recei ver 550 may likewise receive data directly from the one or more sensors 230 via the wireless transmitter 240 or may receive sensor data via an intermediary server using, for example, textual requests and responses over a socket.

[0106] User-end unit / Topaz - According to this example embodiment, the user-end unit 300 may be referred to as a Topaz unit and may include the wireless receiver 550 configured to teceise data turn the temule nun 2oo \ltcrnatels the u ueless teccwet 55o max be evtctnal to and operatively connected to the Topaz uni t 300, The Topaz unit 300 may include a decoder and a rendering engine and may be operatively connected to the UI unit 450. The Topaz unit 300may be configured to operate, for example Windo ws 10 Pro or Windo ws 11; may utilize LA V fillers, for example ffmpeg-based DireetShow Splitters and Decoders; and may utilize Varjo Base v3.10.0.6, for example.

[0107] According to one or more example embodiments, tile Topaz unit 300 may be embodied by a controller. The controller can be implemented, at least in part, in digital electronic circuitiy, analog electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. These components can be implemented, for example, as a computer program product such as a computer program, program code or computer instructions tangibly embodied in an. information carrier, or in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus such as a programmable processor, a computer, or multiple computers.

[0108] The computer program product may comprise a computer-readable media storing program instructions, executable by a processor, to implement various operations. The media may include, alone or in combination with the program instructions, data files, data structures, and the like. The media and program instructions may be specially designed and constructed for the purposes of example embodiments described herein, or they may be of a kind well-known and available to those of skill in the art. Examples of computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVD; magneto-optical media such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as readonly .memory (ROM), random access memory (RAM), flash memory, and the like. The media may also include a transmission medium such as one or more of optical lines, electrically- conducti ve lines, and wave guides. Examples of program instructions include, but are not limitedto: machine code, such as produced by a compiler; and files containing higher level code that may be executed by the controller using an interpreter. Hardware devices described herein may be configured to act as one or more software modules in order to perform the operations of one or more example embodiments described herein.

[0109] A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or other device or on multiple device at one site or distributed across multiple sites and interconnected by a communication network. Also, functional programs, codes, and code segments for accomplishing features described herein can be easily developed by programmers skilled in the art. Operations associated with the example embodiments can be performed by one or more programmable processors executing a computer program, code or instructions to perform functions (e.g., by operating on input data and / or generating an output). Operations can also be performed by, and apparatuses described herein can be implemented as, special purpose logic circuitry, e.g., a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), for example,[0'110] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an ASIC, a FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein, A general purpose processor may be a microprocessor, but in the alternative, the processor may be any con ventional processor.controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0111] A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of comp uting devices, e.g., a combination of a DS P and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0112] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory , including by way of example, semiconductor memory devices, e.g., electrically programmable read-only memory (ROM) (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magnetooptical disks, and CD-ROM and DVD-ROM disks). The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.

[0113] Computer-readable non-transitory media includes all types of computer readable media, including magnetic storage media, optical storage media, flash media and solid state storage media. It should be understood that software can be installed in and sold with a central processing unit (CPU) device. Alternately, software can be obtained and loaded into the CPU device, including obtaining the software through physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator, 'The software can be stored on a server for distribution over the Internet, for example,

[0114] The control system may be communicatively and operationally connected to the monitor system via wired or wireless connections either to the monitor system as a whole, or separately to individual elements or groups of elements of the monitor system. Wireless connections may be, for example, via antennas, Bluetooth or Bluetooth Low Energy (BLE), Neat-Field Communication (NFC), or another means, as would be understood by one of skill in the art.

[0115] Regarding sensor data, for example, the Topaz unit XX Topaz may read text and / or json data from one or more sockets on sensor events, for example every measurement epoch, ar may request, data from a REST / SOAP server. For example, the text, and / or JSON responses may be parsed to obtain measurement and or other status data and may be used to update data of a state of one or more virtual objects in the user’s environment as provided by the UL GPS and or sensor requests may occur automatically by default, may be manually initiated, or may be configured on-the-fly.

[0116] According to one or more example embodiments. Topaz 300 may include a Decoder and a Rendering engine.

[0117] Monitoring system - In addition to the above, according to one or more example embodiments, the Topaz unit 300 may include a monitoring system be configured to check the system health of the remote unit 200 including the camera 210, GPS unit 220, and other sensor elemen ts 230. Topaz 300 may also be configured to determine a health o f t he wireless data stream from the remote unit 200, for example, to obtain a ping status and / or a latency of the wirel ess transmission. The health of the remote un it 200 and of the wireless transmission may each be obtained automatically according io a fixed or adaptive schedule or may be manually initiated. When, an unreliable connection or an issue with one or more of the elements of the remote unit 200 is detected. Topaz 300 can output an alert to a user via . The alert may be in the form of one or more status indicators on the HI, In response to detection of an issue with one or both of the wireless connection and the remote unit 200, Topaz 300 may control the U I un it 450 to enable control features for a user to manually initiate reconnection, or reconfigure camera and sensor sendees on the fly. In response to detection of an issue with one or both of the wireless connection and the remote unit 200, one of various strategies may be selected, either automatically by the Topaz unit 300 or manually by a user via the UI unit 450, The strategies may include, for example, operations to optimize available bandwidth usage in consideration of the rendering overhead of a virtual environment provided by the UI unit 450 and a current priority. The operations may include, but are not limited to, for example: throttling, buffering, pausing, disabling video rendering, and switching to mixed reality mode in low bandwidth mode; increasing bitrate and / or changing a transmission format to thereby adapt to a high signal strength, or at a critical stage of a mission; and pausing sensor updates and visuals at a last known value while initiating reconnection attempts.

[0118] U I unit

[0119] FIG. 10 illustrates a scene representing a system according to one or more example embodmients.

[0120] The UI unit 430 may include a display unit and a user control unit. The display unit and user control unit may be integrated, as in a headset device, or may be physically separate units operatively connected to each other and to the Topaz unit 300.

[0121] The display unit may include any one or more of a headset device; a display of a handheld unit such as a mobile phone or tablet; and a fixed display screen (see FIG. 10).

[0122] The user control unit may include any one or more of a hand-tracking controller; mouse; gloves; sensors configured to determine hand and finger positions and gestures such as pinching, gripping, poking, and palm indications; joystick; keyboard; and any other user control unit as would be understood by one of skill in the art. Any one or more of the elements of the user control unit may be wired or wirelessly connected to the display unit and / or the Topaz unit 300,

[0123] The UI unit 450 may support any of various modes of control and content viewi ng.Content and menus may be provided using video pass through cameras to superimpose content and menu features over a camera feed. The Topaz unit may control the UI unit to enable a user to: toggle between VR and Augmented reality (AR) modes using video pass through; to pop up a magnification window or any one or more of various menus; use hand motions to perform virtual near or distant “grabs” to pick up, move, and / or scale slates. Topaz 300 may receive streaming video information and GPS data and may use the streaming video information and GPS date in combination to display video and / or a virtual environment following a GPS target or control the display unit.

[0124] FIGs. 7, 8, 11 -14, and 17 show examples of screen captures of a display unit of a UI unit 450 according to one or more example embodiments. As shown in FIG. 1 I , a Topaz unit 300 may control the UI unit 450 to enable monitoring of the health of streamed data via text and / or color indicators displayed to a user as “reachable” hi conjunction with an indicator, for example, a green diamond, indi cating the stream health of both video and sensor feeds. One or more display options may include, but are not limited to, an ability to pan, tilt, and yaw a spherical image to provide an optimal viewing perspective from the point of view of an unmanned craft. In other words, a default perspective may provide fore, aft, starboard, and port perspectives or some blend thereof According to an example implementation, a display option may additionally include a bearing or heading compass to thereby enable a user to maintain cardinal direction awareness. As shown in FIGs. 5 and 9, a current location of a vehicle can be shown while a 360 degree video is being displayed, and the map location of the vehicle may be di splayed in an overlay that is capable of being manipulated by a user. According to an example implementation, the overlay may been delivered as a Bing Map (Microsoft) or ArcGIS Map (ESRI), but may alternately be configured for any map or space including user supplied point cloud data, for example.

[0125] Example Embodiments - Operations of User-End Unit / Topaz Unit and UI Unit

[0126] The monitoring system of the Topaz unit 300 may control the UI unit 450 to display information of a status of any one or more elements of the remote unit 200. For example, as shown in FIG. 11, a Status tab (shown in the upper left) may be displayed showing an internetprotocol (IP) address for a camera and GPSD daemon tha t Topaz 300 is trying to reach. The status may be shown as “Reachable” with a green or other color diamond or other indicator shape or as “Unreachable” with a red or oilier color diamond or other indicator shape. A Settingstab may also be displayed for the user. The Settings tab may include text input boxes, for example for Camera IP and GPS 1'P, and a user changing the GPS IP address in the text inp ut box may automatically switch a map pin to the location of a Silvas radio or other GPS-enable unit with that IP. According to an example implementation, Topaz 300 may be configured such that a user does not need to press any Enter key when updating the IP address. According to an example implementation. Topaz 300 may be configured such that when a user changes the camera IP address however, the user may be required to manually click “Refresh Stream” on the Settings tab to switch a displayed camera view to the new IP, If the camera is not online at the time ofclicking the refresh buton, the Topaz unit 300 may automatically perform tsp to a predetermined number of attempts, for example five attempts, to refresh and search for the stream. In such a case, the display may show, for example “Refreshing Stream (Attempt x / n).”

[0127] According to an example implementation, Topaz 300 may be configured to control the UI unit such that a user can move XYZ sliders, use 47- buttons, or enter a desired integer value (see FIG. 11) to rotate the displayed camera stream, view. Accordingly, the user may use this function to reorient a tilted camera, for example.

[0128] According to an example implementation, Topaz 300 may be configured to control the 111 unit.450 such that a user can set a height of the compass as displayed to be at a comfortable level (e.g. at eye-level or j ust above) depending on the user’s preference by using the “Height” slider (See FIG. 11). The user may also reset the bearing 000 to a current orientation of the user ’s (e.g. a headset’s) forward by pressing the related button.

[0129] According to an example implementation, Topaz 300 may be configured to control the UI un it 450 to pro vide an extended reality (XR) -.interface in which a displayed map forms a center of interaction for die user. The map may indicate a single map pin following thecoordinates of a tracked asset with an associated 360 video stream. This asset can be configured by updating the GPS and Camera IP addresses. A button with a camera icon on the map pin can be pressed by the user, for example, via a hand interaction to toggle and display the 360 video feed from the asset around the camera. The U! unit 450 thus enables the user to experience an on- location real-time poiat-of-view of the camera. Topaz 300 may be configured to control the UI unit 450 to provide a palm menu and map interaction options for operational convenience.

[0130] FIGs. 10, 11 , and 12 shown examples of screen captures of a display unit of a UI unit 450 providing a palm unit according to one or more example embodiments. According to an example implementation, Topaz 300 may be configured to control the UI unit 450 to provide a palm unit. The user may hold out either hand with palm facing the user’s face to bring up a palm menu in XR. This fimctionality may provide the user with options such as, but not limited to: zooming the map in / out by one level per button press; recentering the XR space so the map is in front of the user; toggling the visibility of the map; and ex iting one or more appl ications,

[0131] According to one or more example embodiments, as shown in FIG, 12 and 14, Topaz 300 may enable the user to move or scale the displayed map using a transparent black bounding ring around it. For example, pinching the ring with a single hand and moving it may allow the user to move the map; pinching the ring with both hands and then pulling the hands apart or towards each other may allow the user to scale the map to be larger or smaller.

[0132] According to one or more example embodiments, as shown in FIG. 1.3, Topaz 300 may enable the user to press the button with the camera icon on the map pin to thereby allow the user to toggle between a mixed-reality view and a 360 video stream.

[0133] FIG. 15 is a flow diagram of a presentation method according to one or more example embodiments. In a system according to one or more example embodiments described here, apresentation method may include operations of: receiving wireless data 13-1; rendering a 360 degree video 13-2; creating a displaying a UI 13-3; automatically updating the UI 13-4 and / or receiving a user input 13-5 and updating the UI based on the user input 13-6; and displaying the updated U I 13-3. The wireless data may be received, via wireless receiver / itansmiters, from a 360 degree camera and one or more additional elements of a remote unit according to any example embodiment described herein. The creation and display of the UI may include a user- end unit / Topaz unit UI and controlling a unit to display the UI according to any one or more of the example embodiments described herein. The user input may be received via a user interacting with a meau or other element of the displayed UI via a user input unit according to one or more example embodiments herein. For example, a user may enter new information or may manipulate a user input to create an instruction to change a window included in the UI.

[0134] FIG. 16 is a flow diagram of a presentation and monitoring method according to one or more example embodiments. In a system according to one or more example embodiments described here, a presentation and monitoring method may include operations of: receiving wireless data 14-1: rendering a 360 degree video 14-2; creating a displaying a UI 14-3; and obtaining status information of, for example, one of a camera of a remote unit and a wireless connection with a camera or other element of the remove unit 14-4. The status information may- be obtained by transmitting a status update request, such as a ping, to the camera or other element, recei ving a response to the update request, determining that there is no response to the update request, or determining a latency of a transmission. The method may further include updating the UI based on the determined updated status 14-6. The method may retain to displaying the UI 14-3 or may, if there is a change in the status 14-7 (YES), perform a response operation according to any one of the example embodiments discussed herein.

[0135] According to one or more example embodiments described herein, the rendering and display of a 360 video in conj unction with the display of a U1 with one or more over lying windows, menus, mformation, and the like enables a user to have unparalleled situational awareness, to make swift decisions, and to seamlessly integrate the user into existing command structures. In the dynamic theater of military operations, the efficacy of command and control hinges on the quality of visual information and the abil ity to operate in unreliable network environments. According to one or more example embodiments described herein, controlling the display of a UI based on user input contributes to the immersive and holistic perspective of the operational environment provided to the user. Related art video systems installed on unmanned vehicles have been plagued by a critical limitation—mnnel vi sion. This narrow field of view constrains a user’s situational awareness and can be particularly challenging in the ever-changing and often hostile conditions faced by military personnel, 360-degree video technology, offering an immersive and holistic perspective of the operational environment. Unlike related art camera systems, a system, according to one or more example embodiments described herein may contribute to eradication of the limitations of tunnel vision, and contribute to enabling military commanders and operators to monitor potential threats from all angles, track multiple targets simnltaneously, and respond proactively to evolving situations. Thus, a system according to one or more example embodiments described herein may not only enhance security but also contribute to accelerating the decision-making processes-— a vital component in the rapid and dynamic context ofmilitary operations.

[0136] With 360-degree video according to one or more example systems described herein, commanders and operators can detect threats from any angle., monitor multiple points of interest simultaneously, and achieve a level of situational awareness that was once unattainable.

[0137] 360-degree video according to one or more example systems described herein contributes to providing a user with an mimersive and comprehensive view of their operational surroundings. ’This panoramic perspective allows them to monitor multiple unmanned vehicles simultaneously, detect threats from any direction, and make rapid and informed decisions,

[0138] 360-degree video according to one or more example systems described herein contributes to eliminating blind spots and ensuring that no detail escapes notice. Users can survey vast areas and monitor critical points of interest simultaneously.

[0139] 360-degree video according to one or more example systems described herein contributes to a user’s ability to respond swiftly to emerging threats or opportunities. With interactive tools and real-time data overlays, they can quickly assess situations, make informed decisions, and redirect assets as needed.

[0140] According to one or more example embodiments described herein, provision of an updating status, including but not. limited to a status of a remote camera and a status of a wireless connection with a remote camera, a remote GPS unit, and / or another remote unit: as well as the ability to perform operations in response to a change in the status, either automatically or manually, contributes to provision of a groundbreaking to video delivery system that is optimized for unreliable network conditions and tactical environments. In military settings, where bandwidth can be limited, interference is Commonplace, and network reliability is often compromised, the uninterrupted flow of real-time video is indispensable.

[0141] Monitoring and response operations according to one or more example embodiments described herein provide an ability to fluctuating conditions, contributing to ensuring that military commanders and operators receive continuous, low-latency video feeds even when facing unreliable networks. It further contributes to an ability to prioritize video feeds efficientlywithin the military's tactical framework. This interoperability enhances the resilience and effectiveness of command and control operations, even in the most challenging and unpredictable environments.

[0142] Monitoring and response operations according to one or more example embodiments described herein provide an ability to automatically adjusts performance to maintain a consistent and high-quality video stream, ensuring that commanders and operators receive uninterrupted, real-time visual information; to enhance the resi lience and effectiveness of command and control operations, even in bandwidth-constrained or interference-prone scenarios.

[0143] The addition of Internet of Tilings sensors which can collect and relay real-time data from various sources enable capture of critical information such as vehicle status, environmental conditions, equipment performance, and more. The data-driven insights provided by ioT sensors empower commanders with valuable situational intelligence, optimizing their tactical decisions.

[0144] According to one or more example embodiments described herein, integration of GPS data contributes to more precise tracking and coordination of unmanned vehicles . Whether guiding drones through complex airspace or maneuvering autonomous ground vehicles in challenging terrain. Topaz ensures that remote fleets operate with pinpoint accuracy.

[0145] While aspects and implementation of example embodiments have been shown and described herein, it will be understood by those skilled in the art that various changes in form and details may be made therein. For example, various communication protocols can be deployed with various electronic sensors, and / or various visual and / or audio user interfaces can be implemented to facilitate processing and / or displaying information and / or controlling hardware and / or software components of the system.

[0146] It may be understood that the example embodiments described herein may be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each example embodiment may be considered as available for other similar features or aspects in other example embodiments.

[0147] While example embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by foe following claims.

Claims

CLAIMSWhat is chimed is:1 , A presentation system comprising: a wireless receiver configured to wirelessly receive, from a remote unit, 360 degree video data and global positioning data; a display unit; and a user-end unit operatively connected to the wireless receiver and to the display unit, the user-end unit comprising a non-fransitory storage medium storing instructions and a processor configured to execute the instructions and thereby. render a 360 degree video from a point of view based on the 360 degree video data; create a user interface (US) comprising a plurality of windows based on the 360 degree video data and the global positioning data, the plurality of windows comprising; a video window displaying the 360 degree video from the point of view, and at least one overlay window comprising a menu window displaying an interactive settings menu comprising a iiser-controllable indication of the point of view, and the global positioning data; control the display unit to display the UI; receive user input comprising a change of the indication of the point of view; render an updated 360 degree video from an updated poin t of view based on the360 degree video data and the user input; andcontrol the display unit to display an updated video window comprising the updated 360 degree view.2, The presentation system according to claim I „ wherein: the processor is configured to create the UI such that each of the at least one overlay window is an overlay displayed over the video window.3, The presentation system according to claim 2, wherein: the at least one overlay window further comprises a map window displaying a map and an indication on the map of a location of the remote unit ; and control of the display unit to display the UI compri ses controlling the display unit to display, in the map window, an automatically updated map and an automatically updated indication based on the global positioning data.

4. The presentation System according to claim 3, wherein the processor is further configured to: recei ve user input comprising an instrnction to change one of a size and a position of one of the at least one overlay windows to one of a new size and a new position; and control the display unit io display an updated UI displaying the one of the at. least one overlay windows in the one of the new size and the new position.

5. The presentation system accordin g to claim 3, wherein the processor is further configured topreceive user input comprising an instruction to change display of the map in the map window; and control the display unit to display an updated VI displaying a changed display of the map in the map window.

6. A method of data presentation, the method comprising: wirelessly receiving 360 degree video data and global positioning data from a remote unit; rendering a 360 degree video from a point of view based oa the 360 degree video data; creating a user interface (VI) comprising a plurality of windows based on the 360 degree video data and the global positioning data, the phirality of windows comprising: a video window displaying the 360 degree video from the point of view, and at least one overlay window comprising a menu window displaying an interactive settings menu comprising a user-controllable indication of the point of view, and the global positioning data; controlling a display unit to display the UI; recei ving user input comprising a change of the indication of the point of view; rendering an updated 360 degree video from an updated point of view based on the 360 degree video data and the user input; controlling the display unit to display an updated video window comprising the updated 360 degree view.

7. The method according to claim 6, wherein:the creating the UI comprises creating the UI such that each of the at least one overlay window is an overlay displayed over the video window,8. The method according to claim 7, wherein: the at least one overlay window further comprises a map window displaying a map and an indication on the map of a location of the remote wait: and the controlling the display unit to display the UI comprises controlling the display unit to display, in the map window, an automatically updated map and an automatically updated indication based on the global positioning data.

9. The method according to claim 8, further comprising: receiving user input comprising an instruction to change one of a size and a position of one of the at least one overlay windows to one of a new size and a new posi tion: and controlling the display unit to display an updated UI displaying the one of the at least one overlay windows in the one of the new size and the new position.

10. The method according to claim 7„ further comprising: receive user input comprising an instruction to change display of the map in the map window: and controlling the display unit to display an updated UI displaying a changed display of the map in the map window.

11. A presentation and monitoring system comprisinga wireless transceiver configured to wirelessly communicate with a remote unit comprising a camera; a display unit; and a user-end unit operatively connected to the wireless receiver and to the display unit, the user-end unit comprising a non-transitory storage medium storing instructions and a processor configured to execute the instructions and thereby; receive, via the wireless -transceiver, a data stream of 360 degree video data from the camera; render a 360 degree video based on the 360 degree video data; create a user interface (VI) comprisi ng a plurality of windows based on the 360 degree video data and the global positioning data, the plurality of windows comprising: a video window displaying the 360 degree video from the point of view, and at least one overlay window comprising a status menu displaying an internet protocol (IP) address of the camera and a status of one of the camera and a wireless connection between the wireless transceiver and the camera; control the display to display the UI; transmit to the camera, via the wireless transceiver, a status request; determine whether a response to the status request is received; determine an updated status based on one of a received response to the status request and a deteiinination of no response received; create an updated UI based on the updated status; and control the display to display the updated VI.

12. The presentation and .monitoring system according to claim 11, wherein: the processor is configured to create the UI such that each of t he at least one overlay window is an overlay displayed over the video window.

13. The presentation and. monitoring system according to claim 11 , wherein the status comprises one of aping status arid a latency of a wireless transmission.14, The presentation and monitoring system according to claim 11, wherein the processor is further configured to, in response to a change of the status; perform a response operation comprising at least one of; initiating a reconnection with the camera transmitting an instruction for reconfiguration to the camera; throttling, buffering, pausing, or disabling rendering of the 360 degree video; switching to a mixed reality mode; increasing a bitrate; changing a transmission format.15, A presentation and monitoring method comprising: wirelessly communicating with a remote unit comprising a camera; wirelessly receiving a data stream of 360 degree video data from the camera; rendering a 360 degree video based on the 360 degree video data;creating a user interface (Ul) comprising a plurality of windows based on the 360 degree video data and the global positioning data, the plurality of windo ws comprising: a video window displaying the 360 degree video from the point of view, and at least one overlay window comprising a status menu displaying an internet protocol (IP) address of the camera and a status of one of the camera and a wireless connection between the wireless transceiver and the camera; controlling a display unit to display the UI; transmitting to the camera, via the wireless transceiver, a status request ; determining whether a response to the status request is received; determining an updated status based on one of a received response to the status request and a determination of no response received; creating an updated U1 based on the updated status; and controlling the display to display the updated UI.

16. The presentation and monitoring method according to claim 1.5, wherein: the creating the UI comprises creating the UI such that each of the at least one overl ay window is an overlay displayed over the video window.

17. The presentation and monitoring method according to claim 15, wherein the status comprises one of a ping status and a latency of a wireless transmission.

18. The presentation and monitoring method according to claim 15 further comprising; in response to a change of the status, performing a response operation comprising at least one of; initiating a reconnection with the camera transmitting an instruction for reconfiguration to the camera; throttling, buffering, pausing, or disabling rendering of the 360 degree video; switching to a mixed reality mode; increasing a bitrate; changing a transmission format.

Images