Method and system for network interaction with objects and remote display techniques

Abstract

The present disclosure encompasses technical advancements for distributed operating systems in which the mobile device is a controller and the smart television becomes a display of a system in which message transmission, stream propagation and computing processes are stored in the cloud or in decentralized managed operating systems.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 456018, filed March 31, 2023, entitled “Method and System for a WebInteraction with Objects and Remote Display Technologies,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This system, apparatus, and method belong to the field of communications and processing, and more specifically, relate to methods and systems for creating a distributed system for using mobile phones and smart TVs as displays to process controllers, while infrastructure processes all messages passed from the controllers to applications in a distributed system with distributed processing capabilities, just as a traditional operating system processes I / O events and tasks. Background Technology

[0004] Traditional implementations of remote management and control of computers rely on protocols such as Remote Desktop Protocol (RDP), Virtual Network Computing (VNC) (also known as RFB), Citrix Internet Computer Architecture (ICA), and other proprietary mechanisms of both client and server software to remotely control and manage personal computers (PCs), laptops, tablets, mobile devices, or set-top boxes.

[0005] In today's digital landscape, many smart TVs are limited to being controlled by native apps designed for use with remote controls. In most cases, these apps are used for streaming purposes rather than being designed for dynamic interaction with mobile computing devices. A solution is needed to enable a seamless, interactive user experience with the TV via mobile computing devices. Summary of the Invention

[0006] One aspect of the disclosed embodiments includes a method. The method includes: generating a first message for a first browser executing on a first computing device, the first message including an instruction that, when executed by the first browser, causes a representation of an interface of a physical user input device to be displayed by the first browser; providing the first message to the first browser; generating a second message for a second browser executing on a second computing device, the second message including an instruction that, when executed by the second browser, enables user interaction with content provided by the second browser in response to input from a representation of the interface of the physical user input device displayed by the first browser, wherein the physical user input device is configured to interact with the content provided by the second browser; and providing the second message to the second browser.

[0007] Another aspect of the disclosed embodiments includes a system. The system includes: at least one processor circuit; and at least one memory storing instructions to be executed by the at least one processor circuit. The instructions are configured to perform an operation including: generating a first message for a first browser executing on a first computing device, the first message including instructions that, in response to execution by the first browser, cause a representation of an interface of a physical user input device to be displayed by the first browser; providing the first message to the first browser; generating a second message for a second browser executing on a second computing device, the second message including instructions that, when executed by the second browser, enable user interaction with content provided by the second browser in response to input from the representation of the interface of the physical user input device displayed by the first browser, wherein the physical user input device is configured to interact with content provided by the second browser; and providing the second message to the second browser.

[0008] Another aspect of the disclosed embodiments includes a computer-readable storage medium having program instructions recorded thereon, which, when executed by at least one processing circuitry of a computing device, perform a method. The method includes: generating a first message for a first browser executing on a first computing device, the first message including instructions that, in response to execution by the first browser, cause a representation of an interface of a physical user input device to be displayed by the first browser; providing the first message to the first browser; generating a second message for a second browser executing on a second computing device, the second message including instructions that, when executed by the second browser, enable user interaction with content provided by the second browser in response to input from the representation of the interface of the physical user input device displayed by the first browser, wherein the physical user input device is configured to interact with the content provided by the second browser; and providing the second message to the second browser. Attached Figure Description

[0009] In the accompanying drawings, similar reference numerals refer to the same or functionally similar elements in separate views; the drawings are not necessarily drawn to scale, and the drawings, together with the following detailed description, are incorporated in and form a part of this specification. The drawings are used to illustrate various other embodiments and to explain various principles and advantages in accordance with all systems, devices, and methods. The advantages of embodiments of systems, devices, and methods will become apparent from the following detailed description of exemplary embodiments thereof, which should be considered in conjunction with the accompanying drawings, wherein:

[0010] Figure 1 It is a block diagram of exemplary embodiments of device connectivity applications and architectures for network interaction with objects and remote display technologies;

[0011] Figure 2 It is used for Figure 1 A schematic diagram of an exemplary embodiment of the palette controller and game controller in the architecture;

[0012] Figure 3 It is used for Figure 1 A schematic diagram of an exemplary embodiment of the controller for the keyboard, camera, and microphone in the architecture;

[0013] Figure 4 It is used for Figure 1 A schematic diagram of the client and server architecture of the system, and an exemplary embodiment of using SocketIO and WebSockets for message transmission and interaction with a smart TV;

[0014] Figure 5 It enables server-based interfaces to... Figure 1 A flowchart illustrating an exemplary embodiment of a method for a smart TV in a device connectivity platform architecture;

[0015] Figure 6 This is a flowchart illustrating an exemplary embodiment of a method for enabling a server-based interface to be used as a controller for a device connectivity platform application.

[0016] Figure 7(a) is a schematic diagram of an exemplary embodiment of the WebSocket architecture for communication;

[0017] Figure 7(b) is a schematic diagram of an exemplary embodiment of the WebRTC architecture used for communication;

[0018] Figure 7(c) is a schematic diagram of an exemplary embodiment of the WebRTC w / Tunnel architecture for communication;

[0019] Figure 8(a) is a schematic diagram of an exemplary embodiment of loading the game controller and game application in Figure 8(c) served by the device connection platform into a smart TV for swipe-right interaction;

[0020] Figure 8(b) is a schematic diagram of an exemplary embodiment of loading the game controller and game application in Figure 8(c) served by the device connection platform into a smart TV for tap interaction;

[0021] Figure 8(c) is a schematic diagram of an exemplary embodiment of a game controller and game application served by a device connectivity platform used on a smart TV;

[0022] Figure 9(a) is a schematic diagram of an exemplary embodiment of selecting an application for a drawing application using hand interaction;

[0023] Figure 9(b) is a schematic diagram of an exemplary embodiment of drawing interaction in the drawing application selected in Figure 9(a);

[0024] Figure 10 This is a schematic diagram of an exemplary embodiment of mapping a QR code to an application to be loaded and a device connection platform interface;

[0025] Figure 11 This is a schematic diagram of an exemplary embodiment of a controller used in a device connectivity platform application, which includes a touchpad, video, controller, audio, keyboard, 3D gestures, telephone keypad, card reader, and imaging.

[0026] Figure 12(a) is a block diagram of an exemplary embodiment of a device connectivity platform multimedia application and integration with a USB webcam and other USB devices via IP;

[0027] Figure 12(b) is a block diagram of an exemplary embodiment of a device connectivity platform system with WebRTC and WebSockets;

[0028] Figure 12(c) is a block diagram of an exemplary embodiment of a device connection platform stream using CANVAS and a network application;

[0029] Figure 13 This is a block diagram of an exemplary embodiment of a device connection platform for remote access to a Windows computer;

[0030] Figure 14(a) is a block diagram of an exemplary embodiment of using context awareness to detect streams;

[0031] Figure 14(b) is a block diagram of an exemplary embodiment of capturing the screen to eliminate the need for a QR code;

[0032] Figure 14(c) is an illustration showing the detection of a smart TV using Ultra-wide Band (UWB) and location instead of QR codes;

[0033] Figure 15 This is a flowchart of an exemplary embodiment of a method for detecting position based on video feed from a smart TV display;

[0034] Figure 16(a) is a schematic diagram of an exemplary embodiment of a dongle with QR code and Bluetooth Low Energy Wi-Fi configuration;

[0035] Figure 16(b) is a block diagram of an exemplary embodiment of a method for interacting with the dongle and QR code of Figure 16(a) with a Bluetooth Low Energy Wi-Fi configuration;

[0036] Figure 17(a) is an exemplary embodiment of the combination of gesture.js and mevia.js and their application in... Figure 1 A block diagram used in the device connectivity platform architecture;

[0037] Figure 17(b) shows an exemplary embodiment of the combination of gesture.js and mevia.js and their application in... Figure 1 A block diagram used in the device connectivity platform architecture;

[0038] Figure 17(c) is an exemplary embodiment of the combination of gesture.js and mevia.js and their application in... Figure 1 A block diagram used in the device connectivity platform architecture;

[0039] Figure 18 Is Figure 1 A block diagram illustrating an exemplary embodiment of a message processing method in a device connectivity platform architecture.

[0040] Figure 19 Is Figure 1 A schematic diagram illustrating an exemplary embodiment of message.evt and message.command processed in the device connectivity platform architecture;

[0041] Figure 20 It is Figure 1 A schematic diagram illustrating an exemplary embodiment of integrating a device connectivity platform architecture into a broadcast platform and system;

[0042] Figure 21 It is used from Figure 1 A block diagram of an exemplary embodiment of a device connection platform architecture for routing commands and messages of a controller.

[0043] Figure 22(a) is a schematic diagram of an exemplary embodiment of data capture of the controller, which will be trained using a neural network or deep learning using LSTM or other neural networks;

[0044] Figure 22(b) is a schematic diagram of an exemplary embodiment of using a controller to train a neural network to match different commands;

[0045] Figure 22(c) is a schematic diagram of an exemplary embodiment of motion detection used by a controller system with deep learning or neural networks;

[0046] Figure 23(a) is a schematic diagram of an exemplary embodiment of loading the game controller and game application in Figure 8(c) served by the device connection platform into a smart TV for right-swipe interaction;

[0047] Figure 23(b) is a schematic diagram of an exemplary embodiment of tap interaction in which the game controller and game application in Figure 8(c) served by the device connection platform are loaded into a smart TV.

[0048] Figure 23(c) is a schematic diagram of an exemplary embodiment of a game controller and game application served by a device connection platform on a smart TV;

[0049] Figure 24 This is a schematic diagram of an exemplary embodiment of a device connectivity platform application that uses 3DMevia.js to map 3D gestures to 3D display actions; and

[0050] Figure 25 This is a block diagram of an exemplary embodiment of a method for using gesture-driven tools to perform access authentication or gesture recognition for a connected platform application.

[0051] Figure 26 The flowchart illustrates, in general, a method for enabling a user to control and interact with content on a computing device using a browser interface on another computing device, in accordance with the principles of this disclosure. Detailed Implementation

[0052] Detailed embodiments of systems, devices, and methods are disclosed herein as needed; however, it should be understood that the disclosed embodiments are merely examples of systems, devices, and methods that can be embodied in various forms. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather serve merely as the basis of the claims and as a representative basis for teaching those skilled in the art to adopt systems, devices, and methods in various ways with virtually any suitable detailed structure. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of the systems, devices, and methods. Although this specification concludes with claims defining the novel systems, devices, and methods of the invention, it is believed that the systems, devices, and methods will be better understood by considering the following description in conjunction with the accompanying drawings, wherein similar reference numerals are retained.

[0053] In the following detailed description, reference is made to the accompanying drawings, which form part of the invention, and exemplify embodiments that can be practiced. It should be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope. Therefore, the following detailed description should not be considered limiting, and the scope of the embodiments is defined by the appended claims and their equivalents.

[0054] Alternative embodiments may be designed without departing from the spirit or scope of this disclosure. Furthermore, well-known elements of exemplary embodiments of the systems, devices, and methods will not be described in detail, or will be omitted, so as not to obscure the relevant details of the systems, devices, and methods.

[0055] Before disclosing and describing systems, devices, and methods, it should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The terms “comprises,” “includes,” or any other variations thereof are intended to cover exclusive inclusion, such that a process, method, article, or device that includes a list of elements includes not only those elements but may include other elements not expressly listed or not inherent to such process, method, article, or device. Without further limitation, an element continuing with “including…” does not exclude the presence of additional identical elements in the process, method, article, or device that includes that element. As used herein, the terms “including” and / or “having” are defined as including (i.e., open-ended language). As used herein, the terms “a” or “an” are defined as one or more. As used herein, the term “a plurality” is defined as two or more. As used herein, the term “another” is defined as at least a second or more. This specification may use the terms “embodiment” or “embodiments,” which may refer to one or more of the same or different embodiments, respectively.

[0056] The terms “connection” and “linkage”, as well as their derivatives, may be used. It should be understood that these terms are not intended to be synonyms. Rather, in certain embodiments, “connection” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Linkage” may mean that two or more elements are in direct physical or electrical contact (e.g., direct connection). However, “linkage” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other (e.g., indirect connection).

[0057] For descriptive purposes, phrases in the form of "A / B," "A and / or B," or "at least one of A and B" mean (A), (B), or (A and B), where A and B are variables indicating specific objects or attributes. In use, this phrase is intended and thereby defined as a choice of A or B, or both A and B, similar to the phrase "and / or." Where more than two variables exist in such a phrase, it is thereby defined to include only one variable, any one of the variables, any combination of the variables, and all variables; for example, phrases in the form of "at least one of A, B, and C" mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0058] Relational terms (such as first and second, top and bottom, etc.) may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. This specification may use perspective-based descriptions such as up / down, rear / front, top / bottom, and proximal / far. Such descriptions are for ease of discussion only and are not intended to limit the application of the disclosed embodiments. Various operations may be described in a manner that may aid in understanding the embodiments as multiple discrete operations; however, the order of description should not be construed as implying that these operations are order-dependent.

[0059] As used herein, the terms “about” or “approximately” apply to all numerical values, whether explicitly stated or not. These terms generally refer to a range of numbers that a person skilled in the art would consider equivalent (i.e., having the same function or result). In many cases, these terms may include numbers rounded to the nearest significant figure. As used herein, the terms “substantially” and “substantially” mean that, when individual parts are compared to each other, the dimensions of the parts being compared are equal or sufficiently close that a person skilled in the art would consider them identical. As used herein, “substantially” and “substantially” are not limited to a single dimension, but specifically include ranges of values ​​for the parts being compared. Ranges of values ​​above and below (e.g., “+ / -” or greater / less or greater than / less than) include the variance that a person skilled in the art would know is a reasonable tolerance for the part mentioned.

[0060] It should be understood that embodiments of the systems, devices, and methods described herein may include one or more conventional processors, as well as controls for one or more processors to combine certain processorless circuitry and other components to implement some, most, or all of the functions of the systems, devices, and methods described herein. Processorless circuitry may include, but is not limited to, signal drivers, clock circuits, power supply circuits, and user input and output elements. Alternatively, some or all functions may be implemented either by a state machine without stored program instructions, or in one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that implement each function or some combinations of functions as custom logic. Of course, combinations of these methods may also be used. Therefore, methods and apparatus for these functions have been described herein.

[0061] As used herein, the terms “program,” “software,” “software application,” etc., are defined as a sequence of instructions designed to be executed on a computer system. “Program,” “software,” “application,” “computer program,” or “software application” can include subroutines, functions, processes, object methods, object implementations, executable applications, applets, service applets, source code, object code, shared libraries / dynamically loaded libraries, and / or other sequences of instructions designed to be executed on a computer system.

[0062] Various embodiments of systems, devices, and methods are described herein. In many different embodiments, features are similar. Therefore, to avoid redundancy, in some cases, these similar features may not be described again. However, it should be understood that the description of a feature appearing first applies to similar features described later, and thus each corresponding description is incorporated therein without such repetition.

[0063] Although the systems, devices, and methods shown and described herein are implemented in systems and methods for network interaction with objects and in remote display technologies, they are not intended to be limited to the details shown, as various modifications and structural changes can be made therein without departing from the spirit of this disclosure and within the scope of the equivalents of the claims. Furthermore, well-known elements of exemplary embodiments will not be described in detail or will be omitted so as not to obscure the relevant details of the systems, devices, and methods.

[0064] Additional advantages and other features of the system, apparatus, and method will be set forth in the following detailed descriptions, and these additional advantages and other features will become apparent from these descriptions, or may be learned by practicing exemplary embodiments. Other advantages of the system, apparatus, and method may be achieved by any of the tools, methods, or combinations specifically specified in the claims.

[0065] Other features considered characteristic of the system, apparatus, and method are set forth in the appended claims. Detailed embodiments of the system, apparatus, and method are disclosed herein as needed; however, it should be understood that the disclosed embodiments are merely examples of systems, apparatus, and methods that can be implemented in various forms. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as a representative basis for teaching those skilled in the art to adopt the system, apparatus, and method in various ways in virtually any detailed structure. Furthermore, the terminology and phrases used herein are not intended to be limiting, but rather to provide an understandable description of the system, apparatus, and method. While this specification concludes with claims defining the systems, apparatus, and methods of this disclosure as novel, it is believed that a better understanding of the system, apparatus, and method can be achieved by considering the following description taken in conjunction with the accompanying drawings, in which the same reference numerals will continue to be used.

[0066] The embodiments disclosed herein address device connectivity platforms that overcome limitations in existing technologies such as Remote Desktop Protocol (RDP). These platforms eliminate the need for RDP by using standard Hypertext Transfer Protocol (HTTP) and video tagging, enabling access to remote units without a transcoding server. The platforms allow users to control web applications using mobile computing devices such as tablets, smartphones, augmented reality glasses, virtual reality devices, or any other device capable of displaying information. These mobile computing devices act as controllers, eliminating the need for traditional controller interfaces. Furthermore, the platform employs video encoding to project the remote screen into a web browser. For example, JavaScript capabilities are used to transform the mobile computing device into a physical user input device (e.g., a mouse, keyboard, game controller). This allows users to send touch commands displayed on a computing device such as a smart TV to a remote server.

[0067] Remote management is achieved by forwarding touch commands from the mobile web browser to a remote computing device. These commands can be displayed on a smart TV or streamed via various web browser-compatible protocols, such as Web Real-Time Communication (WebRTC), Real-Time Messaging Protocol (RTMP), HTTP Live Streaming (HLS), or DASH. The platform also supports adaptive video quality, meaning it can adapt to different network conditions and device capabilities. For example, the platform can switch video resolution and quality based on the user's needs.

[0068] The platform functions as a decentralized or distributed operating system. It connects web applications and mobile computing devices, which act as controllers for these applications displayed on smart TVs, AR and VR displays (augmented reality and virtual reality), and projectors. The system utilizes JavaScript / CSS libraries (e.g., "gestures.js") for control. The user's mobile device acts as the controller, sending commands and gestures translated into JavaScript macros for efficient processing and interaction with the web applications. Simultaneously, a remote webpage is displayed on the smart TV, which loads one or more other JavaScript files (e.g., Mevia.js) and becomes the display. In some embodiments, an identification mechanism such as Quick Response (QR) codes can be used to load the controller onto the smart TV. This mechanism includes parameters such as application identifiers and other information determining the target server for all user input and interaction. In one embodiment, and to further illustrate, a QR code is scanned to activate a representation of the interface of a physical user input device to be displayed on the mobile computing device, establishing a communication channel between the mobile computing device and the smart TV, enabling the user to interact with content displayed on the TV through input from the representation of the interface of the physical user input device displayed on the mobile computing device.

[0069] The embodiments disclosed herein allow users to use a mobile computing device as an input user device and stream content to a smart TV. By leveraging video encoding and JavaScript, the embodiments disclosed herein provide a dynamic and adaptive user experience. The decentralized management nature of the platform means it can be applied to a wide range of applications and computing devices. Any capability of web applications customized for smartphones, tablets, laptops, AR (augmented reality), virtual reality, extended reality displays, or desktop interfaces can now be extended to use with smart TVs. In this platform, the mobile computing device serves as the physical user input device, while the smart TV acts as the display, and the device connectivity platform operates as a central platform for handling message transmission between the mobile computing device and the smart TV. In this disclosure, the user input device and the controller are used interchangeably.

[0070] To help illustrate this, the following will now be described. Figure 1 In particular, Figure 1 This is a block diagram of an exemplary embodiment of a cloud service network 102 that hosts a device connectivity platform 100 and several applications accessible to the device connectivity platform 100. As used herein, a cloud service network refers to the underlying technology that facilitates access to, storage of, and management of data, applications, and resources hosted in the cloud. Applications mentioned herein can be any type of network-accessible application / service, such as database applications, social networking applications, messaging applications, financial services applications, news applications, search applications, network-accessible productivity applications, cloud storage and / or file hosting applications, etc.

[0071] like Figure 1 As shown, applications include (non-fungible token) NFT Lab 110, doorbells 120 (e.g., RING®), games 130, video conferencing 140 (e.g., Zoom™, Microsoft Teams®, Slack®), and streaming applications 150 (e.g., Netflix). Although Figure 1 The cloud service network 102 is shown as hosting only these applications; however, it should be understood that the techniques described herein can be applied to cloud service networks hosting other applications, such as email services (e.g., Google®, Outlook®), productivity suites (e.g., Google Workspace™, Microsoft 365™), social media platforms (e.g., Facebook®, Twitter®, LinkedIn®), cloud storage and file sharing (e.g., Dropbox®, OneDrive®), and online shopping (e.g., Amazon, eBay®).

[0072] Device connectivity platform 100 may include one or more server devices and / or other computing devices. Any application may be associated with resources stored on one or more application servers. Each component of cloud service network 102 may be connected via one or more networks ( Figure 1 (Not shown in the diagram) Communication connections. These one or more networks may include, for example, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet.

[0073] In addition, such as Figure 1 As depicted, the device connectivity platform 100 uses components as part of the device connectivity platform application 160 (referred to herein as the Mevia App in some embodiments), network integration code 170 (referred to herein as mevia.js when referenced as one or more JavaScript files in some embodiments), and the network integration code 170 is configured to integrate various web technologies from the original web application, such as JavaScript, CSS, and HTML. In some embodiments, utilizing the implementation of mevia.js, this can be achieved by creating components <mevia / > 180 is implemented in frameworks such as React. In some embodiments, the device connectivity platform 100 can operate on different ports, including a secure SSL mode for HTTPS, depending on how the web server is configured. The device connectivity platform 100 is adaptable to different web server configurations. The device connectivity platform 100 can work with servers like NGINX or Apache, and even with NodeJS 175, which supports both HTTP and HTTPS. The network integration code 170 can act as a connector, essentially linking the communication of various components of the device connectivity platform 100 and the data flow between different parts of the system, acting as the "glue" that keeps the components of the device connectivity platform 100 together. Applications can be hosted on their own servers, whether they are within the same network or scattered across different networks. This provides flexibility in resource allocation and deployment. The device connectivity platform 100 can employ technologies such as SocketIO for WebSockets and WebRTC for peer-to-peer communication. These technologies enable real-time, instant communication between the controller and the controlled network applications.

[0074] For example, in some embodiments, an application may use two or more ports, for instance, to run in SSL mode for HTTPS and another port (e.g., 3000). This depends on the configuration of the web server (e.g., NGINX or Apache); a hybrid configuration running NodeJS 175 and a web server supporting HTTP / HTTPS server 165 can also be used to support the service device connectivity platform 100 application and the service controller application. As described above, in some embodiments, network integration code 170 (e.g., mevia.js) is used as glue code to the device connectivity platform 100 to "join" the virtual environment, where messages carry events and other messages from the controller or controller group. Furthermore, in some embodiments, each application may reside on its own server (e.g., such as...). Figure 1 The applications described are nftmevia.art, iotexpert.com, meviagames.com, zoom.mevia.tv, and mevia.tv. Furthermore, all applications can reside on the same network or different networks. Additionally, when establishing peer-to-peer communication between the controller and the controlled network applications, network integration code 170 (e.g., Mevia.js) can also include the use of SocketIO for Web sockets (WS or Web Socket Secured, also known as WSS) and WebRTC.

[0075] Additionally, in some embodiments, the device connectivity platform 100 is configured to convert any mobile terminal using a web browser into a remote device that can be controlled by a browser application to control a smart TV. Typically, when using a web-based terminal, web assets are used to display widgets, but they can also be captured from a web-based headless rendering engine such as Puppeteer (https: / / github.com / puppeteer / puppeteer)). For example, all manipulations and commands generated by the device connectivity platform application can also be broadcast to cable or satellite operators.

[0076] Device connection platform controller

[0077] As described above, the device connectivity platform 100 is configured to work with several types of user input devices, each of which has functionality dependent on a network application it intends to control. In some embodiments, after scanning a QR code displayed on or printed on a smart TV, a representation of the physical user input device's interface is displayed by a first browser. For example, in Figure 1In this context, QR codes or other identification mechanisms can be used to identify user input devices to be displayed on mobile computing devices. For example, a QR code can identify a remote control associated with a smart TV. In some embodiments, the QR code includes parameters such as an application identifier (e.g., an Application Identifier (AppID) or a Universal Unique Identifier (UUID)) and other information that identifies the target server for any touch gesture data, such as key presses, touch movements, gestures including accelerometer data captured at the mobile computing device.

[0078] Figure 2 Exemplary embodiments are provided for representations of an interface of a physical user input device displayed by a browser running on a computing device. For example, a palette controller 200 and a game controller 210 are displayed on a computing device 230. The computing device 230 can be any type of fixed or mobile computing device, including mobile computers or mobile computing devices (e.g., smartphones, laptops, notebook computers, tablets such as Apple iPad™, netbooks, etc.), wearable computing devices (e.g., smartwatches, head-mounted devices including smart glasses such as Google® Glass™), or fixed computing devices such as desktop computers or PCs (personal computers).

[0079] like Figure 2 As shown, the palette controller 200 includes a grid 207 for mapping the display, and other commands shown in 202 that can be associated with JavaScript source code and libraries. For example, as a user touches the grid 207 with a stylus or finger, the palette controller 200 can issue touch start, touch end, and touch movement. For illustration, a touch start can be generated at x=100, y=200, and sent in a message to the drawing network application. In some embodiments, this is tracked via gesture.js 220, a script that detects all gestures. Meanwhile, the receiving application will process those events locally and complete the processing, for example, when x=200, y=400, a touch end event has been detected in the grid 207 or an area on the palette 205 has been touched. Furthermore, in some embodiments, the controller can also associate JavaScript elements to be sent as messages, such as " The jQuery command `('#button').click()` refers to a button that is a remote HTML tag associated with the HTML in the web application being controlled, such as...<buttonid=”button”…. / > .

[0080] The controller can send messages and events to the device's connectivity platform, where the content router will then process all events and messages and direct them to the appropriate application, which includes mevia.js as part of its library. Events generated by the controller's representation are then directed to the web application displayed on the television.

[0081] To help further illustrate, such as Figure 2 As depicted, when a user types a link or loads a Uniform Resource Locator (URL) 208 associated with the Palette Controller 200 (https: / / app.mevia.television / paintbursh / ), the Palette Controller 200 is displayed on a web browser running on the computing device 230. In some embodiments, the input user device may be associated with a specific application, and all events, keyboard messages, gestures, accelerometer readings are transmitted or emitted to a WebSocket command containing JavaScript Object Notation (JSON) objects, such as: Message = { evt: Type of event (eg “touch start”, X:position X, Y: position Y}

[0082] Because these events are captured by the gesture.js library and located and loaded as part of the controller's HTML, many other events can also be triggered, such as swipes to the right, left, up, and down at the local level on mobile devices. These basic touch events (e.g., touch start, touch move, touch end) are processed and used to generate message.comand or message.evt="Command.". In this case, the JSON message will contain the following structure: Message = { evt: Command, Command: “Swipe Right”}

[0083] There are several methods that can be used to detect right swipe events, including touchstart and touchend, arithmetic comparison of x and y coordinates, and the use of k-nearest neighbors (KNN), neural networks, or other machine learning techniques.

[0084] Assuming touchEndX and touchStartX are created, and the function "AbouttheSame" indicates that the two touch events are very close to each other, the following JavaScript code will generate swipe left and swipe right events: if (touchendX <=touchstartX && abouttheSame(touchstartY, touchendY,80) ) { console.log('Swiped left'); sendCommand("Swiped left", x, y); else if (touchendX >=touchstartX && abouttheSame(touchstartY, touchendY,80) ) { console.log('Swiped right'); sendCommand ("Swiped Right", x, y); } …

[0085] In different contexts, gestures can symbolize "tapping" or "double-tapping," and represent "input" or selection as a standard application. The interpretation of such commands is then retrieved from the database of the device connectivity platform 100, where a particular application identifier (AppID) will exhibit behavior distinct from other application identifiers. For example, a "long tap" could be mapped to a jQuery command or a standard HTML command such as document.QuerySelector, where... `("#silder").goto(1)`, or send the slider JavaScript widget to the first slide. Similarly, in a game scenario, the up, down, left, and right keys of the keyboard are mapped to icons or positions, which are represented as part of URL 216 https: / / app.mevia.television / gamecontroller. When the game controller is downloaded, the keypad 218 may contain standard game functions as well as "macro" commands 215 associated with one or more JavaScript messages executed remotely at the controlled web application. For example, for the doorbell 120, a "long knock" (e.g., pressing the screen for four seconds) may mean closing the door, while a short knock may mean opening the door. Specific functions are identified and can change in real time as the user presses and interacts with the controller, depending on how those controllers establish their behavior. In some embodiments, gestures.js 225 tracks any detected gestures.

[0086] Figure 3 Other exemplary embodiments are provided, representing the interface of a physical user input device displayed by a browser. For example, Figure 3 A microphone and camera controller 310 is described, comprising a microphone 315 and cameras 320 and 325, which is available in HTML5 in the latest browsers. These controllers can be combined with WebRTC messages during use and can be removed... <canvas>Drawing on HTML objects or used to transmit voice in voice chat applications, such as when using a video conferencing client application: Message={evt: Audio or Audio + Video or Video Payload: base64 image or webRTC socket to acanvas}

[0087] The remote website where the user opens a web browser and authorizes the use of a WebRTC camera or audio should be able to stream real-time video and audio processing content to the device connectivity platform 100, or process screenshot sequences captured from the camera or audio clips. Figure 3 The keyboard controller 300 and the messages sent from keystrokes at 305 are also depicted, which can be interpreted as: Message ={evt: KeyBoard, Value: "SpaceBar" Therefore, the embodiments disclosed herein illustrate how to process events, multimedia, and macros captured from a controller, and how the receiving application processes these macros.

[0088] Figure 4 The text describes how a device connectivity platform 100 processes user interactions with a representation displayed on a user input device on a computing device. More specifically, Figure 4 This illustrates how a user interacts with the controller, in this example, the keypad 402. For instance, if a user touches the number 7 on the keypad 402, the smart TV will be updated to display the number 7. To achieve this, the controller displayed on the smartphone 400 is located at URL 404 https: / / app.mevia.television / keypad. The keypad web application (web-app) includes multiple HTML / CSS objects with images including mevia.js and gesture.js. Figure 4 In this example, the remote web application is located at URL 432: https: / / callApp.mevia.television / call. Both URL 404 and URL 432 are loaded asynchronously and do not need to be synchronized. However, in some embodiments, URL 432 may be loaded first, before URL 404, and in this example, ready to be used. Thus, the target application can include the HTML tag "". <input type="text" id="”callto” / "> This will be used to receive keypad input 434. Therefore, the value in 434 is initially "" or empty. As the mobile device loads the controller via HTTP or HTTPS, the URL keypad at https: / / app.mevia.television / 406 loads the gesture.js library, HTML and images, and other JavaScript library information. The controller's URL also includes an application identifier (AppID) and a UUID, which are generated from the device connectivity platform 100 to associate user events with the application, and these values ​​are embedded in response 408. The UUID value is unique and is used to represent a session key. The "AppID" or application identifier can also be changed and is used as a "nonce" parameter, but this is not required for this particular example.

[0089] After the key "7" is pressed, `message.evt` and the message payload are created. SocketIO session 410 is used to send a JSON message structure 412, which includes the following members: `message.evt`, `message.value`, `message.x`, `message.y`, `message.z`, and `message.time` (or timestamp), which provides the location of the key press event and its value. The message payload can be expanded to include other parameters. The numeric keypad module can be rendered as part of a Node.js web application, listening on port 3000 at 443 and the HTTPS port, as shown in 414. Therefore, in Figure 4 In step 420, the keypad controller 416 will use a session and SocketIO request 418 instead of message.evt = touchStart at t=0, and touchEnd at=200ms. This message is replaced by a JavaScript command by the device connectivity platform 100, which is embedded as part of the payload in a SocketIO request to the network application with a "JavaScript library". At step 420, the network application receives message.evt=KeyPress, value=7, which is then processed by CallApp.Mevia.television using SocketIO or WebSockets (WS or WSS, secure WebSockets).

[0090] exist Figure 4 In the web application, WebSocket is controlled by mevia.js (422), and any CSS resources (424) are used to display and modify the HTML. <input type="text”" id="”callto”"> The default browser will send the label "Key pressed remotely was:" and the value 7 to the input screen. The key press is then translated into JavaScript using the "keydown" message `message.evt.keyValue`. In this sequence of events, the keypad 402 has effectively written the value "7" to the remote screen. For simplicity, the intervention of the "Content Router" entity is omitted, and simply `socket.broadcast.to()` from SocketIO is used to demonstrate how the value "7" is transmitted from the phone to the application. In practice, other applications retrieving the same webpage, and therefore all monitors, smart TVs, tablets, or other browsers connected to the same page, will see the update received from the mobile phone by pressing the keypad. The update observed at 434 occurs asynchronously, without a new GET request from the web browser at 432, instead triggered by `socket.on("command", ...)`, which in turn simply updates the input screen, which has a standard keyboard connected via USB and defined as in standard HTML page 426.

[0091] Additionally, the macro converter 428 of the device connectivity platform server may need to convert all references to local or remote assets in the device connectivity platform application 100 to fully qualified domain name (FQDN) addresses (e.g., hypertext reference attributes (href) tags in HTML). Even with "image src" or " HTML tags and resource paths must be converted to facilitate the transformation of standard web applications into device connectivity platform applications. For example, applications without FQND... The standard HTML-loaded assets in the URL can be converted to <image src=” https: / / callApp.mevia.television / call / images / myimage.png” / This transformation allows for use in cloud-based systems and enables access to CDNs that are easily handled by NodeJS. Furthermore, cross-origin resource sharing (CORS) or cross-origin resources must be configured to facilitate loading resources from a different Uniform Resource Identifier (URI) or URL than the original FQDN, even if they have the same FQDN but are executed from a different port. Figure 4 Element 430 illustrates how SocketIO delivers the keyboard to the Mevia application by dispatching a KeyboardEvent to the main Document Object Model (DOM) element document. Similarly, the controller can be initialized depending on what application is being used, such as switching from a keyboard to a touch interface. The "InitController" message triggers the mobile terminal to initialize and load a new or different controller, or simply reinitializes the current controller at 440. During this initialization process, authentication keys can be reissued and payment information can be collected from the user. In one embodiment, a payment request can be made from a service such as an online payment system, or token information can be collected using the end user's software cryptocurrency wallet account, making the use of the device as a controller not free.

[0092] Figure 5 and Figure 6 An exemplary method is shown that enables a server-based interface to serve a smart TV through a device connectivity platform architecture. More specifically, Figure 5 It describes how mevia.js is loaded and how it is used to display on a smart TV, and Figure 6 This illustrates a controller interface loaded by a user onto a mobile phone to control a smart TV. First, in... Figure 5 Step 500 initializes network communication using security credentials such as keys, certificates, and certification authorities. For example, a Secure Socket Layer (SSL) is used, where the private key, certificate, and certification authority files are loaded as part of a NodeJS session for secure WebSocket (WSS) transactions. This process starts WebSockets on a secure port (e.g., port 3000) and enables the use of HTTPS-based resources (e.g., loading images, CSS, etc.). In the next step, 505, the URL of the original application that will be used to interact with the application is set, and all resources of the application (including HTML, CSS, images, and JavaScript) are retrieved from the URL. To illustrate, a JavaScript game can be loaded from the original link, and subsequently, as part of the same process, all other assets are loaded into the DOM for a specific HTML page. Once the DOM tree is initialized, the web application used for the game is modified to add mevia.js, mevia.css, and other elements that can be dynamically added using dom.append(..) or dom.appendChild(..). At step 510, the "href" attribute within the application's DOM is modified, including redirecting links or adjusting resource paths. At this step, the DOM can add a mevia.js file, mevia.css, and modify all "href" values ​​required for the application to run. For example, if the game references images, fonts, and other resources in the original HTML to local references, these can be modified to server-side plus path references. At step 502, another path for initializing the application is executed. For example, step 502 involves the initial state of waiting for an application launch command with a specific application ID. To further illustrate, the application launches another application via the "LaunchApp" command using an AppID or application identifier that can be associated with a menu. At step 515, the application ID, such as a UUID, and some form of authentication, such as a token or key, are set. For example, once the device connectivity platform application's HTML has been updated, the device connectivity platform object is set with the AppID, UUID, and any potential authentication requirements (e.g., biometrics, user and password authentication, certificate-based authentication such as 802.1X, PKI, and other private / public key authentication).

[0093] At step 522, a command is sent to initialize the controller. For example, the "InitController Command" can actually be issued by the device connectivity platform infrastructure and sent to a specific controller. The InitController command can load new authentication requirements, which are imposed as part of the process of loading the specific controller. For example, the controller can be "disabled" until authentication is complete. As mevia.js modifies the DOM to evaluate JavaScript macros, this authentication can be added to the DOM as part of the controller. Authentication can require a username and password for a specific user at the controller level. In some embodiments, the controller can request a payment portal, and other payment portals include controllers that require a payment method or web widgets such as a "Pay now" button when a subscription to a specific service expires.

[0094] Following authentication, at step 520, specific parameters are used to configure the controller. For example, these parameters may include "keyboard" and "video," which are the input types for the application. In some cases, the keyboard is a 12-key keypad (0, 1, 2, 3…#). This could be a keypad or a character keyboard. To further illustrate, the device connectivity platform application or web interface will continue to load assets, including HTML, JavaScript, CSS, and images for the application being displayed on a smart TV or any display in general. The application is associated with a controller, and there are several standard controllers that can be implemented using HTML. The device that will control the smart TV can be a touch interface, which can then be controlled via a mobile phone using one or more controllers such as a keypad, keyboard, gesture-driven interface, or other types loaded by the user attempting to interact with the device connectivity platform application. Once the controller is defined, it is initialized, which can be done via the "initController" command sent to the controller at step 522.

[0095] Additionally, at step 525, the Quick Response (QR) code string can be regenerated using the AppID, UUID, and region or any other session value, and stored as part of the "qrcodestr" value. This "qrCode" value can also be a "fingerprint" of the stream being displayed, which can later be used to visually match that fingerprint to retrieve and match identical fingerprints.

[0096] At step 530, a quick response code (QR code) is set as a quick response code string. For example, a QR code can remain constant over a long period, and at the device connectivity platform infrastructure, simply when an HTTP GET / POST request is made using the content of the QR code, the QR code will be remapped accordingly to any other application, service, or interface that has already been redirected. However, for public places, the QR code can be dynamic and regenerated after a timeout is detected. It is well known that web interfaces can simply display QR codes as PNG, JPEG images, or even PDFs, and these can be printed by end users with smart TVs, or the QR code can be dynamically changed for display and presentation on the smart TV screen.

[0097] The QR code has a link that may have an embedded session identifier (session ID), a location identifier (Location ID), and other parameters associated with a specific controller associated with the device connectivity platform application screen displayed at that location. At step 535, the DOM is appended from the URL to the current DOM. For example, once step 530 is complete, the output of the DOM to the device connectivity platform application is updated at step 535. At step 540, the DOM of the URL is used to generate a response. For example, once a response has been requested, this DOM is sent as a reply to an HTTP GET request at step 540. The smart TV, equipped with the mevia.js library and / or gesture.js, can be served via webpage loading, pointing to the device connectivity platform's web application interface, or it can be part of a native application for the smart TV (e.g., Android TV, Google TV, LG's operating system (LG WebOS), Apple TV, Samsung's Tinzen platform), or simply by loading a webpage in the TV's web browser.

[0098] Figure 6 This demonstrates how to load `initController` and the new controller onto a mobile device or any client computer capable of loading HTML web pages. At step 600, the HTTP server is initialized using specified security credentials (e.g., keys, certificates, and certification authorities). For example, the controller requests server initialization, including loading certificates, private keys, and bundled certification authorities. At step 610, the application is loaded (e.g., using the provided AppID, UUID, and authentication details, such as the authentication context). For example, the device connectivity platform context is loaded as part of the device connectivity platform application and the LoadApp API, which includes a UUID that can be generated as part of the initialization process, an AppID that maps the application to resources, and any authentication objects that need to be verified by the controller accessing the Mevia application or have already been verified by that controller. At step 630, the controller's DOM or document object model is retrieved from the device connectivity platform application. For example, the controller's DOM object is initialized using HTML, CSS, images, and other JavaScript, including loading `gesture.js`, `gestures.css`, and other files required to load the controller. The library that handles events such as touch start, touch stop, touch end, keydown, Enter, swipe left, and swipe right is part of the gesture.js file. All of these processes are part of step 640, where the meviaverse object generates responses in HTML, JavaScript, CSS, and other resources based on the controller's DOM. In this example, the controller loads gesture.js, gestures.css, and other related controllers, such as a 12-digit keypad controller, joystick, or camera. As a result, the user subsequently loads the controller as part of an HTTP response generated in the previous step 650 and loaded at the client's device DOM in step 650, which is generated by issuing an HTTP GET or POST request to retrieve such a controller (e.g., HTTP GET / init / controller?keypad=12-digits). In other words, the purpose of gesture.js is to create SocketIO commands and receive all types of events sent to the Mevia application target displayed on a specific smart TV. Device connectivity platforms can create login, screens, and interfaces for application packages used for payments, gates, video conferencing, or display on smart TVs or other displays.Steps 600 to 650 can also be delivered to a mobile phone or tablet upon scanning a QR code associated with a specific smart TV or display, as a response to an HTTP or HTTPS request to retrieve the appropriate controller, for example, https: / / app.mevia.tv / keypad. This page renders all libraries, icons, images, and structures for the specific controller via a response command. With this controller, other controllers can also be loaded; for example, a camera server can be generated by embedding https: / / app.mevia.tv / camera / in a QR code, and the camera sends WebRTC video to the server, where computer vision tools or deep learning (e.g., LSTM networks) are used to recognize the video to load another video controlled based on the camera video input.

[0099] At step 655, if the authentication check is successful, the credentials are verified. For example, if the authentication is known or the user has already paid for access to the controller (e.g., a video game arcade), the controller is displayed.

[0100] At step 645, events from the DOM are used to update the DOM on the controller to reflect user interaction. For example, once authentication is complete and the user simply manipulates and interacts with the controller and receives DOM events, the DOM is updated accordingly. To further illustrate, switching the color palette from red to magenta would require DOM manipulation to dynamically change the color in the CSS style from "red" to "magenta". At step 625, the device connection platform is in a waiting state until the controller initialization process begins. For example, the gesture.js library expects at least "InitController" as a command, which empirically enables the new controller. In some embodiments, the user can "tap" or "double-tap" an application icon (e.g., a menu or slider, and launch a game) by creating an event and sending message.command="Launch," message.appid=AppID or message.uuid=UUID (which is associated with a specific icon selected on the screen). The Message.command "Launch" or "Start" can do two things, for example, in Figure 5 In step 502, it waits for commands from the application and uses a network redirection to the new "URL" to load the new page into the instruction (e.g., mevia.js) or modifies the DOM by replacing the document.body html DOM structure, and reloads the object called document.body and other HTML elements including document.head, as well as other parameters from the HTML of the other web applications being loaded.

[0101] In some embodiments, some applications may require authentication, and in this case, at step 620, the controller waits for the initController command via gesture.js. For example, a QR code is generated with a specified timeout value for the authentication process. Figure 6 As part of the process, the QR code may be displayed on the screen for a few minutes in step 620, or, at step 622, the generated QR code may be printed or displayed. Alternatively, it may be part of a DIV HTML tag that overlays the Mevia app displayed on a smart TV or a display controlled at step 624. In some embodiments, at step 660, payment can be verified if required.

[0102] At step 660, authentication is performed for the use of Figure 6 Whether any controller is loaded is optional, where loading a specific controller may require payment verification. A useful scenario would be in a parking lot, where the controller for opening / closing the door will not be loaded to the user unless payment has already been made previously using the keypad controller to enter the vehicle's tag number. In practice, the system allows users in the parking lot to control the LCD screen by loading a 13-bit keypad controller into their phone, subsequently turning the phone into an NFC reader (e.g., Apple Pay) to submit payment associated with the entered tag.

[0103] Controller in the device connection platform

[0104] As shown in Figures 7(a) through 7(c), there are three types of communication with SocketIO: a) using standard WebSockets as shown in Figure 7(a), where the controller 705 communicates with the device connectivity platform infrastructure 700 and sends commands and displays to Internet of Things (IoT) devices 720 and smart TVs 715, or other displays at 710 that may include other tablets, LCD screens, and devices. In Figure 7(b), WebRTC can be used alternatively to establish peer-to-peer communication between the controller 730 and the display 735, without the device connectivity platform being responsible for routing data packets from the controller device to the target display. Figure 7(c) shows how the controller uses WebRTC, but the device connectivity platform 740 creates an IP (Internet Protocol) tunnel 750 between the controller and the display, where the controller 745 uses WebRTC on the tunnel to send messages and events to those displays 755. These tunnels can also be GRE, GTP, IP in IP, or other VPN tunnels that encapsulate IP traffic from one server to the next. WebRTC is a technology that allows web browsers to stream audio or video media and exchange random data between browsers, mobile platforms, and IoT devices without a man-in-the-middle. In some embodiments, the channel server does this via a STUN / TURN server on the Internet (e.g., stun.eglacorp.com) and by setting them up as part of the WebRTC service when needed.

[0105] Figures 8(a) through 8(c) illustrate a use case of how a user launches a device connection platform application. First, in Figure 8(a), user 800 has scanned the QR code 815 below the smart TV 805. In Figure 8(a), slider 810 depicts several applications called Mevia App1, Mevia App2, Mevia App3, etc. In Figure 8(a), the user can proceed by "swiping right" 812, and the game application 814 is highlighted in Figure 8(b). The only communication channel between the phone and the smart TV is the Internet. Thus, the user can continue by "tapping" the button associated with the game application 814, which is also highlighted on the smart TV, as shown in Figure 8(c). In some embodiments, the game web application may be written in JavaScript and run in a browser, and has been modified to include mevia.js.

[0106] As a result, the initController command is issued to switch from the touchscreen to the game controller 825 and allow the user to play the game application 814. To exit the game web application, the user can use a "long tap" mapped to "Exit" to press any part of the screen and return to the previous menu application. This issues a new initController command and LaunchApp, and loads the previously initialized slider as shown in Figure 8(c). Any user can still send standard commands using the smart TV remote control 830. In some embodiments, optimization can be achieved, for example, by using Redis or other caching patterns and processing optimizations to load / unload the application. Redis can be used as a message broker to distribute and exchange events and messages from all controllers to the display, save images, cache JavaScript files, and add caches to the device-connected platform.

[0107] Figures 9(a) and 9(b) illustrate another example of a device connectivity platform application: a drawing program. The user can swipe right or left and locate the NFT Lab application 905 on the screen. Upon selection or generation of a "tap" 900, the DOM object is updated at the smart TV display, and an "initController" command is delivered to the mobile terminal, where the "Palette" controller 915 is subsequently loaded into the mobile device or tablet. Once the palette controller 915 has been executed (e.g., the window.onload() method), touch events are translated into messages, such as message.evt indicating the start of a touch at some (x, y, z) coordinate, touch movement, and message.evt indicating the end of a touch at some x, y, z position 930. In some cases, z may be equal to zero, due to the fact that most user interfaces are 2D, while for those creating 3D user interfaces, the z-axis will not be zero. As shown in Figure 9(b), the user interaction 925 with the palette controller 915 is displayed on the smart TV 920. In some embodiments, QR code 910 may be used to prompt the loading of the palette controller 915 on a mobile device.

[0108] Each smart TV, display, or sensor is associated with a QR code used to load the controller. For example... Figure 10 As shown, a database containing a list 1001 of QR codes is stored in the database, and application identifiers 1005 and application names, along with UUIDs or sequences of UUIDs 1010, are mapped to each application name in use. Loaded applications may not require certain types of authentication 1015, such authentication could include using WebRTC or the API navigator.mediaDevices.getUserMedia (which can be used for biometric authentication) via a mobile video controller. Each QR code can also be associated with an application URL (served by a web server using HTTP or HTTPS protocols) or a device connectivity platform interface 1020, which may include mevia.js, mevia.css, and other resources to transmit events to applications loaded via QR codes.

[0109] As noted, there are several types of controllers that can be loaded individually or as part of a package of three or more controllers. Figure 11 The types of controllers supported by the device connectivity platform are shown. Touch controller 1100 provides touch events, along with simulated mouse events, to the device connectivity platform application. Touch events are known and defined by the HTML5 specification; therefore, the controller can emit timestamps, touch start, touch end, touch movement 1102, or a combination of these events, which can be combined with timing values ​​to become commands such as "swipe right" 1104. In some embodiments, the controller can be a game controller 1110. The controller can also include haptic feedback, which can be added as part of an asynchronous message haptic controller command as part of an InitController command message, which can then be delivered to the controller to simulate vibration or haptic feedback on a mobile phone or tablet. As a result, game messages are generated and sent to the controlled application 1106, such as up button, down button, and corresponding timestamps. Game developers can also assign key sequences to in-game movements or macros, which are generated by... Figure 4 The macro converter 428 in the device connection platform server generates the data. In some other embodiments, the augmented reality (AR) headset will be used to process head gestures and its own controllers to create a virtual overlay of those controllers that will appear in the AR headset's field of view.

[0110] For example, a developer could implement a function called "abouttheSame()" that returns TRUE when the user has roughly tapped a location between the start and end touch X points and between the start touch X and start touch Y points, such that: function abouttheSame(a,b, thres=20) If Math.abs(ab)<=thres then true else false; … if (abouttheSame(touchstartX, touchendX, 25) && abouttheSame(touchstartY, touchendY, 25)) { if count_number_of_taps=1 within a delta_time of 500ms thenTap; if count_number_of_taps=2 within a delta_time of 250ms thenDoubleTap; if count_number_of_taps=1 within a delta_time of 3000msthen LongTap; if (mouse_move) reset_all_timers() }

[0111] The example pseudocode indicates that taps, double taps, and long taps can be detected by tracking finger events and positions on the screen. The thresholds can be varied depending on the application, and in some cases, game developers will know that digital signal processing filters or other methods can be used. Figure 11 The machine learning model is trained to detect those events.

[0112] Other controllers that can be used with the device connectivity platform include a camera or capturing device 1120. For example, once an image has been captured, a base64 image / png or image / jpg 1122 is then submitted to the device connectivity platform as part of a messaging system. Some other messages may include barcode information. This may be available locally on the device, or as part of the controller interface using the BarCode SDK from Dynamsoft, or directly by a web application running meva.js.

[0113] like Figure 11 As depicted, camera 1124 and microphone 1126 can be used to capture live camera feeds, camera stills, and audio generated from mobile devices or tablets. These controllers can issue messages with sequences of images captured via WSS or WebSockets, which may include audio buffers or still images from the camera. In other embodiments, the camera and audio / microphone can be delivered via WebRTC 1138, which can be channeled locally via IPinIP, IPSec, or any other channel created by the device connectivity platform, or using a standard STUN / TURN server (e.g., stun.eglacorp.com). This information can be collected by the controller via the "initController" command, or created in real-time depending on how the device connectivity platform application is controlled. In the absence of video, message.evt will describe a "StreamAudio" event and a remote WebRTC socket will be opened to interact with the audio tag at the receiver, for example, for voice-only applications (e.g., walkie-talkies, push-button calls). In some embodiments, for multimedia, audio, and video applications, "video streams, audio streams" can be constructed, for example, using a WebM encoder for video and an Opus encoder for audio, using WebSockets or WebRTCPeerConnection JavaScript API 1128.

[0114] In some embodiments, video and audio codecs may require additional transcoding or encapsulation in other messages and may include emulation as a Universal Serial Bus (USB) device for local or non-network applications that expect a USB identifier (USB VHCI root hub), wherein the USB device emulator packages WebSockets or WebRTC streams for the USB identifier for use by those local applications that require it.

[0115] Another controller that can be used with applications in the device connectivity platform is a "3D gesture" generator, which emits the "accel" event 1140 and the accel_x, accel_y, and accel_z elements. Typically, timestamps can be added to the accel_x, accel_y, and accel_z payloads, and the sampling rate of these events is set by the InitController command. However, if the time interval is set to zero and a time delay is required to calculate the gesture, it will depend on the device being used. This functionality can be implemented in HTML5 browsers by publishing the windows.DeviceMotion event, which can be captured in the callback of window.addEventListener("devicemotion", updateData), where: function updateData (e) var acc = e.acceleration || e.accelerationIncludingGravity; accel.push({t:delta_time, x:acc.x, y:acc.y, z:acc.z}); if len(accel) == N: XMLHhttpRequest(accel); accel=[]

[0116] Subsequently, acceleration information is captured in the "accel" array, which contains delta_time or the time in milliseconds between samples, where accel.x, accel.y, and accel.z are the acceleration values ​​at the x, y, and z coordinates. This information is then forwarded to a neural network or any machine learning interface for comparison with the learning data, such as... Figure 11 As shown. In this way, gestures such as up, down, swipe right, swipe left, tap, and exit can then be learned and mapped to a specific user profile or a general user profile with training information stored in the device connectivity platform.

[0117] All these message.evt commands are sent to the device connectivity platform to reach the content routing module ( Figure 21 As described in the document, the content routing module determines the appropriate route for data packets and the events generated by one or more controllers required by a particular application.

[0118] Other controllers could be USB card reader 1150 emulators, which can connect to mobile devices 1154 (e.g., stripe readers). In these cases, custom messages can be created and sent to the content router via the message.evt message stream, and appropriately translated to process JavaScript messages or events to be sent to a remote website. For example, Near Field Communication (NFC) can read NFC cards and send information via the message.evt value using the NFC card payload, which can be translated into a keyboard set of commands or categorized as a custom event with a specific JSON payload that can be available as part of a web application interface.

[0119] Subsequently, typically, the device connection platform is in Figure 18 The system is depicted as a series of controllers 1805, where the control server has HTML 1810, a web HTTP server 1802, gestures.js, and AI / ML to handle controller learning and controller packages. A message transmission queue 1830 contains all message.evt, message.x, y, z coordinates, and all other messages for gesture control devices and other controller packages or series. 1812 and 1808 represent web application (app) ingestion, which may also include AI and machine learning as part of a MEVIA application or Mevia server block 1816, and receiving commands that can be transformed from raw messages originating from the controllers into JavaScript commands 1825, depending on the HTML used by the application 1812. In other words, depending on the configuration used and the type of application loaded, messages originating from the controllers are subsequently transformed into mouse events, JavaScript events, and WebRTC / WebSocket streams. The Content Router 1845 element is responsible for capturing all messages and events "RAW" from the controller, identifying what needs to be converted, and managing other router functions for WebRTC and WebSocket video and audio services. It then maps the video and audio stream to the IP tunnel and manages it under the STUN / TURN server 1820.

[0120] Figure 19 A system for managing platform applications that connect two or more devices is shown. Figure 19 This includes a control server 1900 for the game controller and its associated Mevia app server 1902. The URL is retrieved by screen 1904 and contains the Mevia application, subsequently controlled by the control server 1900. As shown, message.evt or RAW events are sent to the controller, which can receive the InitController command 1910. The core of the system is depicted at the routing component 1940, where all routing QR codes, application identifiers, UUIDs, databases, and routing parameters are configured. As a result of processing in the routing component 1940, events are converted into commands or WeRTC / WeSocket streams, subsequently retrieved or pushed to the Mevia App server 1902 via HTML and JavaScript, CSS, and other media files 1906 and message.commands 1915, and retrieved by the client at screen 1904. Clearly, many of these message.evts are converted into message.commands 1915 and appropriately processed by the Mevia App server architecture. Multiple controllers can then be used, and the game controller 1912 and camera 1920 can be combined.

[0121] Real-time, streaming, and video conferencing applications in device connectivity platforms

[0122] For applications requiring video and microphone controllers and navigators, authorized access may be required in Figures 12(a) and 12(b) via `navigator.mediaDevices.getUserMedia(..)`. Once the user grants device authorization, the controller provides `gesture.js` with access to either WebSocket 1222 or WebRTC stream 1224. The `message.evt` event follows the same routing path, rendered by a page including `gesture.js` and associated JavaScript to establish a WebRTC session or deliver data via WebSocket, image, audio buffer, or data. One objective of Figure 12(a) is to create a simulated USB interface to a web application that loads video and audio via a simulated web client interface. In this embodiment, the Mevia App RTC server 1218 is connected via the simulated web client interface. The "Simulated Network Client to MEVIA APP" module 1216 can then perceive commands (e.g., message.evt) from the controller 1225 as a USB HID interface, CAM0, CAM1, CAM2 1212, or a USB WebCam simulator 1208, which is presented to the simulated network client as a CAM EMU driver. The USB WebCam simulator 1208 encapsulates a USB WebCam descriptor (e.g., bDeviceClass, bDeviceSubClass, and a professional interface association as bFunctionClass 14 for video). Subsequently, the data or payload of the connection created by the USB driver is a WebSocket or WebRTC stream generated from the controller 1225.

[0123] Other web applications can use the mevia-webrtc.js interface 1230 and mevia.js 1228 because those applications have already implemented WebRTC natively. In those cases, Mevia-WebRTC will be directly written to the audio or video HTML tags that are part of the application. In other words, in this embodiment, the Mevia App does not need to be modified at all and simply emulates a web client connecting to the Mevia App server containing the application, which can be a local application on the device's connectivity platform or a remote application hosted on a different domain. The emulated web client application then becomes the client, and the Mevia App server is the content displayed on the television.

[0124] In some embodiments, the Mevia App RTC server 1218 may not display content correctly to a particular user, for example, when the Mevia App is not designed for smaller screens, or due to any other potential drawbacks, such as the Webkit browser of the smart TV being unable to handle or display certain aspects of the application. In these cases, a simple "user agent" check of the HTTP requests from the smart TV may be necessary, and the process in Figure 12(c) will be usable on all TVs regardless of compatibility issues with their web rendering engines. As shown in Figure 12(c), the use of different Mevia App server displays, which may include simple HTML, is depicted, such that: <canvas id="”meviaapp”">< / canvas> <script>Player = jsMpegPlayer("wss: / / ipaddress:8001 / meviapp”,"canvas”: document.getElementById("meviaapp”);}< / script> <script type="”javascript”" src="”jsmpeg-min.js”">

[0125] 图12（c）示出了组件1280包括用于mpeg2视频和mp2音频的FFMPEG编码器，对于基于捕获的所有HTML页面的屏幕的Puppetter，作为Mevia App服务器，mevia.js与mevia-webrtc.js之间的所有消息传输的结果而示出。作为示例，Puppeteer可以写入所有捕获的JPEG图像，并将它们输送到FFMPEG过程，以将它们封装为MPEG帧。Capturing Stage - Video:Capture image in JPEG or PNG from Mevia App Server HTMLSave Images at Local DirectoryWrite image to pipe:0Capturing Stage – AudioCapture audio from Mevia App Audio ChannelRedirect audio as http mp3 streamAt FFMPEGTake pipe-in from Capturing StageTake audio from http3 Capturing StageGenerate mpeg2video and audio and serve it as Web Socket oruse NGINxwith rtmp or write HLS to Apache Web serverWrite to pipe: 0At Websocket Server – Websocket IP address and port XYZRead from pipe: 0 and write all buffers to a websocketfor Mevia AppDisplayWait for WebSocket Requests.At MEVIA App Display PagePoint jsMPEG to wss： / / Websocket IP Address with Port XYZ

[0126] 本质上，图12（c）示出了在组件1280处将音频和视频封装在mpeg2视频帧中。随后，这些视频和音频帧由WebSocket服务器1285提供服务，并且USB或WebRTC映射器1288组件确定适当的网络服务器以用作response.redirect(…)，或者重新定向命令被发布到智能电视显示器以指向显示服务器1290，其中jsMPEG或JavaScript MPEG（例如，jsmpeg.org）项目可以绘制到画布（canvas）。类似地，可以利用RTMP、HLS和针对这些实现的低延迟HLS来完成实现，可以不需要WebSockets服务器，并且将使用具有<video source=”… ">指令的HTML标签来代替jsMPEG播放器。WebRTC映射器1288定义Mevia App服务器显示器1290的位置。

[0127] 图12（c）中示出了USB或WebRTC映射器1288。该模块示出了特定的源IP地址和USBID如何被映射到目标IP地址和由USBIP守护进程（daemon）模拟的USBID。USBIP是Linux和Windows机器的现有组件（http： / / usbip.sourceforge.net）。USBIP命令允许将位于远程机器的具有特定USB标识符的远程USB装置挂载到位于"IP地址1”处的USB装置，将另一个USB IP守护进程挂载到位于"IP地址2”处的机器，USB装置是本地的，并且所有传输都通过因特网链接来完成。

[0128] 然而，由于来自WebRTC流1224或WebSocket 1222的视频信息流从控制器作为实时流递送，这些流由设备连接平台在对端点（peer point）1206处进行处理。当视频和音频信息流由WebRTC对等连接接收时，例如，控制服务器使用的IP地址是1.1.1.1，而在控制器1225处加载客户端的IP地址是1.1.2.122，在设备连接平台内的本地服务器处进行与WebRTC的连接。

[0129] 由于控制器处的HTML标签具有<video id=”remote”>， <video id=”local”>在设备连接平台处就是 远程的。作为WebRTC界面的配置的一部分，可以设定STUN服务器1250或者设备连接平台可以在对端点1206处的控制服务器与服务器之间创建IP通道。本领域专家将知道STUN / TURN服务器将继续实现与IP通道相似的结果。这样，在对端点1206处，来自控制器1225的所得到的视频和音频馈送随后被放入包或帧中，它们随后被递送到在对端点1206处搭配的USB Web Cam模拟器。WebRTC RTCPeerConnection()At the end-point of the RTCPPeerConnection all packets withvideo andaudio are bufferedPackets are written to pipe:0USB WebCam Emulator()The USBIP interface is initialized to issue an USBoverIPinterfaceDescriptors for a MEVIAWeb Camera are initializedPackets from the WebRTC Peerconnection are written to theUSBWebCamEmulatorIf transcoded is required, packets can be first passed toFFMPEG fortranscoding to H.264 video and AAC audio.

[0130] 由于来自USB WebCam模拟器的具有视频和音频的网络信息流被封装在USBIP信息流中，其中具有视频和音频的有效负载必须被正确地路由到特定会话或Mevia App RTC服务器1218。这样，使用USB虚拟路由器1200，并且路由表1202将包括目标IP地址和源IP地址。USB 守护进程从USB虚拟路由器1200加载信息，并向包含网络摄像头模拟的USBID所在的IP地址发出"USBIP挂载”命令。例如，目标IP地址可以作为远程docket容器"运行命令”来执行，或者如果特定服务器已经存在，则可以从USB虚拟路由器1200向USBIP 守护进程服务器1210执行"SSH”命令，例如，从USB虚拟路由器1200，"ssh -u device connect platformDest_IPAddress： / usr / bin / usbip -attach Source_IPAddress”。

[0131] 结果，USBIP 守护进程服务器1210必须与MEVIA APP模块1216的仿真网络客户端（MEVIAAPP的模拟网络客户端）驻留在相同的服务器内，由于使用网络浏览器的浏览器实例随后将连接到MEVIA App服务器。MEVIA应用程序服务器实际上期望客户端可以将标签HTML视频渲染为本地的和远程的，以用于连接到MEVIA App服务器的其它客户端。由于预期智能显示器将模拟网络客户端的结果可视化到MEVIA APP模块1216，或者将由Mevia App RTC服务器1218所观察到的内容的副本可视化到客户端1233中，或者将来自Mevia App RTC服务器1218的视频和音频流渲染到客户端1233中。

[0132] 设备连接平台允许以下使用情况：a）为了可视化MEVIA APP 1216连接到MeviaApp服务器的结果，mevia-webrtc.js和mevia.js 1228被指示将从模拟客户端到MEVIA APP1216的DOM复制到客户端1233的DOM中，其包括将MEVIA APP 1216处的所有视频流复制到客户端1233中的画布对象中，或者使用MEVIA APP 1216的截屏与客户端1233创建WebRTC通道；或者b）如图12（c）继续，其中，FFMPEG和puppeteer也作为MEVIA APP 1216的一部分执行，并且继续生成要由MEVIA App服务器-显示器1219截屏而不是MEVIA App服务器1218可视化的截屏。Mevia App服务器简单地显示在显示服务器1290处示出的内容或者使用MPEG2视频和mp2音频或HTTP实时流传输流（HTTP Live Streaming，HLS）传输到画布的流。观察到mevia.js随后作为模拟服务器的一部分而不是作为MEVIA App服务器的一部分加载。

[0133] 换句话说，mevia-rep.js 1228从MEVIA APP 1216接收DOM对象值，或者使用WebRTC或WebSockets将DOM从MEVIA APP 1216复制到正在客户端1233上显示的内容中，或者简单地经由jsMPEG来流式传输MEVIA APP 1216的实时版本，该版本由组件1280 mpeg2视频处理并用作渲染。与DOM对象副本相反，使用显示服务器1290渲染的优点在于支持CANVAS的任何智能电视显示器将能够渲染和处理来自MEVIA APP 1216的结果。显然，该实施例的使用情况是创建基于网络的视频会议工具，该工具加载可能需要来自操作系统的特定支持的USB摄像头或USB装置。因此，WebSocket服务器1216可以被实现为Windows客户端、Linux或无头软件（headless），这取决于视频会议软件。

[0134] 对于其它基于USB的界面也是如此，其中，期望操作系统或内核模块加载本地驱动器和资产。

[0135] 假设特定的视频会议不需要USB摄像头。图12（b）示出了如何创建依赖于mevia-rtc.js协议的本地解决方案，相似于图12（a）描绘的协议而不是模拟IP上的USB，而是简单地捕获WebRTC信息流和WebSocket信息流并被直接转发给视频会议应用程序。

[0136] 如图所示，WebRTC 1254描绘了经由navigator.mediaDevices.getUserMedia的gesture.js的授权和访问，以及作为控制服务器HTML页面的一部分的WebSocket 1222或WebRTC 1224对象的实例。

[0137] 相似的过程将用于任何其它USB界面，例如，基于USB的显示器、充电器、键盘、AR头显等。

[0138] 这样，ICE服务器或STUN / TURN服务器1250被设定以处理RTC通信。类似地，设备连接平台可以创建IP通道以实现不同端点之间的点对点链接，例如，控制服务器到Mevia应用程序服务器或模拟WebRTC代理1246之间的SSH通道。观察到示出了到WSS实体1240的WebRTC到WebRTC映射器。

[0139] 该服务器可以将WebRTC信息流直接转发给模拟WebRTC代理，或者可以由WSS捕获以创建交换视频和音频信息流的WebSocket界面。换句话说，到WSS实体的WebRTC到WebRTC映射器是WebRTC连接的对等端，并作为对等端，随后它在端口8000处作为WebSocket服务器的一部分来服务。作为对等端的接收方，到WSS实体1240的WebRTC到WebRTC映射器可以检索对等端中的每一个的SDP会话，并且可以将WebRTC信息流来回复制到其它对等端，或简单地将待服务的缓冲WebRTC信息流缓冲为WebSocket信息流1244，例如：wss: / / server:8000 / sdp1和wss: / / server:8000 / sdp0。

[0140] 来自WSS流1252和WebRTC流1254的内容被记载，WSS流1252可以是WebSocket流，而WebRTC 1254可以是WebRTC流1254，其中客户端1233处的对等端是mevia-rtc.js。显然，STUN服务器1250可被放置为从WebRTC到WebRTC映射器到WSS实体1240到客户端1233的WebRTC信息流的一部分。另一实施例可以是模拟WebRTC代理1246，模拟WebRTC代理1246可以加载mevia-webrtc.js 1248，并且WebRTC对等端可以是模拟WebRTC代理1246的一部分。显然，随着模拟WebRTC代理1246将HTML从Mevia App RTC服务器1218处的视频会议应用程序和WebRTC到WebRTC映射器加载到WSS实体1240时，适当的跨源资源共享（Cross OriginResource Sharing，CORS）字段头必须实现为能够从Mevia App RTC服务器1218和到WSS实体1240的WebRTC到WebRTC映射器检索WebRTC信息流。如上所述，Mevia应用程序服务器用户界面可以a）利用WebRTC和视频标签1246在本地进行渲染，或者b）使用图12（c）的流式画布实现方式并使用例如jsMPEG.js库进行渲染。

[0141] 这是与图12（a）中相似的情景，观察到mevia-rtc.js和mevia.js被用于到MEVIAapp RTC服务器1218的模拟WebRTC代理1246。假设WebRTC应用程序已被构建且不需要USB摄像头，而仅进行流式传输，例如来自电视频道和用户摄像头的具有音频的流。在该情景下，远程视频1232和本地视频使用"视频标签”作为WebRTC可视化方法进行呈现。远程视频来自其它流源，而本地视频是来自控制器1225的源。在该情景下，Mevia应用程序RTC服务器1218从用作Mevia应用RTC服务器1218的对等端的模拟WebRTC代理接收WebRTC信息流，在一些实施例中，可以建立随后的对等端。我们识别，移动电话或控制器1225处的对等端0、作为WebRTC映射器设备对等端的对等端1、模拟Web RTC代理1246处的对等端2、实际Mevia AppRTC服务器处的对等端3和挂载到该WebRTC服务器的所有其它对等端（例如，一对多会议）。Mobile Phone to WebRTC MapperPeer0 - Peer1Web RTC Mapper to Emulated Web RTC Proxy Point of View to Show onSmart televisionPeer 1 - Peer 2Emulated RTC Mapper to Clients attached to Mevia App RTC ServerPeer 2 - Peer 3

[0142] 一个目的可能是将对等端0（Peer0）控制器与对等端3（Peer3）客户端链接在一起，使得连接到MEVIA app RTC服务器的所有网络客户端可以看到对等端0的流。

[0143] 然而，记录能力、转码和图12（c）实施例的使用可能已经被禁用并且将被该配置绕过，但是显然对等端0源可以作为流会话的一部分的远程位置处的所有客户端中的标记为"远程”的所有HTML视频标签的一部分来显示。

[0144] 因此，智能电视的视点正好定位在加载mevia-rtc.js和mevia.js的Mevia App服务器的"模拟”WebRTC代理服务器1246处，而不是在1218处的Mevia App RTC服务器。在该情景下，向Mevia App RTC服务器做出HTTP GET请求，并且具有不同标识符的所有画布、视频和 / 或音频HTML标签被加载到Mevia App服务器中。本地视频标签或对控制器1225源的引用被显示为正由WSS实体1240、STUN服务器1250和WSS流1252处理的对等端。包括远程视频1232的所有其它对等端和源可以由STUN服务器1250处理以实现NAT或防火墙下的正确信令。因此，由模拟WebRTC代理1246作出的HTTP GET请求需要由智能电视显示。这是URL，其也将接收来自所有控制器的所有message.evt命令和所有相关联的事件，这些控制器可以与特定MEVIA应用程序相关联。然而，本领域的专家将知道，可以直接建立与Mevia App RTC服务器的另一会话，并且将所有对等端看作观察者，例如作为网络参与者，并且根据智能电视配置以及用户已经扮演的什么类型的角色，可以为使用应用程序的用户渲染不同的URL。

[0145] WebSocket流也可以用作WebRTC视频和音频或数据内容的递送的一部分。在该特定情景中，Mevia App RTC服务器1218可以简单地绕过模拟RTC服务器，并将视频和音频直接加载到本地视频，从WebSocket信息流1244直接加载wss: / / 信息流。视频和音频作为视频帧被检索，该视频帧被渲染到画布和HTML音频标签中，或者直接渲染到HTML视频标签中。在这种情景下，当仅使用WSS流时，则视点是客户端1233不是模拟WebRTC代理1246，或者来自Mevia app RTC服务器的内容，该Mevia app RTC服务器也可以包括作为视频会议事件的一部分的其它对等端。

[0146] 在使用来自WSS 实体 1240的wss: / / 信息流（例如wss: / / server:800 / sdp1）在Mevia APP RTC服务器处更新画布的情况下，在这种情景下，可能需要设备连接平台的预先构建的应用程序，并且该预先构建的应用被用作创建用于设备连接平台的视频会议应用程序的某些应用开发者的推荐实践。

[0147] 另一应用是示出来自多个电视流、摄像头流和其它源，这些源形成其它视频会议服务器或过顶服务器（over-the-top server），过顶服务器可以通过WebRTC流或WSS流传输内容。

[0148] 在这两种情景的任何一种中，图12（c）可被用于通过创建屏幕捕获序列并继续混合来自所得网络客户端的音频来处理任一情景的渲染，由于它被感知为mpeg2视频和mp2音频，或者可以使用H.264视频编解码器和对于HLS或低延迟HLS（HTTP实时流传输）环境的传输流（TS）中嵌入的音频mp3或AAC编解码器进行编码。

[0149] 将Windows机器整合到设备连接平台中

[0150] 迄今为止，已经渲染了由具有基于网络的小部件的智能电视级渲染的网络应用程序。然而，设备连接平台也可以填充有可以使用图11中呈现的控制器系统控制的操作系统，并且WindowsVM或服务器可以成为设备连接平台的一部分。例如，可以经由设备连接平台来渲染和控制Windows应用程序，该Windows应用程序可以包括本地OpenGL、传统DOS应用程序、本地Microsoft Office、计算机辅助设计（CAD）程序、Adobe工具以及作为可执行代码在Windows上运行的许多其它高计算本地应用程序。在图13中，本地应用程序1304描述了用于Windows10或Windows11的本地应用程序。在该实施例中，USBIP模块1307可以用于处理所有USB相关的设备，诸如与控制服务器1320界面的摄像头和麦克风。参照图12（a）至12（c），具体地说，USB虚拟路由器1200，与USB Web Cam模拟器1308的连接是可能的。来自控制器的摄像头和音频可以被发送到虚拟Windows终端1304，并被用作本机功能。

[0151] 主要的要求是作为Windows服务或本地应用程序来加载，Mevia Legacy控制器应用程序1306负责将来自图11所示的控制器的Message.evt命令转换为来自Windows的WM_消息并且使用WindowsAPI系统。Windows消息传输路由器1310负责将来自位于一个或多个Windows机器或装置的若干控制器基于Windows的消息的映射，该Windows机器或装置遵守Windows消息传输。例如，如果"X”按钮位于100，100，则来自设备连接平台的敲击消息可以对应于WM_CLOSE。可以被转换的另一事件是鼠标事件，其通过将电话处的触摸事件转换为例如WM_MOUSEMOVE，x，y，其由1306装置接收并经由Send消息API界面发送到Windows过程。随着声音和屏幕捕获被获取，使用X11抓取（grab）界面从诸如FFMPEG 1302等工具发送这些声音和屏幕捕获，观察者认为不再需要Puppeteer，因为主要目的是渲染Windows本地应用程序而不是基于HTML的应用程序。Web socket服务器1300在例如IPAddress：8001 / office处创建，并且该Websocket或安全的Websocket（WSS）信息流1338随后可以由Mevia App服务器1322检索并经由目标画布1325显示。显然，控制服务器1320将处理来自网格1330（用于打开鼠标模拟1333）和键盘1335（用于键入命令）的所有手势。如图13所示，WebRTC对等映射器服务器1305也可以检索WebSocket服务器1300内容。

[0152] 观察到，服务器和客户端计算机仅是HTTP和其它协议所使用的定义，但是在一些情况下，服务器可以是单用途计算机，并且客户端可以是基于云的系统。因此，CPU、存储器大小、客户端数量不是对定义服务器和客户端的限制。

[0153] 如图所示，设备连接平台随后需要WebSocket路由来确定将什么应用程序IDS和什么UUID映射到什么特定目标，如果那些应用程序正在执行用户的会话，则从应用程序到应用程序的切换将仅需要更新诸如jsMPEG等播放器所更新的目标画布1325或目标div。例如，用户可能已经在一个特定服务器中运行了Office，并且可以在另一个服务器中运行Skype，或者两者均可以在同一服务器1315中运行。然而，用户可以通过简单地切换URL并且重新加载正在播放器1322处显示的播放器来从Office切换到Skype。

[0154] 在设备连接平台中使用语境感知的控制器和智能电视标识

[0155] 迄今为止，我们已经提出使用QR码来将特定设备连接平台应用与控制器进行映射，并且与不同的Mevia应用程序交互。然而，图14示出了如何利用流式传输的异步特性，以及如何快速确定哪个智能电视在使用中以及如何控制它。假设自适应HLS流由M3U8流媒体（streamer）1410和1420使用，示出了如HTTP实时流传输标准所定义的传输流的序列，其中stream0-4到stream0-11.ts 1410是流的一部分，并且Stream0.ts到Stream7.ts是另一个流的一部分，其中一个可以在700Kbps下而另一个可以在2Mbps下。这种流媒体还可以由MPEG DASH分割器实现，该MPEG DASH分割器还可以与包括一组片段的MPD（媒体呈现文档）一起使用，并且使用作为MPD的XML清单的一部分的音频 / mp4、视频 / mp4、"AdaptationSet”实体。出于互相关的目的，M3U8或MPEG DASH流媒体1410和1420高度互相关，由于它们来自于同一个流并且是同步的。然而，在日志文件1405中，系统可以缩小正在下载什么、何时下载以及由谁下载。例如，IP_client0获得Stream4.ts而IP_client2正在处理流Stream0-11.ts，这全部取决于使用中的播放器。因此，通过观察，我们知道stream4.ts和stream0-5.ts分别映射到IP_client0和IP_client1。因此，如果移动设备可以呈现具有与Stream4.ts 1430较高的互相关性的图像，则可以得出结论，设备随后使用IP_Client0，并且可以容易地识别映射到该特定位置的UserAgent信息。该技术是有用的，由于它被设计成替代QR码并且将位置映射到用户界面现在可以被控制的特定设备或智能电视。此外，可以使用三角测量的其它方法来进一步缩小使用中的适当电视机，例如位置指纹、超宽带定位或本领域中已知的其它方法来区分具有相同内容的多个TV。在诸如Android的操作系统中，类android.core.uwb可以被实例化以获得定位信息，并且能够经由webkit应用程序本地访问。UWB服务并且监听TCP端口4444，并且webkit可以经由http 或者https: / / localhost:4444 / uwb检索与空气标签或设备周围的UWB令牌来检索UWB值。在该特定实现中，androidx.core.uwb实例化UwbClientSessionScope以及UwbManager界面，以进行范围测量并检测正在利用正确的流来控制或识别的正确的TV客户端。在该特定情况下，IP_Client1还与UWB_1相关联，而IP_client2也与UWB_2相关联。类似地，通过使用NINearbyAccessoryConfiguration创建和应用并获得装置"会话”（:didGenerateShareableConfigurationData:for:），iOS装置被称为iPhone。因此，类似于针对安卓呈现的，应用程序可以集成任一android.core.uwb或NINearbyAccesorConfiguration或NINearbyPeerConfiguration界面，以检测附近的TV或显示器，并且进一步将设备隔离为控制器，甚至提供需要被控制的TV的位置的视觉反馈。该邻近位置信息被发送到MeviaVerse系统来处理，可以通过其自身进行处理或者与QRCode结合来处理，从电视机捕获视频图像。这样，即使利用如图14（c）所示的映射，也可以呈现对什么TV由MEVIA控制器控制的适当辨别。

[0156] 元素1405和1415是具有与正在广播的流或会话1410和1420相关联的UWB空气标签的智能TV。在1430处的移动电话或AR耳机（headset）现在可以显示TV1和TV2控制器，通过也使用UWB的空气标签和每一个TV的"附近”位置信息来辨别正在广播的每一个TV的流（参见图14（a）和（b）），下载用于每一个TV的AppID、UUID。在一些情况下，扫描QrCode可能不是必需的，由于UWB地址和在每一个TV上显示的图像将足以基于从UWB信号计算的相对距离和位置来确定使用和控制哪个TV。

[0157] 这在机场、餐馆、酒吧、俱乐部中是有用的情景，其中多个智能TV正在显示相同的馈送或不同的馈送，并且QR码没有被设备的摄像头捕获，相反，UWB 空气标签位置信息可以通过在到该TV的距离较近的方向或TV所位于的位置上移动设备并且将该信息发送到MeviaVerse系统以用于处理并且允许命令从电话或AR装置发送到TV来使用。

[0158] 在只有图像区域可用的情况下，在计算设备或电话处，用户可以在特定智能电视串流特定广播1435时从该特定智能电视捕获图片和视频流，随后可以找到流1438。可以利用消息（例如，SMS、WhatsApp、iMessage等）来捕获信息，随后客户端从HTTP服务器记录1440，该HTTP服务器正从HTTP实时流传输进行访问。随后，互相关1442存在于N帧的捕获的信息、Windows捕获流之间，并且在特定时间生成QR码1444，随后将其与认证值一起显示给已经被定位或识别的假定的智能电视1446。随后，用户将继续扫描QR码，并且认证和验证是发出请求的相同源，并且已经错误地识别了另一智能电视。

[0159] 图15更详细地示出了该方法，在步骤1500处从用于smart TV的所有源读取access.log并且在步骤1505处利用所有源从HTTP服务器标识access.log，其中，来自HTTP服务器的日志被识别为所有源，其中所有智能电视检索在任何特定位置处投影的流。日志文件包含电视的用户代理、每一个电视源的IP地址、以及什么是电视检索的最新流。因此，IP地址被映射到特定传输流（TS）段。首先，假设移动设备与智能电视位于相同的IP地址或子网处，这将显著地缩小设备数量。在步骤1510处，如果存在少于一个源，则仅一个智能电视是应答，或者如果不存在，则没有实现的识别，由于这样的用户将简单地接收超时消息，或者通过观察不到反馈，用户将理解如果QR码大于一个潜在匹配，则必须从"所有源”手动扫描QR码。该系统还可以使用诸如ip2location.com等额外API或其它提供者，以根据由第三方服务辅助的潜在IP地址匹配来缩小搜索和流的数量。此外，图15可以与超宽带（Ultra-Wide Band，UWB）标签结合，以提供电话可以发送的额外定位信息，如通过使用Apple空气标签或用于定位和本地化的其它UWB令牌的毫米位置跟踪。

[0160] 在没有找到匹配的情况下，可以应用蛮力法，并且在步骤1515处可以在循环中开始，并且来自所有流的所有源将需要被匹配。

[0161] 假设已经实现了IP地址与智能电视的匹配，并且源的数量很少（例如，NAT在公司处转换所有请求），则在步骤1525处，在互相关计算中，将来自所有源和最近在步骤1520处检索的所有流的IP地址与移动设备在步骤1515接收的所有帧进行比较，并且计算最高值。

[0162] 在步骤1525处，帧数中的最高2D互相关标识Ri上的特定索引I的位置。在步骤1530处，随后将该位置与可以使用内容路由器映射的特定智能电视相关联，并且随后，具有"DisplayQR Command”的message.evt利用生成的QR码发出，该QR码可以覆盖在特定时间正在流式传输的内容上。一旦用户在智能显示器上可视地看到QR码显示，则正常过程将开始，并且在其它情况下，在步骤1535处认证可以被设定为假，并且在步骤1540处通过接收来自原始消息的应答（例如，其可以包括iMessage、SMS、WhatsAPP消息等）而不使用QR码来简单地获得对电视的控制，其中，用户将继续加载URL并通过连接到与智能电视相关联的特定控制器来控制电视。

[0163] 用于设备连接平台的WIFI加密锁

[0164] 应该理解，并非所有电视都与WebKit或其它浏览器兼容，并且一些电视，尽管与HDMI输入兼容，但不是智能电视。对于这些情况或对于用户需要更多控制的情景，图16（a）提出了硬件替换方案，其生成QR码并生成HDMI信号以显示流，并且还使用WIFI连接到因特网。如图所示，LCD显示器用于显示QR码1630并且在加密锁1600内部，HDMI输出1603用于控制操作系统1625（例如，ChromiumOS），并且加载mevia.js. mevia-3d.js和允许经由操作系统1625进行控制的任何其它mevia相关的源1620。MeviaApp 1605和AppUI 1610包含加载操作系统1625和验证所提供的"meviapp”（指向特定URL的配置）被加载并用于配置WIFI所需的固件。为了配置WIFI，使用蓝牙低能耗协议将配置从移动设备"传递”到加密锁。图16（b）描绘了如步骤1650所示的经由BLE方法的认证和WIFI准备，例如，EValue或加密值是具有公钥的UUID，随后通过"设备连接平台URL”（例如，https: / / device connect platform.com / auth?Evalue=XYZ....”）的连接生成QR码，EValue设定为"XYZ...”，这被显示60秒并且在步骤1656处从移动电话扫描QR码，并且在步骤1660处提取待激活的UUID。

[0165] 随后，在步骤1665处，加密锁状态机将显示器替代为"Waiting BLE WIFI”，以通知用户BLE正被用于配置WIFI。一旦激活了加密锁WIFI，则可以利用BLE协议对其进行配置，并且一旦加密锁连接到因特网和设备连接平台就检查UUID是否被激活，如果没有，则将停用加密锁并且将重新开始该过程。

[0166] 存在多种加载gesture.js和mevia.js库的方式，以及控制器如何可以仅是控制器1700或仅是显示器1710，如图17（a）至17（c）所示。而且，控制器还可以是显示器，并且反之亦然，这意味着屏幕的一部分可以用于触摸，而一部分用于显示，如客户端1720和客户端1730中所示。在客户端是控制器和显示器的那些情况下，可以使用不同的端口来处理组合的mevia-gesture.js的信息流。最后，设备连接平台允许两个或更多个混合控制器-显示器1740、1750和仅是显示器1755的其它装置。

[0167] 有线及卫星系统中的设备连接平台

[0168] 本公开的另一方面是与有线卫星系统集成的能力。如图20所示，流2000正在本地存储中与QR码2010一起显示。正如标准遥控器一样，小键盘控制器2014被示出为扫描之后的移动电话界面的一部分。控制器2014可以连接到支付门户2015，该支付门户随后可以通过验证特定用户是服务的支付客户（例如，如Sling电视或Comcast）来验证系统的使用。设备连接平台2016向终端用户提供交互式体验，并且控制器和MeviApp服务器生成HTML和JavaScript结果，随后从小键盘或其它控制器2014管理这些结果。头端系统2020随后可以使用高速缓存服务器2022检索与音乐和 / 或视频2030一起渲染的所有生成的HTML和JavaScript，随后可以将其多播（multi-cast）到有线电视和卫星系统，由若干机顶盒2040（例如，DOCSIS 2.0或3.0）或OTT系统检索该有线电视和卫星系统，并且在流2000处观察该HTML和JavaScript。因此，创建了交互式电视体验，并且任何使用电话的用户都可以改变频道，例如，加载可视小部件，以及覆盖从设备连接平台处的MEVIA服务器生成的内容，以便通过无线大型有线和卫星系统或OTT平台（诸如Sling电视或FUBO电视）或NextGenTV（诸如ATSC 3.0标准广播系统）广播。例如，在2016处的"Mevia服务器”生成具有动画的网页，以及从控制器2014生成的交互。控制器与QR码2010交互，并且在200处看到被投影的图像，在2025处看到多播装置以及在2040处看到单播装置，在2022处从高速缓存服务器检索内容，该内容在2018处渲染和广播具有HTMO和JavaScript的产生的网页。在一些实施例中，在2030处音乐和视频可以由如‘074专利中所提出的高速缓存单元添加或检索，如图14所示，1410和1420处的HLS流是m3u8格式的。

[0169] 可以附加诸如游戏、视频界面等其它控制器，并且由设备连接平台向新颖的广播系统供电。

[0170] 显然，用户可以与设备连接平台中的内容交互，并且该内容可以与音乐频道组合。如图20所示，音乐频道2030可以作为在高速缓存单元中分配和创建的背景音乐运行。例如，用户可以玩游戏（例如，PACMAN）十到十五分钟，并且来自游戏的声音片段可以与存储在音乐频道2030处的歌曲混合，以及集成用户的评论和文本，其可以作为CSS样式"Absolute”的一部分而被覆盖，并且使用出现在游戏的不同位置处的半透明。而且，使用图12（c）的系统，"Mevia App服务器显示器”可以由与视频会议呼叫中的多个人的对话或者来自其它网络应用程序的任何HTML以及可以与一个用户或多个用户交互的任何其它网络应用程序组成。例如，用户可以决定绘制NFT，并且存储绘制从初始x1、y1位置向量到另一x2、y2位置向量的所有路径的过程，以及绘制特定图形的所有步骤。来自HTML或Canvas加HTML的图像和所有流部分被存储为高速缓存系统的一部分，该高速缓存系统从HTML生成准备好要被多播或串流到有线电视运营商的屏幕。因此，设备连接平台内容变成正在广播的音乐的背景，其增强或可以装饰博物馆、咖啡店或任何其它位置的氛围。

[0171] 设备连接平台中的内容路由

[0172] 迄今为止，已经公开了主要涉及QR码映射、WebSockets和WebRTC路由以及Windows设备消息传输的若干路由系统。然而，内容路由将在本章节中更详细地解释，内容路由是设备连接平台的关键特征，并且图21描绘了具有多个UUID和控制器的多个应用程序如何被正确地路由到相应的Mevia应用程序。

[0173] 在设备连接平台中，每一个应用程序具有应用程序标识符（AppID），并且用户在与特定应用程序交互时生成UUID 2180，同时，每一个控制器被映射到存储在内容路由表中的QrUrl。在一些情况下，内容路由器可能需要身份验证2165，并且前一应用程序ID也被存储，或者序列或前一应用程序标识符被存储到PrevAppID字段2168以便能够返回到设备连接平台会话中的前一应用程序。而且，对于当前使用的控制器类型，保持当前控制器值2170。控制器每次初始化接收的第一消息是"initController”2124，initController命令设定特定用户所需的资源和参数。控制器可以生成手势2134或小键盘，或者在按下某些事件时可以生成命令Macro 2140或手势（作为日立Magic Wand设备）。来自"wand设备”的手势是被转换为向上、向下、向向左滑、向右滑命令的3D加速器读取。

[0174] 内容路由器将能够管理从触摸2120到音频和视频2130的数千个命令和事件（参见图21）。内容队列2118接收数据包，这些数据包携带关于目标的应用程序标识符、由设备连接平台为特定用户生成的UUID以及具有消息有效负载2114的message.evt的信息。消息有效负载不被内容队列分析，内容队列可以使用HashMap或NoSQL数据库、链表（Linked List）来实现。负载平衡器2155接收诸如带宽、每应用负载和QoS 2150的值，并且可以被调整以确定何时处理以及如何处理内容队列。一旦内容队列选择了要处理的数据包，则将其传递到应用程序映射器2142，应用程序映射器负责确定特定命令是否必须被转换到JavaScript宏转换器2144，或者是否是需要控制器重新初始化2146的"退出”命令。作为示例，该系统可以包括"长敲击”作为通用"ESC”命令，其意味着退出设备连接平台应用程序。这样，应用程序映射器2142可以被转换以将特定屏幕绘制到画布2140中或者在AppID2处利用UUID2 2105播放特定音频流作为通知蜂鸣声，或者应用程序映射器可以在2100处将evt.mousestartx，y，z 2110发送到具有AppID1的Mevia app UUID1。在另一变型中，应用程序映射器将把jQuery宏发送到UUID4和具有JavaScript "(‘#slider’).next();”的AppID1，其随后被"评估”或者在具有APPID2 WebSocket的UUID3处使用eval命令，或者简单地将具有message.evt=”SwipeRight” 以及 message.evt.KeyB=<ctrl>的命令2108发送到具有UUID3 2112的应用程序AppID3。

[0175] 在该示例中，内容队列系统和控制器可经由QrUrl1访问QrUrl4（QR码的通用资源定位符），并且每一个控制器可以包括如图10所示的任何控制器，在1001处，QrUrl1与游戏相关联，QrUrl2与NFT Lab相关联，QrUrl3是视频会议，而QrUrl4是门控制器界面或应用程序。实质上，内容路由器正在使用SocketIO处理所有消息，并且可以以NodeJS或其它语言实现。例如，内容路由器创建为组以将特定App标识符的所有消息与UUID加入。为了帮助说明，伪代码可以在WebSocket服务器上定义用于连接事件的事件监听器。另外，当建立新连接时，以连接的套接字作为自变量执行回调函数。在回调内部，通过调用具有两个自变量（UUID和AppID）的getContentRouter来获得pathID。随后，套接字将输入数据解析为JSON。它将指示已经接收到命令"X，Y”的消息记录到控制台，包括从解析数据提取的data.evt和data.evt.x、data,evt.y属性。最后，服务器广播具有事件名称"ComandXY”的消息，并且通过pathID识别到房间中的所有套接字的解析数据。

[0176] 在NodeJS中来自SocketIO库的socket.broadcast.to（pathId）可以用于在内容路由器已经生成pathId时向特定应用程序ID发出特定命令。另一方面，内容队列可以被实现为具有Round Robin或加权Round Robin访问技术或使用"Apache ActiveMQ”或其它技术的Hashmap，以处理来自多个源的大量消息。同时，Kubernetes和基于云的方法可用于解决竖直和水平的可扩展性问题。

[0177] 在图22（a）至22（c）上，我们提出了包括nodejs应用程序的数据捕获系统，该nodejs应用程序服务所有命令并将命令发送到MEVIA的某一版本，但是不是2维的，现在它实际上可以是3维的。如图22（a）所示，第一步是处理，设备运动2200示出设备如何能够从加速度计和来自移动设备的其它仪表捕获数据，使得定位和运动能够被捕获并被馈送到深度学习算法中，function handleOrientation(event) {var absolute = event.absolute;var alpha = event.alpha;var beta = event.beta;var gamma = event.gamma; / / var interval = event.interval; / / elapsed_o = elapsed_o + interval; / / if (elapsed_o>= delta_t){(".alpha").text("alpha (z axis, 0 to 360): " +alpha.toFixed(4));(".beta").text("beta (x axis, -180 to 180): " +beta.toFixed(4));(".gamma").text("gamma (y axis, -90 to 90): " +gamma.toFixed(4));(".absolute").text("absolute: " + absolute);elapsed_o = 0; / / }}function handleMotion(event) {var acceleration = event.acceleration;var accelerationIncludingGravity = event.accelerationIncludingGravity;var rotationRate = event.rotationRate;var interval = event.interval;elapsed = elapsed + interval;…

[0178] 随后，将从2205捕获的事件本地存储到移动设备2206中。移动设备2206可以经由JSON命令递送到用于在服务器2208处训练神经网络的服务，在那里它们被首先存储并保存在Clean Data 2210中，同时映射到特定的命令集。这些命令2212可以是上、下、左、右、画一个字符、打一个数字。每一个命令以16ms的采样速率存储两到三秒，代表大约200个样本，其将用于长期短期记忆训练（LSTM）或其它的一个或多个神经网络。

[0179] 一旦加速器数据被保存为JSON对象2214，图22（b）提出了可以使用Keras或其它时间训练序列来完成的标准训练过程，观察到可以使用deta_t和accel_x，y，z值来完成该训练，该accel_x，y，z值可以从图22（a）捕获的所有元素中合并。训练序列可以被分成70%用于训练2216，30%用于测试2217。

[0180] 例如，用于手势的训练算法则如下：N个样本的X_train = [[0…. STEPS], [0…STEPS…]….. ]。目的是"STEPS”的值为200至300个步骤，并且可以被归一化以处理相同的样本量，并且Y_Train具有选定为"上”、"下”、"左”的值，以及如图22（b）所示的利用N个样本训练的其它手势。一旦权重或深度学习机收敛为使用20到25个历元，训练就以94-97%的精度收敛，机器可以被保存以在电话2218本地使用，如图22（c），并且第二神经网络可以被存储在远程NN 2220，在一个训练中，延迟为零，并且远程NN 2220的值包括来自网络信息流的所有延迟。目标是结果2225，其将生成现在作为message.evt命令发送到设备连接平台的适当消息。

[0181] 图23示出了所有手势随后如何能够处理并能够受控地发送命令到设备连接平台应用程序，例如在（a）处手将从2300移动到2312，通过在空中从左到右移动或挥动手的手势，如挥动，在图23（a）处，设备连接平台应用所示的应用程序将是"Mevia App 1”、"MeviaApp 2”和"Mevia App 3”。通过扫描与智能电视2315相关联的QR码，从移动电话的2300手势发出的命令被发送或发出到加载到设备连接平台中的网络应用程序。因此，可以使用如运动2300中所示的手势来控制设备连接平台应用程序。在该示例中，发出向右滑，并且菜单从Mevia App 3切换到游戏应用程序2312。作为设备连接平台内容路由器的一部分，命令可以从message.evt = "Swipe Right”转换到(‘slider’).next()。一旦选择了菜单选项，则通过另一手势来完成选择以加载游戏应用程序。敲击手势可以被发布到智能电视2310，随后所选择的应用程序2305被加载，并且规律使用的游戏控制器2325被加载以与设备连接平台应用程序就那些交互，由于在这种情况下是游戏应用程序。

[0182] 图24示出了随后如何通过添加3dmevia.js来改变MEVIA应用程序，以处理Z轴定位变化，例如WebGL或ARKit应用程序除了X轴和Y轴运动之外还可以转换深度运动。作为设备连接平台应用程序，message.evt还可以包含具有message.x、message.y和message.z位置的结构，并且处理MeviaAccelectrometer实例2430。加速器和陀螺仪实例在JavaScript上作为对象window.ondevicemotion、window.ondeviceorientation或window.onmozorientation的一部分是常见的，作为由iPhone、Firefox和Chrome浏览器支持。此外，3Dmevia.js 2405库包含3D定位和也可以与摄像头LIDAR传感器组合使用以在AR / VR头显（增强现实 / 虚拟现实）中定位的加速度计值、mevia.js 2422和由mevia.css 2424加载的其它资源样式表资源。随后，处理这种类型的应用程序以与3D画布2426交互，2410也是SocketIO消息的接收者，该SocketIO消息包含所有控制器命令，此外，网络应用程序中的HREF（HTML引用）和其它网络引用被转换以在设备连接平台2428上运行。WebGL和3D渲染的使用对于AR / VR情景是有用的，其中，移动设备或终端是AR / VR头显（例如，Vision OS、MetaQuest、Vivero等），或者是使用电话的摄像头（诸如iOS或Android）的具有增强现实（AR）的移动电话。

[0183] 图25示出了在2500处使用手势的认证的所有初始化开始时，通过加载神经网络（例如，LSTM）以训练更多信息，在2D或3D系统中训练手势所必需的步骤。在2510处，训练数据与2505交互，其中是用来自装置的加速度计的信息对那些手势的训练，被称为"Accel数据”，一旦在若干次训练迭代之后完成训练，就可以利用训练数据的子集或"测试数据”来测试神经网络，如在2515处所示。另外，一旦在2520处以期望的精度完成验证和训练相关，则保存所有神经网络权重，并且在2214处，当捕获如图22所示的向右滑或向左滑手势的样本序列时，得到的神经网络可以用作检测"向右滑”、"敲击”的一部分，或者甚至使用如图23所示的手势信息来训练"游戏控制器”。2510处的元素使用加载有多个"向右滑”训练值或"向右或向左滑”的训练信息的装置加速度计数据（Accel Data）。可以包括时间、x、y、z、坐标、或者甚至accel_x、accel_y和accel_z值的值序列被用于训练。一旦被训练，图23所示的检测功能2300，2312中所示的"向右或向左滑”检测功能可用于在屏幕上移动位置或光标的发布message.evt向Mevia App 2310发出命令。保存的神经网络（NN）或LSTM一旦被训练就可以通过将NN保存下载到装置中或通过将NN作为Mevia.js的一部分的处理、通过接收SocketIO消息或者通过经由来自移动设备或AR耳机的HTTP GET或POST请求来识别如图22中所描绘的手势。

[0184] 图26是根据本公开原理的一般示出用于使得用户能够使用另一计算设备上的浏览器界面来控制计算设备上的内容并与之交互的方法2600的流程图。在2602处，方法2600针对在第一计算设备上执行的第一浏览器生成第一消息，其中，第一消息包括指令，该指令响应于由第一浏览器执行而使得物理用户输入设备的界面的表示被第一浏览器显示。

[0185] 在2604处，将第一消息提供给第一浏览器（例如，智能TV上的网页）。

[0186] 在2606处，方法2600针对在第二计算设备上执行的第二浏览器生成第二消息，其中，第二消息包括指令，在该指令由第二浏览器执行时，该指令响应于来自第一浏览器显示的物理用户输入设备的界面的表示的输入而使得能够与第二浏览器提供的内容进行用户交互。物理用户输入设备被配置为与第二浏览器提供的内容进行交互。（例如，具有在移动设备处显示的控制器的网页）

[0187] 在2608处，方法2600将第二消息提供给第二浏览器，并且创建从第一浏览器到第二浏览器的交互（例如，从控制器到第一浏览器的网页和响应的命令）。

[0188] 缩写IP：因特网协议AAC：高级音频编码CDN：内容分发网络UUID：唯一通用标识符BLE：低功耗蓝牙CABSAT：有线及卫星系统OTT：过顶平台SocketIO：用于管理WebSockets的JavaScript库MP3：MPEG-2音频层IIIOGG：用于音频的Theora编解码器OGV：用于视频的Theora编解码器MPEG：运动图像专家组FFMPEG：快速前向MPEGWebRTC：网络实时通信HLS：HTTP实时流传输TS：传输流EC2：弹性计算类2VP8：由Google开发的使用LibVPX的开放式视频编码SSH：安全接壳RTMP：实时消息传输协议WSS：安全WebSocket协议HTTP：超文本传输协议HTTP GET：当数据作为URL的一部分提交时，用于提交数据的HTTP方法HTTP POST：用于在数据是有效负载的一部分而不是URL时提交数据的HTTP方法HTML：超文本标记语言HLS：HTTP实时流传输LSTM：长短期记忆网络NAT：网络地址转换NFT：非同质化令牌NN：神经网络RTP：实时协议SIP：会话发起协议URI：通用资源标识符URL：通用资源定位器WLAN：无线局域网或WiFiOTT：过顶流传输平台LAN：局域网PAN：个人区域网WAN：无线区域网

[0189] 应该注意，本发明的过程和系统的各种单独的特征可以仅在本文中的一个示例性实施例中描述。在本文中关于单个示例性实施例的用于描述的特定选择不应被认为是限制，即特定特征仅适用于描述该特征的实施例。本文所述的所有特征同样适用于、添加到或与本文所述的任何或所有其它示例性实施例可互换，并且可以任何组合、分组或布置。特别地，在本文中使用单个附图标记来说明、定义或描述特定特征并不意味着该特征不能与另一附图或描述中的另一特征相关联或等同。另外，在附图中使用两个或更多个附图标记的情况下，这不应被解释为仅限于那些实施例或特征，它们同样适用于相似的特征，或者不使用附图标记或省略另一附图标记。

[0190] 前面的描述和附图示出了系统、设备和方法的原理、示例性实施例和操作模式。然而，系统、设备和方法不应被解释为限于上述特定实施例。本领域技术人员将理解上述实施例的额外变化，并且上述实施例应该被认为是说明性的而非限制性的。因此，应该理解，本领域技术人员可以对这些实施例进行改变，而不脱离由所附权利要求限定的系统、设备和方法的范围。< / script> < / canvas>

Claims

1. A method comprising: A first message is generated for a first browser running on a first computing device, the first message including an instruction that, in response to being executed by the first browser, causes a representation of the interface of the physical user input device to be displayed by the first browser; Provide the first message to the first browser; A second message is generated for a second browser running on a second computing device. The second message includes an instruction that, when executed by the second browser, responds to input represented by the interface of the physical user input device displayed by the first browser, enabling user interaction with content provided by the second browser, wherein the physical user input device is configured to interact with the content provided by the second browser. as well as The second message is provided to the second browser.

2. The method according to claim 1, further comprising: In response to user interaction with the representation of the interface of the physical user input device displayed by the first browser, a third message including touch gesture data is received from the first browser; The touch gesture data is then converted into an update of the content rendered or executed by the second browser; The content rendered or executed by the second browser is updated based on the touch gesture data; as well as A fourth message is provided to the second browser, the fourth message including the update to the content provided by the second browser.

3. The method according to claim 1, further comprising: Receive an identification message, the identification message including application identification data associated with the network application; Based on the application identification data, determine the identification information of the physical user input device used to interact with the network application; as well as The instruction that generates the second message, when executed, causes the interface of the physical user input device to be displayed on the first browser.

4. The method according to claim 3, wherein, The identification information of the physical user input device indicates the brand, type, and model of the physical user input device.

5. The method according to claim 3, wherein, The identification message is associated with a quick response code.

6. The method according to claim 3, wherein, The application identification data includes information identifying a target server, which is used to receive any user interaction with the interface of the physical user input device.

7. The method according to claim 1, further comprising: Receive user selections from web applications; as well as A third message is generated for the first browser executing on the first computing device. The third message includes an instruction that, when executed by the first browser, causes an alternative representation of the interface of the physical user input device associated with the web application to be displayed by the first browser.

8. The method according to claim 1, wherein, The representation of the interface of the physical user input device includes the entire layout of the physical user input device.

9. The method according to claim 1, wherein, The instruction executed by the first browser to cause the representation of the interface of the physical user input device to be displayed by the first browser includes at least one JavaScript library.

10. The method according to claim 1, wherein, The instructions that enable user interaction with content provided by the second browser when executed by the second browser include at least one JavaScript file.

11. A system comprising: At least one processor circuit; as well as At least one memory storing instructions executable by the at least one processor circuit, the instructions being configured to perform an operation including: A first message is generated for a first browser running on a first computing device, the first message including an instruction that, in response to being executed by the first browser, causes a representation of the interface of the physical user input device to be displayed by the first browser; Provide the first message to the first browser; A second message is generated for a second browser executing on a second computing device. The second message includes instructions that, when executed by the second browser, respond to input from the representation of the interface of the physical user input device displayed by the first browser, enabling user interaction with content provided by the second browser, wherein the physical user input device is configured to interact with the content provided by the second browser; and The second message is provided to the second browser.

12. The system according to claim 11, wherein, The instructions are also configured to perform the following operations: In response to user interaction with the representation of the interface of the physical user input device displayed by the first browser, a third message including touch gesture data is received from the first browser; The touch gesture data is then converted into an update of the content rendered or executed by the second browser; Update the content rendered or executed by the second browser based on the touch gesture data; and A fourth message is provided to the second browser, the fourth message including the update to the content provided by the second browser.

13. The system according to claim 11, wherein, The instructions are also configured to perform the following operations: Receive an identification message, the identification message including application identification data associated with the network application; Based on the application identification data, determine the identification information of the physical user input device used to interact with the network application; and The instruction that generates the second message, when executed, causes the interface of the physical user input device to be displayed on the first browser.

14. The system according to claim 13, wherein, The identification information of the physical user input device indicates the brand, type, and model of the physical user input device.

15. The system according to claim 13, wherein, The identification message is associated with a quick response code.

16. The system according to claim 13, wherein, The application identification data includes information identifying a target server, which is used to receive any user interaction with the interface of the physical user input device.

17. The system according to claim 11, wherein, The instructions are also configured to perform the following operations: Receive user selections in a web application; and A third message is generated for the first browser executing on the first computing device. The third message includes an instruction that, when executed by the first browser, causes an alternative representation of the interface of the physical user input device associated with the web application to be displayed by the first browser.

18. The system according to claim 11, wherein, The representation of the interface of the physical user input device includes the entire layout of the physical user input device.

19. The system according to claim 11, wherein, The instruction executed by the first browser to cause the representation of the interface of the physical user input device to be displayed by the first browser includes at least one JavaScript library.

20. A computer-readable storage medium having program instructions recorded thereon, wherein when the program instructions are executed by at least one processing circuitry of a computing device, the program instructions perform a method, the method comprising: A first message is generated for a first browser running on a first computing device, the first message including an instruction that, in response to being executed by the first browser, causes a representation of the interface of the physical user input device to be displayed by the first browser; Provide the first message to the first browser; A second message is generated for a second browser running on a second computing device. The second message includes an instruction that, when executed by the second browser, responds to input represented by the interface of the physical user input device displayed by the first browser, enabling user interaction with content provided by the second browser, wherein the physical user input device is configured to interact with the content provided by the second browser. as well as The second message is provided to the second browser.

Images