Cross-network interaction mediation device based on wireless communication, control system, interaction system

Abstract

The application discloses a cross-network interaction intermediary device based on wireless communication, a control device, an interaction system, and a retrieval method and device. The cross-network interaction intermediary device comprises at least one HDMI interface, one or more processors, at least one memory, and at least one wireless communication module. One or more modules are stored in the memory and are configured to be executed by the one or more processors. The one or more modules comprise a module for converting a real-time video stream into an uplink video stream and a module for pushing the uplink video stream to a server in a push stream mode through the wireless communication module. The cross-network interaction intermediary device can realize the isolation of an internal network and a periphery. The video stream acquisition based on the cross-network interaction intermediary device can ensure the high smoothness and achieve the purpose of displaying multiple controlled terminals in one control terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cross-network interaction, in particular to a cross-network interaction mediation device based on wireless communication, a control device, an interaction system, and a retrieval method and device. BACKGROUND

[0002] Remote assistance is a specific implementation technology of cross-network interaction. The remote assistance technology is generally based on two devices in the same local network or simultaneously accessing the public network. Both devices need to install remote assistance program software in advance and need to call and respond to form a link. This technology cannot realize cross-network remote assistance between a specific internal network and an external network. At the same time, since the remote assistance technology sends the desktop video stream of one device to another device through the TCP / IP protocol of the Ethernet, the other device sends its mouse events and keyboard events to the device through the Ethernet TCP / IP.

[0003] When implementing remote assistance by simultaneously accessing the public network, the current public network access mode is mostly asymmetric, that is, the uplink rate is much lower than the downlink rate. Since remote assistance needs to send the desktop video stream uplink to another device, if you want to achieve lossless and smooth remote assistance, the user needs to configure gigabit bandwidth to smoothly perform public network remote assistance.

[0004] For some confidential units, the network is strictly required to be a private local area network, and access to the public network is not allowed and physical isolation is required. It is particularly difficult for such units to achieve automatic collaboration with other units, especially how to achieve the purpose of synchronously monitoring multiple controlled devices in one interface.

[0005] At present, whether it is remote assistance based on simultaneous access to the public network or remote assistance for cross-network interaction, they all face the technical problem of requiring a large bandwidth support for uplink bandwidth. Therefore, how to realize cross-network interaction between the internal network and the periphery and solve the bandwidth adaptation problem is a technical problem that needs to be solved by the present application. Especially in terms of flexibility, how to break away from wired networks and realize cross-network interaction of wireless networks is a technical problem that needs to be further faced. SUMMARY

[0006] The purpose of the present application is to propose a cross-network interaction mediation device based on wireless communication, a control device, an interaction system, and a retrieval method and device. Through the cross-network interaction mediation device, the internal network and the periphery are isolated, the video stream acquisition based on network isolation by the cross-network interaction mediation device is realized, and the purpose of displaying multiple controlled ends in one control end is achieved while ensuring high smoothness.

[0007] To solve the above technical problems, the present application adopts the following solutions:

[0008] In one aspect, a wireless communication based cross-network interaction intermediary device, comprising:

[0009] at least one HDMI interface for connecting with a parallel HDMI interface of a host of the controlled terminal to obtain a real-time video stream, the real-time video stream being an active video signal of the parallel HDMI interface of the controlled terminal, and the parallel HDMI interface being in parallel relationship with an executing HDMI interface of the controlled terminal for transmitting the active video signal to a display;

[0010] one or more processors,

[0011] at least one memory,

[0012] at least one wireless communication module,

[0013] one or more modules stored in the memory and configured to be executed by the one or more processors,

[0014] the one or more modules comprising:

[0015] a module for converting the real-time video stream into an uplink video stream;

[0016] a module for pushing the uplink video stream to a server in a push mode through the wireless communication module.

[0017] Preferably, the module for converting the real-time video stream into the uplink video stream comprises:

[0018] a module for obtaining the real-time video stream;

[0019] a module for processing the real-time video stream by a video encoding algorithm of H series or MPEG series to obtain a binary encoded uplink video stream, the uplink video stream further comprising an ID number of the cross-network interaction intermediary device.

[0020] Preferably, the module for converting the real-time video stream into the uplink video stream comprises:

[0021] a module for obtaining a current network strength value of the wireless communication module;

[0022] a module for obtaining the real-time video stream;

[0023] a module for matching a corresponding current uplink network speed S 上 according to the current network strength value;

[0024] a module for matching a byte number conversion parameter adaptive to the current uplink network speed S 上 according to a single frame byte number obtained according to a resolution and a bit depth of a frame picture of the active video signal;

[0025] A module that converts real-time video streams into uplink video streams based on byte conversion parameters. The uplink video stream also includes the ID number of the cross-network interactive intermediary device.

[0026] Preferably, the number of bytes per frame obtained based on the resolution and bit depth of the active video signal's frame images is used as a benchmark to match the current uplink network speed S. 上 The modules for byte number conversion parameters include:

[0027] A module for extracting the frame rate, resolution of a single frame, and bit depth of a real-time video stream.

[0028] A module that adaptively solves for the byte number conversion parameters based on the byte number conversion parameter formula and constraints;

[0029] The formula for the byte count conversion parameter is: S 上 =((L*K*(W / 8)) / 1024 / 1024)*m%*n%*i,

[0030] S 上 This represents the current uplink network speed of the wireless communication module, in MB / s.

[0031] L*K represents the resolution of one frame in a real-time video stream.

[0032] W represents the bit depth of one frame of a real-time video stream, measured in bytes.

[0033] i represents the optional frame rate of the uplink video stream, in frames per second (fps).

[0034] m% is the optional resolution compression ratio for frame images in the uplink video stream.

[0035] n% is the optional precision compression ratio for frame images of the uplink video stream.

[0036] Byte count conversion parameters include: i, m%, n%.

[0037] The constraints include:

[0038] Constraint 1: The frame rate of the real-time video stream ≥ i ≥ 1.

[0039] Constraint 2: 100% ≥ m% ≥ 30%,

[0040] Constraint 3: 100% ≥ n% ≥ 30%.

[0041] Preferably, the number of bytes per frame obtained based on the resolution and bit depth of the active video signal's frame images is used as a benchmark to match the current uplink network speed S. 上 The modules for byte number conversion parameters include:

[0042] a module for obtaining a conversion parameter matching table,

[0043] according to the current uplink network speed S 上 a module for searching the byte number conversion parameter corresponding to the uplink network speed corresponding to the calibrated network intensity value of each level in the conversion parameter matching table and outputting the byte number conversion parameter,

[0044] The conversion parameter matching table comprises:

[0045] the calibrated network intensity value of each level, the uplink network speed corresponding to the network intensity value of each level;

[0046] the optional frame rate of the uplink video stream, the optional resolution compression ratio of the frame picture of the uplink video stream, and the optional precision compression ratio of the frame picture of the uplink video stream, which are adapted to the uplink network speed corresponding to the network intensity value of each level, are calibrated;

[0047] The optional resolution compression ratio and the optional precision compression ratio are both calculated in proportion to the single-frame byte number obtained by calibrating the resolution and the bit depth of the frame picture of the active video signal.

[0048] Preferably,

[0049] When the wireless communication module is a 5G wireless communication module,

[0050] When the resolution of the frame picture of the calibrated active video signal is 1024*768 and the bit depth is 8,

[0051] The network intensity value is divided into 9 levels, and the uplink network speeds corresponding to the 9-level network intensity values are respectively: “0.3-0.7”, “0.7-1.1”, “1.1-1.5”, “1.5-2.5”, “2.5-3.5”, “3.5-4.5”, “4.5-6.5”, “6.5-8.5”, and “8.5 or above”, and the unit of the uplink network speed is MB / s;

[0052] The optional frame rate of the uplink video stream, the optional resolution compression ratio of the frame picture of the uplink video stream, and the optional precision compression ratio of the frame picture of the uplink video stream, which are adapted to the uplink network speed corresponding to the network intensity value of each level, are respectively: “10, 30%, 30%”, “10, 30%, 40%”, “10, 40%, 40%”, “20, 40%, 40%”, “20, 40%, 50%”, “20, 50%, 50%”, “30, 50%, 50%”, “30, 50%, 60%”, and “30, 60%, 60%”;

[0053] The unit of the optional frame rate of the uplink video stream is frame / s.

[0054] Preferably, the device further comprises:

[0055] at least one PS / 2 interface for connecting with a parallel PS / 2 interface of a host of the controlled terminal,

[0056] The one or more modules further comprise:

[0057] a module for receiving a mouse event or / and a keyboard event of the controlled terminal through the wireless communication module,

[0058] a module for compiling the mouse event or / and the keyboard event into a format recognizable by the PS / 2 interface.

[0059] In another aspect, a control system comprises: a server and a control terminal,

[0060] the server is configured to receive and store an uplink video stream uploaded by a cross-network interaction intermediary device based on wireless communication;

[0061] the control terminal is configured to call and display the uplink video stream in real time;

[0062] the control terminal comprises a display, one or more processors, at least one memory, and one or more modules stored in the memory and configured to be executed by the one or more processors,

[0063] The one or more modules comprise:

[0064] a module for receiving the uplink video stream uploaded in response to the server;

[0065] a module for sending the uplink video stream to the display for display;

[0066] The module for sending the uplink video stream to the display for display comprises:

[0067] a module for selecting the uplink video stream corresponding to the ID number in the first mode to a full-screen playing display window in the display for display;

[0068] a module for simultaneously displaying the uplink video streams corresponding to the multiple ID numbers in the second mode to multiple ID playing display windows in the display;

[0069] a module for selecting a cutting range of the uplink video streams corresponding to the multiple ID numbers in the third mode and sending the cutting range to a playing display window for display.

[0070] Preferably, the one or more modules further comprise at least one of the following modules:

[0071] The module records the mouse event of the mouse of the control terminal in the playing display window of the uplink video stream corresponding to the ID number and forwards the mouse event to the cross-network interaction intermediary device based on wireless communication corresponding to the selected ID number.

[0072] The module records the keyboard event of the control terminal and forwards the keyboard event to the cross-network interaction intermediary device based on wireless communication corresponding to the selected ID number.

[0073] Preferably, the mouse event includes at least one of movement and selection.

[0074] The mouse event includes at least one of movement, selection, instruction and data input.

[0075] In another aspect, an interaction system includes

[0076] At least one controlled terminal, the cross-network interaction intermediary device based on wireless communication configured for the controlled terminal in one-to-one manner, and the control system,

[0077] The HDMI interface and the PS / 2 interface of the interaction intermediary device are connected with the parallel HDMI interface and the parallel PS / 2 interface of the controlled terminal respectively, the interaction intermediary device is linked with the service end of the control system through the wireless communication module, and the service end of the control system is linked with the control terminal of the control system.

[0078] The controlled terminal is divided into an inner network, and the interaction intermediary device, the service end of the control system and the control terminal of the control system are divided into an outer network.

[0079] The controlled terminal, the interaction intermediary device, the service end of the control system and the control terminal of the control system form a video data uplink and a control instruction downlink.

[0080] The application has the following beneficial effects:

[0081] (1) The interaction intermediary device based on wireless network module has the low-cost advantages of low operation cost and low equipment investment cost.

[0082] (2) Regarding smoothness, smoothness is related to network data transmission latency and insufficient screen refresh rate. In this invention, the bandwidth required for the uplink video stream reaches 39MB. Taking a high-speed wireless network module like 5G as an example, its uplink bandwidth is at most 9MB / s. If a 1-second video stream is transmitted to a remote location, it will take at least 4.3 seconds. However, if we adjust the frame rate from 50 frames / s to 30 frames / s within 1 second, although 20 frames are lost within 1 second, there is no noticeable jump in the image. This invention adapts conversion parameters to the current network strength based on frame rate, resolution compression, and precision compression, and performs conversion based on these conversion parameters to obtain a video stream without network latency. This invention has smoothness.

[0083] (3) The present invention allows one control terminal to access multiple controlled terminals simultaneously, realizing one-to-many remote assistance interaction. Compared with the traditional point-to-point remote interaction system, the present invention uses the server as a medium, that is, one server faces multiple cross-network interaction intermediary devices based on wireless communication, realizing one-to-many interaction.

[0084] (4) The interactive system provided by this invention utilizes a cross-network interactive intermediary device for passive data acquisition. Passive acquisition means that after the port is connected, data can be obtained through port detection without a network connection request. This passive acquisition method of the cross-network interactive intermediary device is used to monitor the interface information of intranet devices, enabling the interface information of intranet devices to be transferred to the external network. This achieves interface interaction acquisition even when the internal and external networks are physically isolated. Simultaneously, using the server as a springboard, multiple cross-network interactive intermediary devices send data to the server, and the server responds to the control terminal's request, redirecting the interface information to the control terminal for display. This achieves a one-to-many synchronous monitoring effect, meaning one control terminal can simultaneously observe multiple intranet devices across networks. Furthermore, the downlink is used to send events for device operation. This route between the cross-network interactive intermediary device and the intranet device is an instruction set relationship; that is, the cross-network interactive intermediary device sends mouse and keyboard instruction sets to the intranet device. The two do not interact via network information. In this case, the cross-network interactive intermediary device can be understood as a mouse or keyboard simulator, sending analog signals. The cross-network interactive intermediary device, server, and control terminal designed in this invention are integrated as an external network, achieving complete physical isolation from internal network devices, and adapting to the network conditions of various classified units.

[0085] (5) The server provided by the present invention can simultaneously respond to the storage of uploaded data from multiple cross-network interactive intermediary devices, and simultaneously respond to simultaneous or single calls from the control terminal in real time. It can stitch and process the frame images uploaded by multiple cross-network interactive intermediary devices and synchronize them to the control terminal as a new video stream.

[0086] (6) The cross-network interaction intermediary device provided by the application directly performs proportional compression and frame number extraction processing on the frame pictures in the original stream media monitored by the HDMI port, so that the fidelity processing based on the original frame pictures is guaranteed in the case of adapting to the network, that is, the number of transformations is reduced as much as possible, and in the application, there is only one transformation processing for the purpose of network speed adaptation, compared with multiple conversions, the fidelity effect is very good, and at the same time, since there is only one transformation processing in the cross-network interaction intermediary device, the responsiveness of the entire system is greatly improved. That is, the application reduces the network delay and system response delay as a whole, and achieves the transmission effect of high-quality pictures. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 is a schematic diagram of an interactive system.

[0088] Figure 2 is a flowchart of a video stream using encoding processing.

[0089] Figure 3 is a schematic diagram of adaptively matching byte number conversion parameters.

[0090] Figure 4 is a schematic diagram of matching byte number conversion parameters by table lookup method.

[0091] Figure 5 is a schematic diagram of network adaptation processing outputting uplink video stream according to network strength value.

[0092] Figure 6 is a parameter table of real-time video stream.

[0093] Figure 7 is a conversion parameter matching table.

[0094] Figure 8 is a schematic diagram of an interface of a control terminal.

[0095] Figure 9 is another schematic diagram of an interface of a control terminal.

[0096] Figure 10 is another schematic diagram of an interface of a control terminal. DETAILED DESCRIPTION

[0097] The application will be further described in detail below in combination with embodiments and drawings, but the embodiments of the application are not limited thereto.

[0098] Embodiment 1

[0099] As shown in Figure 1 , 2 :

[0100] The cross-network interaction intermediary device based on wireless communication comprises:

[0101] at least one HDMI interface, configured to obtain a real-time video stream by connecting with a parallel HDMI interface of a host of the controlled terminal, the real-time video stream being an active video signal of the parallel HDMI interface of the controlled terminal, and the parallel HDMI interface being in parallel with an executing HDMI interface of the controlled terminal for transmitting the active video signal to a display;

[0102] one or more processors,

[0103] at least one memory,

[0104] at least one wireless communication module,

[0105] one or more modules are stored in the memory and configured to be executed by the one or more processors,

[0106] the one or more modules comprise:

[0107] a module for converting the real-time video stream into an uplink video stream;

[0108] a module for pushing the uplink video stream to a server in a streaming mode through the wireless communication module.

[0109] the module for converting the real-time video stream into the uplink video stream comprises:

[0110] a module for obtaining the real-time video stream;

[0111] a module for processing the real-time video stream by using a video encoding algorithm of an H series or an MPEG series to obtain a binary coded uplink video stream, and the uplink video stream further comprises an ID number of the cross-network interaction intermediary device.

[0112] Embodiment 2

[0113] as shown in Figure 1 :

[0114] the cross-network interaction intermediary device based on wireless communication comprises:

[0115] at least one HDMI interface, configured to obtain a real-time video stream by connecting with a parallel HDMI interface of a host of the controlled terminal, the real-time video stream being an active video signal of the parallel HDMI interface of the controlled terminal, and the parallel HDMI interface being in parallel with an executing HDMI interface of the controlled terminal for transmitting the active video signal to a display;

[0116] one or more processors,

[0117] at least one memory,

[0118] at least one wireless communication module,

[0119] one or more modules are stored in the memory and configured to be executed by the one or more processors described above,

[0120] one or more modules include:

[0121] a module for converting the real-time video stream into an uplink video stream;

[0122] a module for pushing the uplink video stream to the server in a streaming mode through the wireless communication module.

[0123] the module for converting the real-time video stream into an uplink video stream includes:

[0124] a module for obtaining a current network strength value of the wireless communication module;

[0125] a module for obtaining the real-time video stream;

[0126] a module for matching a corresponding current uplink network speed S 上 according to the current network strength value;

[0127] a module for matching byte number conversion parameters suitable for the current uplink network speed S 上 according to the resolution and bit depth of the frame picture of the active video signal to obtain the single-frame byte number;

[0128] a module for converting the real-time video stream into an uplink video stream according to the byte number conversion parameters, and the uplink video stream further includes the ID number of the cross-network interaction intermediary device.

[0129] Specifically, the wireless communication module can be a 4G or 5G wireless communication module. Since the 4G and 5G uplink network speed of each operator is as follows: China Mobile 4G uplink 0.72 MB / s, China Telecom 4G uplink 1.62 MB / s, China Unicom 4G uplink 0.78 MB / s, China Mobile 5G uplink 8.83 MB / s, China Telecom 5G uplink 8.96 MB / s, and China Unicom 5G uplink 8.51 MB / s. Since the uplink speed of 5G is larger, 5G will be taken as an example in the subsequent description.

[0130] As Figure 1As shown, the host of the controlled terminal has an executing HDMI interface connected with the display of the controlled terminal, and it also has a parallel HDMI interface, generally, the parallel HDMI interface and the executing HDMI interface are in the normal open state, that is, the host is powered on to generate an active video signal. Similarly, the host of the controlled terminal has an executing PS / 2 interface and a parallel PS / 2 interface; the HDMI interface has a one-way transmission, that is, it cannot be written, the PS / 2 interface only has a one-way write feature, and these interfaces are all analog signal interfaces, so they cannot be monitored; the application utilizes the above-mentioned features of the two interfaces, adopts a cross-network interaction mediation device to monitor the ports, and the other side is directly linked with the periphery, so as to transfer the picture of the internal network to the external network across the network, and convert the control instruction of the external network into an analog signal to write into the controlled terminal of the internal network across the network, thereby realizing cross-network interaction. Therefore, the cross-network interaction mediation device comprises at least one HDMI interface, which is used to connect with the parallel HDMI interface of the host of the controlled terminal to obtain a real-time video stream, the real-time video stream is an active video signal of the parallel HDMI interface of the controlled terminal, and the parallel HDMI interface and the executing HDMI interface of the controlled terminal have a parallel relationship in transmitting the active video signal to the display.

[0131] Meanwhile, as described above, for a real-time video stream of a controlled terminal, the frame rate is generally 50 frames / s, the resolution of a single frame picture is 1024*768, and the bit depth is 8 bytes, so the byte number of a single frame picture is 0.78 MB. If there is no lossy transmission and no delay transmission, the required network speed is 39 MB / s. For a 5G wireless network module, it cannot perform lossless and delay-free transmission. Meanwhile, although the wireless network module has good maneuverability, it is affected by environmental factors, climate, surrounding interference, and other factors, so the network strength will change. When the network strength decreases, the uplink and downlink network speeds decrease. Therefore, fixedly setting the size of the uplink video stream still has the problem of transmission delay caused by network fluctuation. Therefore, in order to solve the problems of not meeting the total size of the real-time video stream and the change of the uplink speed caused by the network strength fluctuation, the application proposes a method of adjusting the byte number conversion parameter based on the network strength, that is, after the byte number conversion parameter is adapted according to the current network strength value, the conversion process of a single frame picture and a multi-frame remodeled video stream is performed according to the byte number conversion parameter, and finally the byte number of the obtained uplink video stream is adapted to the current uplink network speed, so as to avoid the problem of reduced smoothness caused by network delay. Therefore, the application is configured with one or more modules stored in the memory and configured to be executed by the above-mentioned one or more processors,

[0132] The one or more modules comprise: a module for obtaining the current network strength value of the wireless communication module; a module for obtaining a real-time video stream;

[0133] Match the current uplink network speed S based on the current network strength value. 上 The module; based on the resolution and bit depth of the frame image of the active video signal, the number of bytes per frame is matched to adapt to the current uplink network speed S. 上 The module that converts the byte count to a parameter; the module that converts the real-time video stream to an uplink video stream based on the byte count to parameter; and the module that forwards the uplink video stream to the wireless communication module.

[0134] In order to obtain byte count conversion parameters that are adapted to the current uplink network speed, this invention proposes to either configure an adaptive algorithm to obtain the parameters adaptively or configure a conversion parameter matching table to obtain the parameters by looking up the table.

[0135] The adaptive algorithm adaptively obtains the byte number conversion parameters by the following modules:

[0136] like Figure 3 As shown, the number of bytes per frame obtained based on the resolution and bit depth of the active video signal frame image is used as a benchmark to match the current uplink network speed S. 上 The modules for byte number conversion parameters include:

[0137] A module for extracting the frame rate, resolution of a single frame, and bit depth of a real-time video stream.

[0138] A module that adaptively solves for the byte number conversion parameters based on the byte number conversion parameter formula and constraints;

[0139] The formula for the byte count conversion parameter is: S 上 =((L*K*(W / 8)) / 1024 / 1024)*m%*n%*i,

[0140] S 上 This represents the current uplink network speed of the wireless communication module, in MB / s.

[0141] L*K represents the resolution of one frame in a real-time video stream.

[0142] W represents the bit depth of one frame of a real-time video stream, measured in bytes.

[0143] i represents the optional frame rate of the uplink video stream, in frames per second (fps).

[0144] m% is the optional resolution compression ratio for frame images in the uplink video stream.

[0145] n% is the optional precision compression ratio for frame images of the uplink video stream.

[0146] Byte count conversion parameters include: i, m%, n%.

[0147] The constraints include:

[0148] Constraint 1: Frame rate of real-time video stream ≥ i ≥ 1,

[0149] Constraint 2: 100% ≥ m% ≥ 30%,

[0150] Constraint 3: 100% ≥ n% ≥ 30%.

[0151] In the above, S 上 , L*K, W are known values obtained by monitoring.

[0152] (L*K*(W / 8)) / 1024 / 1024) represents the number of bytes of a single frame obtained according to the resolution and bit depth of the frame picture of the active video signal.

[0153] m% and n% can be selected alternatively, but in the present application, since the resolution of the input active video signal is large, they need to be used simultaneously, otherwise the results solved by i, m% are small or cannot be solved, and the practicability is poor. The accuracy refers to the picture quality, and the higher the accuracy, the clearer the picture quality.

[0154] Wherein, the process of obtaining the byte number conversion parameter according to the configuration of the conversion parameter matching table is completed by the following modules:

[0155] As shown in Figure 4 , preferably, the module for matching the byte number conversion parameter according to the resolution and bit depth of the frame picture of the active video signal to adapt to the current uplink network speed S 上 includes:

[0156] a module for obtaining the conversion parameter matching table,

[0157] a module for searching the byte number conversion parameter corresponding to the uplink network speed corresponding to each level of network intensity value calibrated in the conversion parameter matching table according to the current uplink network speed S 上 and outputting the byte number conversion parameter,

[0158] the conversion parameter matching table includes:

[0159] each level of network intensity value calibrated, and the uplink network speed corresponding to each level of network intensity value;

[0160] the optional frame rate of the uplink video stream, the optional resolution compression ratio of the frame picture of the uplink video stream, and the optional accuracy compression ratio of the frame picture of the uplink video stream, which are adapted to the uplink network speed corresponding to each level of network intensity value calibrated,

[0161] The optional resolution compression ratio and the optional precision compression ratio are calculated based on the number of bytes of a single frame of the calibrated active video signal.

[0162] Preferably,

[0163] When the wireless communication module is a 5G wireless communication module,

[0164] When the resolution of the frame of the calibrated active video signal is 1024*768 and the bit depth is 8,

[0165] The network strength value is divided into 9 levels, and the uplink network speed corresponding to the 9 levels of network strength values is respectively: "0.3-0.7", "0.7-1.1", "1.1-1.5", "1.5-2.5", "2.5-3.5", "3.5-4.5", "4.5-6.5", "6.5-8.5", "8.5 or more", and the unit of the uplink network speed is MB / s.

[0166] The optional frame rate of the uplink video stream, the optional resolution compression ratio of the frame of the uplink video stream, and the optional precision compression ratio of the frame of the uplink video stream that are adapted to the uplink network speed corresponding to each level of network strength value are respectively: "10, 30%, 30%", "10, 30%, 40%", "10, 40%, 40%", "20, 40%, 40%", "20, 40%, 50%", "20, 50%, 50%", "30, 50%, 50%", "30, 50%, 60%", and "30, 60%, 60%".

[0167] The unit of the optional frame rate of the uplink video stream is frame / s.

[0168] Preferably, the control system further comprises:

[0169] The at least one PS / 2 interface is used to connect with the parallel PS / 2 interface of the host of the controlled terminal.

[0170] The one or more modules further comprise:

[0171] The module receives the mouse event or / and the keyboard event of the downlink remote control of the control terminal through the wireless communication module.

[0172] The module compiles the mouse event or / and the keyboard event into a format recognizable by the PS / 2 interface.

[0173] Embodiment 3

[0174] As shown in Figure 1 a control system comprising a server and a control terminal,

[0175] The server is configured to receive and store the uplink video stream uploaded by the wireless communication based cross-network interaction intermediary device;

[0176] The control terminal is configured to call and display the uplink video stream in real time;

[0177] The control terminal comprises a display, one or more processors, at least one memory, and one or more modules stored in the memory and configured to be executed by the one or more processors,

[0178] The one or more modules comprise:

[0179] a module configured to receive the uplink video stream uploaded in response to the server;

[0180] a module configured to send the uplink video stream to the display for display.

[0181] Preferably,

[0182] The module configured to send the uplink video stream to the display for display comprises:

[0183] Referring to the accompanying drawings, Figure 10 in the first mode, a module configured to select the uplink video stream corresponding to the selected ID number to a full-screen playing display window in the display for display;

[0184] Referring to the accompanying drawings, Figure 8 in the second mode, a module configured to simultaneously display the uplink video streams corresponding to the multiple ID numbers in the display in the multiple ID playing display windows corresponding to the multiple ID numbers;

[0185] Referring to the accompanying drawings, Figure 9 in the third mode, a module configured to select the cutting range of the uplink video streams corresponding to the multiple ID numbers and send the cutting range to a playing display window for display.

[0186] Preferably,

[0187] The one or more modules further comprise at least one of the following modules:

[0188] As shown in the accompanying drawings, Figure 8

[0189] after selecting an ID number, a module configured to record the mouse event of the mouse of the control terminal in the playing display window of the uplink video stream corresponding to the ID number and forward the mouse event to the wireless communication based cross-network interaction intermediary device corresponding to the selected ID number;

[0190] after selecting an ID number, a module configured to record the keyboard event of the control terminal and forward the keyboard event to the wireless communication based cross-network interaction intermediary device corresponding to the selected ID number.

[0191] Preferably,​

[0192] The mouse event includes at least one of movement, selection,

[0193] The mouse event includes at least one of movement, selection, instruction and data input.

[0194] Embodiment 4

[0195] As Figure 1 shown, an interactive system includes

[0196] At least one controlled end, a one-to-one wireless communication-based cross-network interactive mediation device is configured for the controlled end, and one control system,

[0197] The HDMI interface and the PS / 2 interface of the interactive mediation device are connected with the parallel HDMI interface and the parallel PS / 2 interface of the controlled end respectively, the interactive mediation device is wirelessly linked with the service end of the control system through a wireless communication module, and the service end of the control system is linked with the control end of the control system in networking.

[0198] The controlled end is divided into an inner network, and the interactive mediation device, the service end of the control system and the control end of the control system are divided into an outer network.

[0199] The controlled end, the interactive mediation device, the service end of the control system and the control end of the control system form a video data uplink and a control instruction downlink.

[0200] In terms of networking, the controlled end of the inner network provided by the application is captured to a video stream through a one-to-one interactive mediation device, the networks of the two are physically isolated, the peripheral interactive mediation device serves as a push stream device, and the interactive mediation device faces one service end in a many-to-one manner, and the service end and the control end are re-networked to form a viewing system, the control end can simultaneously obtain video streams of multiple controlled ends for synchronized display or display after screenshot splicing, so that in some special application scenarios, such as joint exercises of the army, since each exercise unit uses its own inner network, the command observation unit is not connected with the exercise units, at this time, the video streams of multiple interactive mediation devices can be requested from the service end on the control end of the command observation unit by relying on the application, so that the interfaces of multiple controlled ends are synchronized to one interface for live broadcast across the network, and on the basis of the uploaded frame pictures, the frame pictures can be provided to the service end by using the frame picture compression based on the number of bytes in the application to adapt to the network speed, and the control end can also perform fast screenshot splicing live broadcast on the frame pictures of multiple controlled ends based on the frame pictures.

[0201] For network adaptation, in view of the limitation of existing uplink network speed and the interference of network fluctuation, the main purpose of the present application is to establish the ability of video stream fusion and interactive operation for the purpose of interface between computer terminals in different closed private networks. For the problem of network fluctuation, the conversion technology suitable for the current uplink network speed is proposed, and the video stream with large byte number is converted into uplink video stream suitable for the current uplink network speed according to the adaptive parameters, so as to avoid the problem of poor fluency caused by network delay.

[0202] The cross-network interaction mediation device in the application is an electronic device based on single-chip microcomputer application, which cooperates with the functional software module of the above-mentioned adaptive conversion parameter or lookup table conversion parameter. The interactive system constructs only the video data uplink for video stream and only the control instruction downlink for mouse and keyboard event.

[0203] Using HDMI interface connection, collecting HDMI video stream, using PS / 2 interface to connect the keyboard and mouse interface of the controlled end, obtaining the keyboard and mouse instruction set of the air end from the server in real time, and issuing the keyboard and mouse instruction to the controlled end, so as to realize remote control of the keyboard and mouse behavior of the controlled end, and only used for simulating keyboard and mouse events to ensure no security risk; using 4G / 5G wireless network module, which can obtain networking capability after inserting SIM, but is designed only for uploading HDMI video stream and issuing keyboard and mouse instruction set, and is not used for other purposes.

[0204] For the server and the control end, C / S architecture is adopted, and the main functions are:

[0205] The server is used for receiving uplink video stream, waiting for control end calling and display, receiving mouse and keyboard instruction set uploaded by the control end, and forwarding to the controlled end to realize remote control;

[0206] Referring to the accompanying Figure 7 The control end is the operation end of the staff, and multiple controlled terminal interfaces can be displayed in the large screen in parallel, or the interface area can be cut and spliced for fusion display according to actual needs; after selecting a certain controlled terminal, the terminal interface can be enlarged alone, and the local keyboard and mouse can be used at this time, and the local keyboard and mouse events will be issued to the controlled terminal through the server, so as to realize remote interface fusion and interactive interaction.

[0207] The core device of the interactive intermediary device includes MCU / CPU processor, 4G / 5G network module, FLASH flash memory and power management module, and is provided with HDMI interface, PS / 2 interface, USB interface and SIM card slot, and the USB interface is powered. The processor acquires the real-time video stream of the HDMI interface, extracts the frame picture of the original size at the matched frame rate, performs compression of resolution and / or precision at the matched compression ratio parameter, temporarily stores one frame in the FLASH flash memory, finally constitutes the uplink video stream, and sends the uplink video stream to the designated server using the networking capability of the wireless network module. The USB interface power supply completes power taking from the general power adapter and supplies power to the entire electronic device through the power management module. In addition, the device automatically acquires the keyboard and mouse instruction set data of the device number from the server at a fixed frequency, sends the keyboard and mouse instruction set data to the controlled end through the PS / 2 interface, and realizes the response event of the keyboard and mouse.

[0208] The operator uses the control end to acquire the interfaces of multiple controlled ends from the server and display on the large screen. In order to obtain the effective area on the interface and remove the invalid area, a part of the interface can be intercepted and spliced into other windows in a cutting manner, so as to realize free splicing and fusion of multiple terminal interfaces. After selecting the interface of each controlled end, the interface can be enlarged on the full screen, and the local keyboard and mouse are operated. The keyboard and mouse instruction set is sent to the corresponding electronic device through the server, and then acts on the controlled end through the PS / 2, so as to realize the interface fusion of multiple terminals across the network.

[0209] The above is an embodiment of the present application, which is not a specific limitation on the protection scope of the present application.

Claims

1. A cross-network interaction mediation device based on wireless communication, characterized by, The device comprises: at least one HDMI interface, which is used to connect with a parallel HDMI interface of a host of a controlled terminal to obtain a real-time video stream, the real-time video stream being an active video signal of the parallel HDMI interface of the controlled terminal, and the parallel HDMI interface being in parallel with an execution HDMI interface of the controlled terminal for transmitting the active video signal to a display; one or more processors, at least one memory, at least one wireless communication module, a module for converting the real-time video stream into an uplink video stream, the module for converting the real-time video stream into the uplink video stream comprises: a module for obtaining a current network strength value of the wireless communication module, a module for obtaining the real-time video stream, According to the current network strength value, a corresponding current uplink network speed S is matched out 上 of the module; The module for matching the byte number conversion parameter according to the resolution and bit depth of the frame picture of the active video signal to adapt to the current uplink network speed S 上 . a module for converting the real-time video stream into the uplink video stream according to a byte number conversion parameter, the uplink video stream further comprising an ID number of the cross-network interaction intermediary device, a module for sending the uplink video stream to a server in a push mode through the wireless communication module.

2. The cross-network interaction intermediary device based on wireless communication according to claim 1, wherein the module for converting the real-time video stream into the uplink video stream comprises: the module for obtaining the real-time video stream, a module for processing the real-time video stream by using a video encoding algorithm of an H series or an MPEG series to obtain a binary coded uplink video stream, the uplink video stream further comprising the ID number of the cross-network interaction intermediary device.

3. The cross-network interaction intermediary device based on wireless communication according to claim 1, wherein The module for matching the byte number conversion parameter according to the resolution and bit depth of the frame picture of the active video signal to adapt to the current uplink network speed S 上 includes: a module for extracting a frame rate, a resolution of one frame picture, and a bit depth of the real-time video stream, a module for adaptively solving the byte number conversion parameter according to a byte number conversion parameter formula and a constraint condition, Wherein, the byte conversion parameter formula is: S 上 = ((L*K*(W / 8)) / 1024 / 1024)*m%*n% *i, S 上 Current uplink network speed for the wireless communication module, in MB / s, L*K is the resolution of one frame picture of the real-time video stream, W is the bit depth of one frame picture of the real-time video stream, and is in byte, i is a selectable frame rate of the uplink video stream, and is in frame / s, m% is a selectable resolution compression ratio of a frame picture of the uplink video stream, n% is a selectable precision compression ratio of a frame picture of the uplink video stream, the byte number conversion parameter comprises i, m%, and n%, the constraint condition comprises: constraint condition 1: the frame rate of the real-time video stream ≥ i ≥ 1, constraint condition 2: 100% ≥ m% ≥ 30%, constraint condition 3: 100% ≥ n% ≥ 30%.

4. The wireless communication based cross-network interaction brokering apparatus of claim 1, wherein, The module for matching the byte number conversion parameter according to the resolution and bit depth of the frame picture of the active video signal to adapt to the current uplink network speed S 上 includes: a module for obtaining a conversion parameter matching table, According to the current uplink network speed S 上 The module for searching the byte number conversion parameter corresponding to the uplink network speed corresponding to the network intensity value of each level calibrated in the conversion parameter matching table and outputting the byte number conversion parameter. the conversion parameter matching table comprises: each level of network strength value and an uplink network speed corresponding to each level of network strength value obtained by calibration, a selectable frame rate of the uplink video stream, a selectable resolution compression ratio of a frame picture of the uplink video stream, and a selectable precision compression ratio of a frame picture of the uplink video stream, which are adaptively matched with the uplink network speed corresponding to each level of network strength value obtained by calibration, the selectable resolution compression ratio and the selectable precision compression ratio are both calculated by taking the single-frame byte number of the frame picture of the active video signal obtained by calibration as a reference.

5. The wireless communication based cross-network interaction brokering apparatus of claim 1, wherein further comprising: at least one PS / 2 interface, which is used to connect with a parallel PS / 2 interface of a host of a controlled terminal, one or more modules further comprise: a module for receiving a mouse event or / and a keyboard event of a downlink remote control of a control terminal through the wireless communication module, A module for compiling mouse events or / and keyboard events into a format recognizable by a PS / 2 interface.

6. A control system characterized by, The control system comprises: a service end and a control end, the service end is configured to receive and store an uplink video stream uploaded by the cross-network interaction intermediary device based on wireless communication in any one of claims 1-5; the control end is configured to call and display the uplink video stream in real time; the control end comprises a display, one or more processors, and at least one memory, a module for receiving the uplink video stream uploaded in response to the service end; a module for sending the uplink video stream to the display for display; the module for sending the uplink video stream to the display for display comprises: a module for selecting the uplink video stream corresponding to one ID number in a first mode to a full-screen playing display window in the display for display; a module for simultaneously displaying the uplink video streams corresponding to multiple ID numbers in a second mode in multiple ID playing display windows in the display; a module for selecting a cutting range of the uplink video streams corresponding to multiple ID numbers in a third mode and sending the cutting range to a playing display window for display.

7. The control system according to claim 6, wherein the one or more modules further comprise at least one of the following modules: a module for recording mouse events of a mouse of the control end in a playing display window of the uplink video stream corresponding to the selected ID number and forwarding the mouse events to the cross-network interaction intermediary device based on wireless communication corresponding to the selected ID number after the ID number is selected; a module for recording keyboard events of the control end and forwarding the keyboard events to the cross-network interaction intermediary device based on wireless communication corresponding to the selected ID number after the ID number is selected.

8. The control system according to claim 7, wherein the mouse events comprise at least one of movement and selection, the mouse events comprise at least one of movement, selection, instruction, and data input.

9. An interactive system, characterized by The control system comprises at least one controlled end, a cross-network interaction intermediary device based on wireless communication in any one of claims 1-5 configured for the controlled end in a one-to-one manner, and a control system in any one of claims 6-8, the HDMI interface and the PS / 2 interface of the cross-network interaction intermediary device are connected to the parallel HDMI interface and the parallel PS / 2 interface of the controlled end, respectively, the cross-network interaction intermediary device is wirelessly connected to the service end of the control system through a wireless communication module, and the service end of the control system is connected to the control end of the control system in a networked manner; the controlled end is classified as an internal network, and the cross-network interaction intermediary device, the service end of the control system, and the control end of the control system are classified as an external network; the controlled end, the cross-network interaction intermediary device, the service end of the control system, and the control end of the control system form an uplink of video data and a downlink of control instructions.

Images