Remote terminal equipment control method and system

Abstract

The invention relates to a remote terminal equipment control method and system. The method comprises the following steps: capturing operation of a keyboard and / or a mouse of a control end system; encoding the native event into a cloud PC private keyboard and mouse protocol packet; the cloud PC private keyboard and mouse protocol packet is sent to a remote coding card through a network, unpacking is carried out, event types and parameters of a keyboard and a mouse are recognized, and description content meeting the USB HID standard is created according to the event types; generating a virtual device of a keyboard / mouse in the coding card, and converting the event description into a USB HID message; the virtual equipment is connected to a blade computer of a controlled end through a hardware interface of the coding card and simulates direct keyboard / mouse operation of a user; and the blade computer regards the virtual equipment of the coding card as a keyboard and a mouse which are locally connected. According to the control method combining the hardware coding card and the private keyboard and mouse protocol, remote control over the keyboard and the mouse of the controlled end can be achieved no matter how the operating system state of the controlled end is.

Description

Technical Field

[0001] The invention relates to a remote terminal equipment control method and system, belonging to the technical field of remote control and virtualization equipment. Background Art

[0002] With the rapid development of cloud computing and virtualization technology, more and more users are using cloud PCs, virtual desktops and remote desktop protocols (such as RDP, VNC, etc.) to achieve remote office, operation and management. However, the current mainstream remote control solutions usually rely on installing client programs on the controlled end to capture the keyboard and mouse input of the control end and transmit it to the controlled end to achieve remote operation.

[0003] In some of the existing technologies, the controlled PC must install a specific client program to receive remote commands, which has high requirements on the system environment. The controlled PC needs to have installation permissions and may be restricted by the operating system or security policies. At the same time, the long-term operation of the controlled client software is also likely to lead to system resource consumption, reduce the performance of the controlled PC, and if the client program crashes or is affected by network fluctuations, it will cause the keyboard and mouse operations to be out of control, affecting the reliability of remote control. In addition, the existing software remote control method depends on the normal operation of the operating system. Once the controlled end enters the BIOS, the operating system crashes or is in a state without an operating system (such as a bare metal state), the remote operation function fails and the necessary maintenance operations cannot be performed. Summary of the invention

[0004] The present invention provides a remote terminal equipment control method and system, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of the present invention relates to a remote terminal device control method, and the method according to the present invention comprises the following steps:

[0006] S100, capturing the keyboard and / or mouse operations of the control end system in real time as native input events; encoding the native events into cloud PC private keyboard and mouse protocol packets;

[0007] S200, sending the cloud PC private keyboard and mouse protocol package to the remote encoding card through the network, so as to unpack the cloud PC private keyboard and mouse protocol package and identify the event type and parameters of the keyboard and mouse, and create a description content that complies with the USB HID standard according to the identified event type;

[0008] S300, generating a keyboard / mouse virtual device in the encoding card, and converting the event description into a USBHID message; the virtual device is connected to the blade computer of the controlled end through the hardware interface of the encoding card, and simulates the user's direct keyboard / mouse operation;

[0009] S400: The blade computer regards the virtual device of the encoding card as a locally connected keyboard and mouse.

[0010] Furthermore, the private keyboard protocol of the cloud PC private keyboard and mouse protocol package adopts a keyboard data format with a frame length of 8 bytes.

[0011] Furthermore, in step S200, the data structure of the private keyboard protocol includes an auxiliary key for representing a key combination and including a control key state, and the auxiliary key supports a combination of a left Ctrl key, a right Ctrl key, a Shift key, an Alt key, and a Win key.

[0012] Furthermore, in step S200, the keyboard key events are processed by a keyboard event parsing algorithm to convert the keyboard key events into a standardized data structure;

[0013] The keyboard event parsing algorithm includes the following steps:

[0014] A110, initializing key states and keyboard capture; including the steps of: initializing the control key states of the keyboard, and calling the system API to obtain the current pressing states of each auxiliary key; wherein the control keys include the states of the left Ctrl key, the right Ctrl key, the Shift key, the Alt key, and the Win key;

[0015] A120, performing key code mapping; including the steps of: mapping the key input value to a HID keyboard code according to the source and type of the event;

[0016] A130, executing key press event processing; including the steps of: A131, updating auxiliary key status; A132, recording the normal key pressed;

[0017] A140, executing key release event processing; including the steps of: A141, updating auxiliary key status; A142, removing ordinary keys;

[0018] A150. Execute state change detection and update; including the steps of: comparing the current standardized data structure with the last key state to detect whether there is a change; if the state has changed, updating the last integrated data to the current state, and returning true to indicate that the key state has changed.

[0019] Further, in step S200: the private mouse protocol of the cloud PC private keyboard and mouse protocol package adopts a mouse data format with a frame length of 8 bytes; wherein a frame of data can represent any group of mouse operations, including:

[0020] Type indicates that the remote control terminal uses absolute position data;

[0021] Id is used to identify the check digit;

[0022] act represents the combined value of mouse button pressing and releasing;

[0023] rol indicates the direction of the roller;

[0024] x_low and x_high assign the lower 8 bits and higher 8 bits of the local mouse x coordinate to x_low and x_high, respectively, to indicate the position of the mouse on the X axis;

[0025] y_low and y_high assign the lower 8 bits and higher 8 bits of the local mouse y coordinate to y_low and y_high respectively, indicating the position of the mouse on the Y axis.

[0026] Further, in step S200, a plurality of mouse and keyboard events are encoded into standardized data messages through a mouse event parsing algorithm to transmit and identify events in a remote control or virtual terminal;

[0027] Wherein, the mouse event parsing algorithm comprises the following steps:

[0028] A210, executing event reception and type determination; including the steps of: receiving data information of mouse or keyboard events from the input structure MK_MSG, and identifying the event type according to the usType field therein;

[0029] A220, record and update the execution key status;

[0030] A230, performing position data encoding; including the steps of: converting the X and Y coordinates of the input event into low-order bytes and high-order bytes, and storing them in the x_low, x_high, y_low and y_high fields of the message respectively;

[0031] A240, generating a message; wherein the finally generated HID_MOUSE is transmitted to the target end through the network to support the mouse event simulation operation of the remote system.

[0032] Further, the step A220 includes:

[0033] When a mouse button event is detected, the mouse event parsing algorithm updates the global button state variable based on whether the button is pressed or released;

[0034] When a mouse button press event is detected, the corresponding status flag will be set, and the act field of the message will be modified to set the corresponding button bit to 1;

[0035] When a key-up event is detected, the corresponding status flag is cleared and the key bit is set to 0 in the act field;

[0036] In the processing of the wheel scrolling event, the wheel event will trigger the update of the rol field to indicate the scrolling direction.

[0037] The technical solution of the present invention also relates to a computer-readable storage medium on which program instructions are stored, and the above-mentioned method is implemented when the program instructions are executed by a processor.

[0038] The technical solution of the present invention also relates to a remote terminal equipment control system, wherein the system comprises a computer device, and the computer device comprises the above-mentioned computer-readable storage medium.

[0039] Furthermore, the remote terminal equipment control system includes:

[0040] An event monitoring module for capturing control-side events;

[0041] An event conversion module used to encode native events into cloud PC private keyboard and mouse protocol packets;

[0042] A data transmission module for sending cloud PC private keyboard and mouse protocol packets to a remote encoding card;

[0043] A protocol parsing module for unpacking PC private keyboard and mouse protocol packets and identifying event types;

[0044] A USB HID simulation module used to generate a virtual device and convert event descriptions into USB HID messages.

[0045] The beneficial effects of the present invention are as follows:

[0046] The remote terminal device control method of the present invention proposes a control method combining a hardware encoding card and a private keyboard and mouse protocol for remote control scenarios. It captures and converts private keyboard and mouse messages at the control end, transmits them to the encoding card through the network, and then converts them into USB HID device signals by the encoding card program, simulating keyboard and mouse operations at the hardware level, so that no matter what the operating system status of the controlled end is, remote control of its keyboard and mouse can be achieved, reducing dependence on system permissions, improving the stability, low latency and security of remote control, and making it suitable for a wider range of remote operation and maintenance scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the process according to the method of the present invention.

[0048] Figure 2 Schematic diagram of the structure of the system according to the present invention. DETAILED DESCRIPTION

[0049] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0050] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0051] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, will not impose limitations on the scope of the present invention.

[0052] Reference Figure 1 to Figure 2 In some embodiments, the remote terminal device control method and system according to the present invention at least include the following steps:

[0053] S100, capturing the keyboard and / or mouse operations of the control end system in real time as native input events; encoding the native events into cloud PC private keyboard and mouse protocol packets;

[0054] S200, sending the cloud PC private keyboard and mouse protocol packet to the remote encoding card through the network, so as to unpack the cloud PC private keyboard and mouse protocol packet and identify the event type and parameters of the keyboard and mouse, and create a description content that complies with the USB HID standard according to the identified event type;

[0055] S300, generating a keyboard / mouse virtual device in the encoding card, and converting the event description into a USB HID message; the virtual device is connected to the blade computer of the controlled end through the hardware interface of the encoding card, and simulates the user's direct keyboard / mouse operation;

[0056] S400, the blade computer regards the virtual device of the encoding card as a locally connected keyboard and mouse.

[0057] Specifically, the remote terminal device control of the present invention is provided with an event monitoring module for capturing control-end events, an event conversion module for encoding native events into cloud PC private keyboard and mouse protocol packets, a data transmission module for sending the cloud PC private keyboard and mouse protocol packets to a remote encoding card, a protocol parsing module for unpacking the PC private keyboard and mouse protocol packets and identifying event types, and a USB HID simulation module for generating a virtual device and converting event descriptions into USB HID messages.

[0058] The remote terminal device control method of the present invention is a remote operation method for controlled computing devices such as cloud PCs by matching hardware devices. Specifically, it realizes the control end system (such as Windows, Linux, macOS, etc.) without the need to install any client software on the controlled end PC through the design of the encoding card and the private protocol. Stable, low-latency keyboard and mouse operations are performed to improve the compatibility and reliability of remote operation and maintenance. Among them, the present invention adopts a hardware-combined software architecture, which is an architecture for protecting software-to-hardware control solutions, and concentrates the collection, conversion, and transmission of operation data on the client encoding module, and integrates the processes of parsing and converting USB HID description signals in the encoding card software module. Among them, the definition and application of the cloud PC private protocol of the present invention adopt a newly designed private protocol format, as well as a keyboard event parsing algorithm and a mouse event parsing algorithm. Among them, in the implementation of the encoding card virtual USB HID device of the present invention, the encoding card directly simulates the USB HID virtual device to transmit the control signal, so that the controlled PC can be free of installing the client software.

[0059] In some embodiments, when the present invention executes control end event capture, the keyboard and mouse operations of the control end system are captured in real time through the event monitoring module, and the native system (Windows, Linux, macOS) events (such as key codes, mouse position coordinates) are passed to the event conversion module.

[0060] In some embodiments, when the present invention performs event conversion, the event conversion module encodes the native event into a cloud PC private keyboard and mouse protocol packet to control data transmission on the network and can be uniformly parsed. The conversion to a private protocol packet format is transmitted over the Internet through an encrypted secure channel to improve the security of data transmission.

[0061] In some embodiments, when the present invention executes data transmission, the data transmission module sends the cloud PC private keyboard and mouse protocol package to the remote encoding card through the network, thereby ensuring reliable transmission in different network environments. If necessary, a transmission protocol stack can be added to increase flow control and packet loss retransmission mechanism.

[0062] In some embodiments, when the present invention executes the encoding card to parse the protocol packet, after the encoding card receives the cloud PC private keyboard and mouse protocol packet, the protocol parsing module unpacks the PC private keyboard and mouse protocol packet and identifies the type and parameters of the keyboard and mouse events. Further, based on the event type, a description content that complies with the USB HID standard is created, such as key press / release and mouse movement / scroll wheel operations.

[0063] In some embodiments, when the present invention executes the encoding card to simulate the USB HID device, the USB HID simulation module generates a keyboard / mouse virtual device in the encoding card and converts the event description into a USB HID message. The keyboard / mouse virtual device is connected to the controlled blade computer through the hardware interface of the encoding card and simulates the user's direct keyboard / mouse operation.

[0064] In some embodiments, when the present invention executes an event on a cloud PC, the blade computer on the cloud PC side regards the virtual USB device of the encoding card as a locally connected keyboard and mouse, thereby eliminating the need to install any specific client software and relying on operating system permissions.

[0065] In some embodiments, in the cloud PC private keyboard and mouse protocol package of the present invention, the private keyboard protocol adopts a keyboard data format with a frame length of 8 bytes, which is expressed as follows:

[0066] Assist Keys blank Normal key 1 Normal key 2 Normal key 3 Normal key 4 Normal key 5 Normal key 6 8-bit 8-bit 8-bit 8-bit 8-bit 8-bit 8-bit 8-bit

[0067] The data structure of the private keyboard protocol is described as follows:

[0068] struct hid_input_event {

[0069] unsigned char data0;

[0070] unsigned char data1;

[0071] unsigned char codes[6];

[0072] };

[0073] data0 is 8 bits: it represents the combination key field, supports multiple combination key encodings, and is represented by a bit mask.

[0074] data1 is 8 bits: it represents a reserved field, used to distinguish it from a common key, and is always 0.

[0075] codes[0] is 8 bits: represents normal key 1.

[0076] codes[1] is 8 bits: represents normal key 2.

[0077] codes[2] is 8 bits: represents normal key 3.

[0078] codes[3] is 8 bits: represents normal key 4.

[0079] codes[4] is 8 bits: represents normal key 5.

[0080] codes[5] is 8 bits: it represents normal key 6.

[0081] In an application embodiment, in the private keyboard protocol of the present invention, the auxiliary key is used to represent the combination key, including the state of the control key. It supports the combination of the left Ctrl, right Ctrl, Shift, Alt, and Win keys, which are encoded in a bit mask manner. Among them, the control key values ​​are defined as: KEY_L_CTRL = 0x01, KEY_L_SHIFT = 0x02, KEY_L_ALT = 0x04, KEY_L_WIN = 0x08, etc., which represent the left Ctrl, left Shift, left Alt, and left Win keys respectively; KEY_R_CTRL = 0x10, KEY_R_SHIFT = 0x20, KEY_R_ALT = 0x40, KEY_R_WIN = 0x80, which represent the right Ctrl, right Shift, right Alt, and right Win keys respectively; if multiple combination keys are pressed at the same time, the key values ​​are merged using the bitwise OR (|=) operation, for example: if the left Ctrl and left Alt keys are pressed at the same time, data0 = KEY_L_CTRL | KEY_L_ALT. Furthermore, its blank segment is set to 0 to distinguish the combination key information from the ordinary keys. Furthermore, a 6-byte unsigned char array is used to fill 6 common key values, which are used to record the common keys pressed by the user (such as letters, numbers, function keys, etc.), wherein all 6 bytes are 0, which represents that the common keys are not pressed.

[0082] In an application embodiment, the present invention processes keyboard key events through a keyboard event parsing algorithm, and converts the keyboard key events into a standardized hid_input_event data structure to support continuous tracking of key states and recognition of multiple key combinations.

[0083] Specifically, the workflow of the keyboard event parsing algorithm includes the following steps:

[0084] A110. Initialize key status and keyboard capture. Specifically, the keyboard event parsing algorithm initializes the keyboard control key status (including the left and right Ctrl, Shift, Alt, and Win keys), and calls the system API to obtain the current pressing status of each auxiliary key (Ctrl, Shift, Alt, and Win). It should be noted that for macOS systems, if a Meta key event is detected, it is considered as the Win key being pressed.

[0085] A120, execute key code mapping. It maps the key input value to the HID keyboard code according to the source and type of the event. Specifically, the keyboard event parsing algorithm of the present invention first determines whether the key is from the numeric keypad, wherein if it is a numeric keypad, it is mapped to the HID code of the numeric keypad; if it is the Tab key, a predefined key value is used to ensure that it does not affect the window focus; other keys are converted to standard keyboard codes.

[0086] A130, execute key press event processing. Specifically, for a keyboard press event, the keyboard event parsing algorithm of the present invention performs the following processing:

[0087] A131, update the auxiliary key status. Specifically, if the Ctrl, Shift, Alt, Win and other key combinations are pressed, the algorithm updates the data0 field in hid_input_event through bit operations and sets the corresponding auxiliary key bit flag.

[0088] A132. Record the normal keys pressed. Specifically, for non-auxiliary keys, the keyboard event parsing algorithm records the HID key value in the codes array, which supports storing up to 6 keys. The algorithm traverses the codes array to find an empty position. If the key value is new and the array is not full, it is added to the codes array.

[0089] A140, executing key release event processing. Specifically, for a keyboard release event, the keyboard event parsing algorithm of the present invention releases the key state through the following operations:

[0090] A141, update the auxiliary key status. Specifically, when the release event of the Ctrl, Shift, Alt, Win key is detected, the algorithm clears the corresponding auxiliary key bit mark in the data0 field through bit operation, indicating that the key combination has been released.

[0091] A142, remove common keys. Specifically, the keyboard event parsing algorithm of the present invention checks the codes array, and if a key matching the released HID key value is found, it is removed and the subsequent keys are moved forward to maintain the continuity of the codes array.

[0092] A150, perform state change detection and update. Specifically, after processing the key event, the keyboard event parsing algorithm compares the current standardized data structure hid_input_event with the last key state to detect whether there is a change. If the state changes, the keyboard event parsing algorithm updates the last integrated data to the current state, and the algorithm function returns true, indicating that the key state has changed.

[0093] In some embodiments, in the cloud PC private keyboard and mouse protocol package of the present invention, the data structure of the private mouse protocol is described as follows:

[0094] struct HID_MOUSE

[0095] {

[0096] unsigned char type = 0xA1; / / 0xA1 is used as absolute position mouse mode

[0097] unsigned char id;

[0098] unsigned char act;

[0099] unsigned char rol;

[0100] unsigned char x_low;

[0101] unsigned char x_high;

[0102] unsigned char y_low;

[0103] unsigned char y_high;

[0104] };

[0105] / / Local system

[0106] struct MK_MSG

[0107] {

[0108] unsigned short usType;

[0109] unsigned short usX;

[0110] unsigned short usY;

[0111] unsigned short usKey;

[0112] MK_MSG() : usType(0), usX(0), usY(0), usKey(0)

[0113] {}

[0114] };

[0115] / / Local system events

[0116] enum EM_MK_TYPE

[0117] {

[0118] MK_MOUSE_MOVE, / / Mouse move

[0119] MK_MOUSE_LEFT_PRESS, / / left button pressed

[0120] MK_MOUSE_LEFT_RELESE, / / left button released

[0121] MK_MOUSE_MID_PRESS, / / Middle button pressed

[0122] MK_MOUSE_MID_RELESE, / / Middle button released

[0123] MK_MOUSE_RIGHT_PRESS, / / right button pressed

[0124] MK_MOUSE_RIGHT_RELESE, / / right button released

[0125] MK_KEY_PRESS, / / Keyboard pressed

[0126] MK_KEY_RELESE, / / Keyboard pops up

[0127] MK_WHEEL_UP, / / Roll wheel up

[0128] MK_WHEEL_DOWN, / / Roll wheel down

[0129] };

[0130] / / Mouse button type of the transmission protocol

[0131] enum ACT_BIT

[0132] {

[0133] BIT_BT_LEFT, / / left button pressed

[0134] BIT_BT_RIGHT, / / right button pressed

[0135] BIT_BT_MID, / / The middle mouse button is pressed

[0136] };

[0137] Furthermore, the private mouse protocol of the cloud PC private keyboard and mouse protocol package of the present invention adopts a mouse data format with a frame length of 8 bytes, which is expressed as follows:

[0138] type id act rol x_low x_high y_low y_high 8-bit 8-bit 8-bit 8-bit 8-bit 8-bit 8-bit 8-bit

[0139] Among them, one frame of data can represent any group of mouse operations, including:

[0140] type: indicates that the remote control terminal uses absolute position data.

[0141] id: identifies the check digit.

[0142] act: Indicates the combined value of mouse button pressing and releasing. Among them, BIT_BT_LEFT=0 means the left button is pressed, BIT_BT_RIGHT=1 means the right button is pressed, and BIT_BT_MID=3 means the middle button is pressed.

[0143] rol: indicates the direction of the scroll wheel. rol = 0x01 means the scroll wheel is scrolling up; rol = 0xff means the scroll wheel is scrolling down, and 0x00 means no scroll wheel operation.

[0144] x_low and x_high: Assign the lower 8 bits and higher 8 bits of the local mouse x coordinate to x_low and x_high respectively, indicating the position of the mouse on the X axis.

[0145] y_low and y_high: Assign the lower 8 bits and higher 8 bits of the local mouse y coordinate to y_low and y_high respectively, indicating the position of the mouse on the Y axis.

[0146] In an application embodiment, the present invention encodes various mouse and keyboard events into standardized data messages through a mouse event parsing algorithm so as to transmit and identify these events in a remote control or virtual terminal.

[0147] Specifically, the processing process of the mouse event parsing algorithm of the present invention includes the following steps:

[0148] A210, execute event reception and type judgment. Specifically, the mouse event parsing algorithm of the present invention receives data information of mouse or keyboard events from the input structure MK_MSG, and identifies the event type according to the usType field therein. Among them, the usType field defines a variety of mouse and keyboard events, including mouse movement, button pressing / release, scrolling of the wheel, etc.

[0149] Among them, in the mouse movement event, when usType indicates a mouse movement event, the mouse event parsing algorithm calls a dedicated parsing function to encode the current mouse button state and movement position data into the target message.

[0150] Among them, in the mouse button event, for the left, middle and right buttons of the mouse, the mouse event parsing algorithm can identify their pressed or released status, and update the button status flag accordingly to ensure that the current status of the button is accurately represented in the target message.

[0151] Among them, in the scroll wheel event, the mouse event parsing algorithm identifies the scrolling direction of the scroll wheel (such as scrolling up or scrolling down) through the usType field, and sets the corresponding identification data into the message.

[0152] A220. Record and update the execution key status.

[0153] Among them, when a mouse button event is detected, the mouse event parsing algorithm will update the global button state variables according to the button pressing or releasing status. These variables are used to record the button status of the left, middle and right buttons of the mouse, thereby supporting continuous tracking of the buttons. It should be noted that in the message, the button status bit is implemented by setting a specific bit of the act field.

[0154] Among them, when a key press event (such as left key press) is detected, the mouse event parsing algorithm will set the corresponding status flag and modify the act field of the message to set the corresponding key bit to "1".

[0155] Among them, when a key pop-up event (such as left key pop-up) is detected, the corresponding status flag is cleared and the key bit is set to "0" in the act field.

[0156] Among them, in the processing of the wheel scrolling event, the wheel event will trigger the update of the rol field to indicate the scrolling direction. The mouse event parsing algorithm marks the wheel scrolling up event as 0x01 and the wheel scrolling down event as 0xff to ensure that the wheel state is accurately described in the target message.

[0157] A230. Perform position data encoding. Specifically, the mouse event parsing algorithm converts the X and Y coordinates of the input event into low-order bytes and high-order bytes, which are stored in the x_low, x_high, y_low and y_high fields of the message, respectively, to support high-resolution mouse position transmission. Among them, the low-order bytes of the X and Y axes are extracted through bitwise AND operations, while the high-order bytes are obtained through right shift operations to ensure that the position data is correctly encoded in the message.

[0158] A240, generate a message. Specifically, after the above steps, the mouse event parsing algorithm generates the final HID_MOUSE message, which contains information such as event type, button status, scroll wheel direction and mouse position. The message is transmitted to the target end through the network to support the mouse event simulation operation of the remote system.

[0159] In some application embodiments, in the network transmission of the present invention, in the intranet environment scenario, the LAN is directly connected to the network, and the direct control terminal and the encoding segment can directly transmit the cloud PC private keyboard and mouse protocol package via TCP / UDP. Specifically, in the Internet environment scenario, for the sake of data transmission security and transmission reliability, a complex network environment is required in the middle. The software of the present invention adopts a QUIC protocol transit server in the network transmission module, which is specifically: keyboard and mouse protocol package-->QUIC client<-->QUIC server--> encoding end receives and parses the protocol package. Furthermore, through TLS encryption on the Internet, the external network environment can communicate efficiently, encrypted, and with low latency.

[0160] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0161] In addition, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0162] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, an RSM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0163] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0164] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A remote terminal device control method, characterized in that: The method comprises the following steps: S100, capturing the keyboard and / or mouse operations of the control end system in real time as native input events; encoding the native events into cloud PC private keyboard and mouse protocol packets; S200, sending the cloud PC private keyboard and mouse protocol package to the remote encoding card through the network, so as to unpack the cloud PC private keyboard and mouse protocol package and identify the event type and parameters of the keyboard and mouse, and create a description content that complies with the USB HID standard according to the identified event type; S300, generating a keyboard / mouse virtual device in the encoding card, and converting the event description into a USB HID message; the virtual device is connected to the blade computer of the controlled end through the hardware interface of the encoding card, and simulates the user's direct keyboard / mouse operation; S400: The blade computer regards the virtual device of the encoding card as a locally connected keyboard and mouse.

2. The method according to claim 1, characterized in that The private keyboard protocol of the cloud PC private keyboard and mouse protocol package adopts a keyboard data format with a frame length of 8 bytes.

3. The method according to claim 2, characterized in that In the step S200, the data structure of the private keyboard protocol includes an auxiliary key for representing a key combination and including a control key state, and the auxiliary key supports a combination of a left Ctrl key, a right Ctrl key, a Shift key, an Alt key, and a Win key.

4. The method according to claim 3, characterized in that In the step S200, the keyboard key events are processed by a keyboard event parsing algorithm to convert the keyboard key events into a standardized data structure; The keyboard event parsing algorithm includes the following steps: A110, initializing key states and keyboard capture; including the steps of: initializing the control key states of the keyboard, and calling the system API to obtain the current pressing states of each auxiliary key; wherein the control keys include the states of the left Ctrl key, the right Ctrl key, the Shift key, the Alt key, and the Win key; A120, performing key code mapping; including the steps of: mapping the key input value to a HID keyboard code according to the source and type of the event; A130, executing key press event processing; including the steps of: A131, updating auxiliary key status; A132, recording the normal key pressed; A140, executing key release event processing; including the steps of: A141, updating auxiliary key status; A142, removing ordinary keys; A150. Execute state change detection and update; including the steps of: comparing the current standardized data structure with the last key state to detect whether there is a change; if the state has changed, updating the last integrated data to the current state, and returning true to indicate that the key state has changed.

5. The method according to claim 1, characterized in that In step S200: the private mouse protocol of the cloud PC private keyboard and mouse protocol package adopts a mouse data format with a frame length of 8 bytes; wherein one frame of data can represent any group of mouse operations, including: Type indicates that the remote control terminal uses absolute position data; Id is used to identify the check digit; act represents the combined value of mouse button pressing and releasing; rol indicates the direction of the roller; x_low and x_high assign the lower 8 bits and higher 8 bits of the local mouse x coordinate to x_low and x_high, respectively, to indicate the position of the mouse on the X axis; y_low and y_high assign the lower 8 bits and higher 8 bits of the local mouse y coordinate to y_low and y_high respectively, indicating the position of the mouse on the Y axis.

6. The method according to claim 5, characterized in that In the step S200, a plurality of mouse and keyboard events are encoded into standardized data messages through a mouse event parsing algorithm to transmit and identify the events in a remote control or virtual terminal; Wherein, the mouse event parsing algorithm comprises the following steps: A210, executing event reception and type determination; including the steps of: receiving data information of mouse or keyboard events from the input structure MK_MSG, and identifying the event type according to the usType field therein; A220, record and update the execution key status; A230, performing position data encoding; including the steps of: converting the X and Y coordinates of the input event into low-order bytes and high-order bytes, and storing them in the x_low, x_high, y_low and y_high fields of the message respectively; A240, generating a message; wherein the finally generated HID_MOUSE is transmitted to the target end through the network to support the mouse event simulation operation of the remote system.

7. The method according to claim 6, characterized in that The step A220 includes: When a mouse button event is detected, the mouse event parsing algorithm updates the global button state variable based on whether the button is pressed or released; When a mouse button press event is detected, the corresponding status flag will be set, and the act field of the message will be modified to set the corresponding button bit to 1; When a key-up event is detected, the corresponding status flag is cleared and the key bit is set to 0 in the act field; In the processing of the wheel scrolling event, the wheel event will trigger the update of the rol field to indicate the scrolling direction. 8 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 7.

9. A remote terminal equipment control system, characterized in that: include: A computer device comprising a computer readable storage medium according to claim 8.

10. The system according to claim 9, characterized in that include: An event monitoring module for capturing control-side events; An event conversion module used to encode native events into cloud PC private keyboard and mouse protocol packets; A data transmission module for sending cloud PC private keyboard and mouse protocol packets to a remote encoding card; A protocol parsing module for unpacking PC private keyboard and mouse protocol packets and identifying event types; A USB HID simulation module used to generate a virtual device and convert event descriptions into USB HID messages.

Images