Remote control system, method and storage medium

Abstract

The present application provides a remote control system, method and storage medium of an application example. It relates to the field of remote control technology. The system includes: a control device and at least one type of controlled device. The controlled device includes a master device and a slave device. The control device is configured to generate a touch event in response to a user's touch operation, and send the touch event to the master device through the message queue telemetry transmission MQTT protocol; the touch event is used to characterize the click data and touch position generated by the user's touch on the control device. The master device is configured to receive a touch event from the control device, determine an operation instruction corresponding to the touch event, and in response to the operation instruction, perform an operation corresponding to the operation instruction, and send a touch event to the slave device through the MQTT protocol. The slave device is configured to receive a touch event from the master device, determine an operation instruction, and in response to the operation instruction, perform an operation corresponding to the operation instruction.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a remote control system, method and storage medium. Background Art

[0002] With the continuous development of science and technology, electronic devices have spread to all aspects of life, and electronic devices have greatly improved people's quality of life. When it is necessary to debug electronic devices in the same area (such as electronic shopping guide devices in shopping malls), each device needs to be debugged one by one. In addition, some types of electronic devices (such as large-screen electronic devices such as advertising machines, information screens, conference machines, etc.) are restricted by the usage scenario and cannot perform corresponding touch screen operations. The mouse is needed to control the electronic devices to achieve human-computer interaction. Summary of the invention

[0003] In one aspect, a remote control system is provided, comprising: a control device and at least one type of controlled device; the controlled device comprises a master device and at least one slave device.

[0004] The control device is configured to generate a touch event in response to a user's touch operation, and send the touch event to the main device via a message queuing telemetry transport (MQTT) protocol; the touch event is used to characterize click data and a touch position generated by the user touching the control device.

[0005] The master device is configured to receive the touch event from the control device, determine an operation instruction corresponding to the touch event, and in response to the operation instruction, perform an operation corresponding to the operation instruction, and send the touch event to the slave device through the MQTT protocol.

[0006] The slave device is configured to receive the touch event from the master device, determine the operation instruction, and execute the operation corresponding to the operation instruction in response to the operation instruction.

[0007] In some embodiments, the operation instruction is used to instruct to perform touch simulation on the touch event, wherein the master device and the slave device are both specifically configured to display information corresponding to the operation instruction in response to the operation instruction.

[0008] In some embodiments, the controlled device is further configured to feed back the displayed information corresponding to the operation instruction to the control device. The control device is configured to display the information fed back by the controlled device.

[0009] In some embodiments, the operation instruction is used to instruct mouse simulation of the touch event, wherein the master device and the slave device are specifically configured to control the local mouse pointer to perform a mouse operation corresponding to the operation instruction in response to the operation instruction.

[0010] In some embodiments, the controlled device is further configured to obtain preset coordinates, and control the local mouse pointer to move to a preset starting position based on the preset coordinates.

[0011] In some embodiments, the control device is further configured to: obtain and display identification information of at least one candidate device. In response to a user's first selection operation on the at least one candidate device, determine the master device. In response to a user's second selection operation on other candidate devices, determine at least one slave device; the other candidate devices include candidate devices in the at least one candidate device that do not include the master device.

[0012] In some embodiments, the control device is further configured to send a startup instruction to the main device.

[0013] The master device is configured to receive the startup instruction from the control device, start the MQTT service according to the startup instruction, subscribe to the coordinate message, and generate response information and send it to the control device; the response information is used to indicate that the master device successfully starts the MQTT service; the subscribed coordinate message is used to obtain the touch event.

[0014] The control device is configured to receive response information from the master device, and establish a communication connection with the master device through the MQTT protocol according to the response information.

[0015] The control device is further configured to send a connection instruction to the slave device according to the response information.

[0016] The slave device is configured to receive the connection instruction from the control device, establish a communication connection with the master device through the MQTT protocol according to the connection instruction, and subscribe to coordinate messages from the master device.

[0017] In some embodiments, the control device is further configured to subscribe to coordinate messages from the master device.

[0018] In some embodiments, the controlled device includes an input uinput driver module and an operating system module; the operating system is a Linux operating system or an Android operating system.

[0019] The uinput driver module is configured to determine an operation instruction corresponding to the touch event and send the operation instruction to the operating system module.

[0020] The operating system module is configured to receive the operation instruction from the uinput driver module, and in response to the operation instruction, execute the operation corresponding to the operation instruction.

[0021] In some embodiments, the controlled device includes an input window operating system application program interface sendinputWinAPI module and an operating system module; the operating system is a windows operating system.

[0022] The sendinput WinAPI module is configured to determine an operation instruction corresponding to the touch event and send the operation instruction to the operating system module.

[0023] The operating system module is configured to receive the operation instruction from the sendinput WinAPI module, and in response to the operation instruction, execute the operation corresponding to the operation instruction.

[0024] In some embodiments, the control device includes a touch screen; the touch screen includes at least one sub-screen; one sub-screen corresponds to a type of controlled device; wherein the control device is specifically configured to generate the touch event in response to a user's touch operation in the sub-screen.

[0025] On the other hand, a remote control method is provided, which is applied to a remote control system. The remote control system includes: a control device and at least one type of controlled device. The controlled device includes a master device and a slave device.

[0026] In response to the user's touch operation on the control device, a touch event is generated, and the touch event is sent to the master device through the message queue telemetry transmission MQTT protocol; the touch event is used to characterize the click data and touch position generated by the user's touch on the control device. The touch event from the control device is received by the master device, and the operation instruction corresponding to the touch event is determined, and in response to the operation instruction, the operation corresponding to the operation instruction is performed, and the touch event is sent to the slave device through the MQTT protocol. The touch event from the master device is received by the slave device, and the operation instruction is determined, and in response to the operation instruction, the operation corresponding to the operation instruction is performed.

[0027] In some embodiments, the operation instruction is used to instruct touch simulation of the touch event; and in response to the operation instruction, executing the operation corresponding to the operation instruction includes: in response to the operation instruction, displaying information corresponding to the operation instruction.

[0028] In some embodiments, after displaying the information corresponding to the operation instruction, the method further comprises: feeding back the displayed information corresponding to the operation instruction to the control device through the controlled device. Displaying the information fed back by the controlled device through the control device.

[0029] In some embodiments, the operation instruction is used to instruct mouse simulation of the touch event, and the response to the operation instruction and the execution of the operation corresponding to the operation instruction include: in response to the operation instruction, controlling the local mouse pointer to execute the mouse operation corresponding to the operation instruction.

[0030] In some embodiments, before generating a touch event in response to a user's touch operation on the control device, the method further includes: acquiring preset coordinates through the controlled device, and controlling the local mouse pointer to move to a preset starting position based on the preset coordinates.

[0031] In some embodiments, before generating a touch event in response to a user's touch operation on the control device, the method also includes: acquiring and displaying identification information of at least one slave device through the control device, determining the master device in response to the user's first selection operation on the at least one slave device, and determining at least one slave device in response to the user's second selection operation on other alternative devices; the other alternative devices include alternative devices that do not include the master device in the at least one slave device.

[0032] In some embodiments, before generating a touch event in response to a user's touch operation on the control device, the method includes: sending a startup instruction of the control device to the master device through the control device. The startup instruction from the control device is received by the master device, and the MQTT service is started according to the startup instruction, and a coordinate message is subscribed, and a response message is generated and sent to the control device; the response information is used to indicate that the master device successfully starts the MQTT service; the subscribed coordinate message is used to obtain the touch event. The response information from the master device is received by the control device, and a communication connection is established with the master device through the MQTT protocol according to the response information. The control device sends a connection instruction to the slave device according to the response information. The connection instruction from the control device is received by the slave device, and a communication connection is established with the master device through the MQTT protocol according to the connection instruction, and a coordinate message is subscribed to the master device.

[0033] In some embodiments, after receiving the response information from the master device through the control device and establishing a communication connection with the master device through the MQTT protocol based on the response information, the method further includes: sending a subscription coordinate message of the control device to the master device.

[0034] In some embodiments, the controlled device includes an input uinput driver module and an operating system module; the operating system is a Linux operating system or an Android operating system; the determining the operation instruction corresponding to the touch event, and executing the operation corresponding to the operation instruction in response to the operation instruction, specifically includes: determining the operation instruction corresponding to the touch event through the uinput driver module, and sending the operation instruction to the operating system module. The operating system module receives the operation instruction from the uinput driver module, and executes the operation corresponding to the operation instruction in response to the operation instruction.

[0035] In some embodiments, the controlled device includes an input window operating system application program interface sendinputWinAPI module and an operating system module; the operating system is a windows operating system; the determining the operation instruction corresponding to the touch event, and executing the operation corresponding to the operation instruction in response to the operation instruction, specifically includes: determining the operation instruction corresponding to the touch event through the sendinput WinAPI module, and sending the operation instruction to the operating system module. The operating system module receives the operation instruction from the sendinput WinAPI module, and executes the operation corresponding to the operation instruction in response to the operation instruction.

[0036] In some embodiments, the control device includes a touch screen; the touch screen includes at least one sub-screen; one sub-screen corresponds to a type of controlled device; generating a touch event in response to a user's touch operation on the control device specifically includes: generating the touch event in response to a user's touch operation on the sub-screen of the control device.

[0037] In another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed on a processor, the processor executes one or more steps of the remote control method described in any of the above embodiments.

[0038] In another aspect, a computer program product is provided, wherein the computer program product comprises computer program instructions, and when the computer program instructions are executed on a computer, the computer program instructions cause the computer to execute one or more steps of the remote control method as described in any of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present disclosure, the following briefly introduces the drawings required to be used in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings. In addition, the drawings described below can be regarded as schematic diagrams, and are not limitations on the actual size of the product involved in the embodiments of the present disclosure, the actual process of the method, the actual timing of the signal, etc.

[0040] Figure 1 is one of the structural diagrams of a remote control system according to some embodiments;

[0041] Figure 2A is one of the structural diagrams of a controlled device according to some embodiments;

[0042] Figure 2B This is a second structural diagram of a controlled device according to some embodiments;

[0043] Figure 3A is one of the structural diagrams of a control device according to some embodiments;

[0044] Figure 3B The second structural diagram of the remote control system according to some embodiments;

[0045] Figure 3C This is a second structural diagram of a control device according to some embodiments;

[0046] Figure 4is one of the flow charts of the remote control method according to some embodiments;

[0047] Figure 5 This is a second flowchart of the remote control method according to some embodiments. DETAILED DESCRIPTION

[0048] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in some embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present disclosure.

[0049] Unless the context requires otherwise, throughout the specification and claims, the term "comprise" and other forms thereof, such as the third person singular form "comprises" and the present participle form "comprising", are to be interpreted as open, inclusive, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics described may be included in any one or more embodiments or examples in any appropriate manner.

[0050] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0051] When describing some embodiments, the expressions "coupled" and "connected" and their derivatives may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" or "communicatively coupled" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents of this document.

[0052] “At least one of A, B, and C” has the same meaning as “at least one of A, B, or C” and both include the following combinations of A, B, and C: A only, B only, C only, the combination of A and B, the combination of A and C, the combination of B and C, and the combination of A, B, and C.

[0053] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0054] The use of "adapted to" or "configured to" herein is meant to be open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps.

[0055] In addition, the use of “based on” or “according to” is meant to be open and inclusive, because the process, step, calculation or other action “based on” or “according to” one or more stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond the stated ones.

[0056] Exemplary embodiments are described herein with reference to cross-sectional views and / or plan views that are idealized exemplary drawings. In the drawings, the thickness of layers and regions are exaggerated for clarity. Therefore, variations in shape relative to the drawings due to, for example, manufacturing techniques and / or tolerances are conceivable. Therefore, the exemplary embodiments should not be interpreted as being limited to the shapes of the regions shown herein, but include shape deviations due to, for example, manufacturing. For example, an etched region shown as a rectangle will typically have curved features. Therefore, the regions shown in the drawings are schematic in nature, and their shapes are not intended to illustrate the actual shape of regions of the device, and are not intended to limit the scope of the exemplary embodiments.

[0057] With the popularization of electronic devices, when it is necessary to debug electronic devices in the same area (such as electronic shopping guide devices in shopping malls), it is necessary to debug each device one by one, resulting in repetitive work and low work efficiency. In addition, some types of electronic devices (such as large-screen electronic devices such as advertising machines, information screens, conference machines, etc.) are restricted by the usage scenarios and cannot perform corresponding touch-screen operations. It is necessary to use a mouse as a medium to control the electronic device to achieve human-computer interaction. For example, an ordinary mouse is connected to an electronic device wirelessly or wired for control in a human-computer interaction method, but the disadvantage of this method is that it is highly dependent on a mouse carrier (such as a mouse pad), which reduces the control experience of the electronic device to a certain extent. Therefore, there is an urgent need for a human-computer interaction solution that can get rid of the dependence on the mouse carrier and realize the simultaneous control of multiple electronic devices.

[0058] Based on the above technical background and problems, combined with Figure 1 , a remote control system provided by some embodiments of the present disclosure is described to solve the above technical problems.

[0059] The technical principle of the remote control system provided by the embodiment of the present disclosure is as follows: first, the remote control system mainly includes a control device and at least one type of controlled device. Among them, at least one type of controlled device includes a master device and at least one slave device, and the master device can provide MQTT services. The control device can provide split-screen operation in the touch screen, and each sub-screen after the split screen can correspond to a type of controlled device. When the user touches the control device, the control device can receive the user's touch operation, generate a touch event corresponding to the touch operation, and send the touch event to the slave device through the master device based on the MQTT protocol. The master device and the slave device can determine the operation instruction corresponding to the touch event, so that the master device and the slave device both perform corresponding operations according to the operation instruction. In this way, not only can the problem of the controlled device's high dependence on the mouse carrier be solved. Moreover, the use of the MQTT protocol for data transmission can ensure instant communication between the control device and the controlled device when the device's usage scenario is limited. In addition, the user can control different types of controlled devices in different sub-screens of the touch screen of the control device according to actual needs, which greatly improves the user's work efficiency.

[0060] Specifically, refer to Figure 1 The system includes a control device 10 and at least one type of controlled device 20. The controlled device 20 includes a master device 21 and at least one slave device 22.

[0061] In one implementation, the master device 21 and at least one slave device 22 included in each type of controlled device 20 are devices of the same type, such as an advertising machine, an information screen, or a conference machine.

[0062] Specifically, the control device 10 is configured to generate a touch event in response to a user's touch operation, and send the touch event to the main device 21 via the message queue telemetry transmission MQTT protocol; the touch event is used to characterize the click data and touch position generated by the user's touch on the control device 10.

[0063] Typically, the control device 10 may be a terminal device with a touch-sensitive surface; the terminal device may be a device with a wireless transceiver function. The terminal device may have different names, such as user equipment (UE), access device, terminal unit, terminal station, mobile station, mobile station, remote station, remote terminal, mobile device, wireless communication device, terminal agent or terminal device. The terminal device may be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it may also be deployed on the water surface (such as a ship); it may also be deployed in the air (such as an airplane, a balloon, and a satellite). The terminal device includes a handheld device, a vehicle-mounted device, a wearable device, or a computing device with a wireless communication function. For example, the terminal device may be a mobile phone, a tablet computer, or a computer with a wireless transceiver function. The terminal device may also be a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in unmanned driving, a wireless terminal in telemedicine, a wireless terminal in a smart grid, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. In the embodiment of the present application, the device for realizing the function of the control device 10 may be the control device 10, or may be a device capable of supporting the control device 10 to realize the function, such as a chip system, etc. In the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.

[0064] The controlled device 20 may be an electronic device with a screen display function, such as an electronic shopping guide device, an advertising machine, an information screen, a conference machine, an intelligent garbage classification device, etc.

[0065] In actual applications, the touch operation is an operation in which a user contacts the touch-sensitive surface of the touch screen of the control device 10 with his finger. Touch screens are currently divided into two main types: resistive screens and capacitive screens. Theoretically, for resistive screens, any object with a sufficiently large resistance can replace a finger to operate on its touch-sensitive surface; such as a stylus. For capacitive screens, any object that can form an electric field to bring about a change in current can replace a finger to operate on its touch-sensitive surface. For example, common substitutes include capacitive pens or capacitive sensing gloves. Therefore, the embodiments of the present disclosure do not impose any limitations on the specific implementation of the touch operation.

[0066] In some embodiments, touch operations generally include pressing, sliding and lifting, while mouse operations generally include sliding, left key and right key. Therefore, a mapping relationship is required for correspondence. For example, when a user needs to control the movement of the mouse pointer of the controlled device 20 through the control device 10, a series of operations such as pressing, sliding and lifting with one finger can be used to achieve the operation. The specific mapping relationship is shown in Table 1:

[0067] Table 1

[0068]

[0069] In Table 1, the press-and-release operation in the touch operation corresponds to the mouse click operation; the mouse click operation includes but is not limited to single-click, double-click, and long-press operations of the left or right button.

[0070] The master device 21 is configured to receive a touch event from the control device 10, determine an operation instruction corresponding to the touch event, and in response to the operation instruction, perform an operation corresponding to the operation instruction, and send the touch event to the slave device 22 via the MQTT protocol.

[0071] Furthermore, the touch event includes the first touch coordinates generated by the touch operation and the first data corresponding to the press and lift operation; after receiving the first touch coordinates of the control device 10, the master device 21 and the slave device 22 perform coordinate conversion on it to determine the second coordinates.

[0072] In one implementation, the operation instruction is used to instruct to perform touch simulation on a touch event, wherein both the master device 21 and the slave device 22 are specifically configured to respond to the operation instruction and display information corresponding to the operation instruction.

[0073] The operation instruction includes the second coordinate and the second data. It should be noted that when the operation instruction is used to instruct to perform touch simulation on a touch event, the second data is the first data.

[0074] Specifically, if the screen resolution of the control device 10 is (Wa, Ha), after receiving the screen resolution of the control device 10, the master device 21 and the slave device 22 calculate in combination with their own screen resolutions (Wb, Hb) to obtain the conversion resolution (Wd=Wb / Wa, Hd=Hb / Ha). When the master device 21 and the slave device 22 receive the first touch coordinates (Xa, Ya) of the control device 10, they perform coordinate conversion according to the coordinate conversion formula (Xb=Xa*Wd, Yb =Ya*Hd) in combination with the conversion resolution to obtain the second coordinates (Xb = Xa*Wb / Wa, Yb = Ya*Hb / Ha).

[0075] Optionally, the controlled device 20 is further configured to feed back the displayed information corresponding to the operation instruction to the control device 10. The control device 10 is configured to display the information fed back by the controlled device 20.

[0076] Exemplarily, the displayed information corresponding to the operation instruction may be an interface, a page, or a pop-up box, etc. Taking the master device 21 and the displayed information corresponding to the operation instruction as an interface as an example, after displaying the interface, the master device 21 takes a screenshot of the interface and sends the screenshot to the control device 10. Similarly, after displaying the interface, the slave device 22 takes a screenshot of the interface and sends the screenshot to the control device 10 through the master device 21. Afterwards, the control device 10 displays the screenshot for the user to view.

[0077] In the disclosed embodiment, the controlled device feeds back the displayed information to the control device, and the control device displays the information to the user so that the user can check the information and determine whether it is the operation he needs, thereby avoiding the problem of incorrect information display caused by accidental touch, and improving the user experience of human-computer interaction.

[0078] In another implementation, the operation instruction is used to instruct to perform mouse simulation on the touch event, wherein both the master device 21 and the slave device 22 are specifically configured to control the local mouse pointer to perform the mouse operation corresponding to the operation instruction in response to the operation instruction.

[0079] The operation instruction includes the second coordinate and the second data. It should be noted that when the operation instruction is used to instruct to perform mouse simulation on a touch event, the second data is the data of the mouse click operation corresponding to the press and lift operation.

[0080] Exemplary, combined Figure 1, since the coordinates generated by touch are generally absolute coordinates, and the coordinates generated by controlling the mouse are generally relative coordinates, it is necessary to convert the coordinate data. When the user touches the control device 10, the control device 10 generates a corresponding touch event based on the user's touch operation and notifies the main device 21 of the occurrence of the touch event. When the main device 21 successfully connects to the control device 10 through the MQTT protocol, it will save the absolute coordinate value O(x, y) sent by the control device 10 last time. After that, the relative coordinate value (dx1 = x1-x, dy1 = dy1-y) generated by the touch operation sent by the control device 10 is calculated based on the absolute coordinate value A(x1, y1) of this time and the absolute coordinate value O(x, y) of the last time. After that, the mouse pointer is controlled to move based on the relative coordinate value (dx1, dy1) of this time. And the absolute coordinate value A(x1, y1) of this time is replaced with the absolute coordinate value O(x, y) sent by the control device 10 last time, so as to be used for the next calculation. It should be noted that after the master device 21 successfully connects to the control device 10 through the MQTT protocol, before receiving the first touch event sent by the control device 10, the master device 21 will clear the absolute coordinate value O(x, y) previously saved in interaction with other devices.

[0081] Furthermore, since the control device 10 and the controlled device 20 are different types of devices in most cases, their resolution ratios may be different. When the resolutions are different, after obtaining the absolute coordinate value A(dx1, dy1), it is necessary to adjust the ratio. For example, the control device 10 sends its own resolution (W1, H1) to the master device 21, which is then forwarded by the master device 21 to the slave device 22. Both the master device 21 and the slave device 22 can calculate the ratio data (Wd = Wn / W1, Hd=Hn / H1) based on their own resolutions (Wn, Hn) and the control device 10's own resolutions (W1, H1). Thus, the mouse pointer displacement data is finally calculated as (Wd×dx1, Hd×dy1).

[0082] The slave device 22 is configured to receive a touch event from the master device 21, determine an operation instruction, and execute an operation corresponding to the operation instruction in response to the operation instruction.

[0083] It should be noted that after the slave device 22 is successfully connected to the master device 21 through the MQTT protocol, the relevant process of converting the coordinate data is the same as that of the master device 21, which will not be repeated here.

[0084] In one implementation, when the user controls the controlled device 20 through the control device 10 to perform related mouse operations, if the mouse pointer of the controlled device 20 is not in the same position, the mouse operations that the user needs to perform through the control device 10 to control the controlled device 20 may be deviated. Therefore, after the control device 10 and the master device 21, and the master device 21 and the slave device 22 establish a communication connection through the MQTT protocol, the master device 21 and the slave device 22 need to initialize the position of the local mouse pointer. Specifically, it includes:

[0085] The controlled device 20 is further configured to obtain preset coordinates, and control the local mouse pointer to move to a preset starting position based on the preset coordinates.

[0086] In some embodiments, an Android device is used as an example for explanation. Because when an Android device performs a mouse operation, only the first mouse operation after startup starts from the middle initial position, and the subsequent operations will be relatively offset from the previous position and will not start from the initial position again. Therefore, when one of the controlled devices 20 has been controlled by the mouse before this remote control, the subsequent position will not be unified with other controlled devices 20, resulting in an operation coordination error. Therefore, the controlled device 20 needs to synchronize the initial position (i.e., initialize the position of the local mouse pointer) when the remote coordination is turned on after the control device 10 and the master device 21, the master device 21 and the slave device 22 establish a communication connection through the MQTT protocol. Since the mouse pointer moves relative to the screen coordinates, if its offset value exceeds the screen coordinate range after superposition, it will be moved to the extreme position. Therefore, after obtaining the remote coordination function, the controlled device 20 can send the (dx, dy) coordinates larger than the screen size to the operating system by calling the input uinput driver interface, so that the operating system controls the local mouse pointer to automatically move to the preset starting position (such as the lower right corner) to achieve unified coordination of the controlled device 20.

[0087] In this implementation, the local mouse pointer is controlled to move to a preset starting position based on preset coordinates, so that unified coordination of the controlled devices can be achieved.

[0088] In the disclosed embodiment, the control device generates a touch event based on the user's touch operation. And the touch event is sent to the slave device through the master device based on the MQTT protocol. The master device and the slave device can determine the operation instruction corresponding to the touch event, so that the master device and the slave device perform the corresponding operation according to the operation instruction. So that the user can control the device through the touch operation. Not only can the function of controlling multiple controlled devices at the same time be realized, but also the user's work efficiency can be improved. It also enables the controlled device to get rid of the dependence on the mouse carrier, thereby providing a better human-computer interaction experience for the controlled device that cannot perform the corresponding touch screen operation. In addition, considering that not all control devices can provide the MQTT service function. Therefore, a device can be arbitrarily selected from the controlled devices as the device that provides the MQTT service function in the MQTT protocol as the master device. It is only necessary to deploy the function of providing the MQTT service in the MQTT protocol in the master device, and send the touch event generated by the control device to the slave device through the forwarding operation of the master device, so as to realize the data forwarding operation. In this way, not only the scope of application of the control device is greatly expanded, but also the devices that can provide the touch screen function and can use the MQTT protocol can be used as control devices. Moreover, selecting one device from the controlled devices as the device providing the MQTT service function in the MQTT protocol can avoid the problem of resource waste caused by selecting another device dedicated to providing the MQTT service function.

[0089] In one implementation, the user can select the controlled device 20 (including the master device 21 and the slave device 22) to be controlled by operating the control device 10. Therefore, the control device 10 is further configured to: obtain and display the identification information of at least one candidate device. In response to the user's first selection operation on at least one candidate device, the master device 21 is determined. In response to the user's second selection operation on other candidate devices, at least one slave device 22 is determined; the other candidate devices include at least one candidate device that does not include the master device 21.

[0090] Exemplarily, the control device 10 first broadcasts its own IP in the global area network through the broadcast method of the user datagram protocol (UDP); after receiving the broadcast message, the alternative device takes out its IP address and sends a UDP message containing its own IP address and / or device ID and other identification information to the control device 10; the control device 10 displays the identification information of the IP address and / or device ID in the received UDP message in the device list. The user can determine the master device 21 by performing a first selection operation on the alternative device displayed in the device list. Perform a second selection operation to determine the slave device 22. It should be noted that the embodiment of the present disclosure does not limit the order of the first selection operation and the second selection operation.

[0091] In this implementation, the control device 10 obtains and displays identification information of multiple candidate devices for the user to select and determine the master device 21 and the slave device 22, so as to provide the user with a better human-computer interaction experience.

[0092] Furthermore, the user does not need to select the master device 21 and the slave device 22 through the control device 10 , and the control device 10 directly displays the identification information of the master device 21 and the slave device 22 to the user.

[0093] In one implementation, before the control device 10 sends a touch event to the master device 21, it is necessary to establish a communication connection through the MQTT protocol, specifically including: the control device 10 is also configured to send a startup instruction to the master device 21. The master device 21 is configured to receive the startup instruction from the control device 10, and start the MQTT service according to the startup instruction, and subscribe to the coordinate message, and generate a response message and send it to the control device 10; the response information is used to indicate that the master device 21 successfully starts the MQTT service; the subscribed coordinate message is used to obtain the touch event. The control device 10 is configured to receive the response information from the master device 21, and establish a communication connection with the master device 21 through the MQTT protocol according to the response information. The control device 10 is also configured to send a connection instruction to the slave device 22 according to the response information. The slave device 22 is configured to receive the connection instruction from the control device 10, and establish a communication connection with the master device 21 through the MQTT protocol according to the connection instruction, and subscribe to the coordinate message from the master device 21.

[0094] It should be noted that the MQTT protocol includes multiple client terminals (the terminals can also be called modules) and a server terminal (message broker). Generally, the client terminal is deployed in the terminal device and is responsible for subscribing to messages; while the server terminal is deployed in the server and is responsible for forwarding messages. Since the collaborative operation between devices requires low latency and low cost, if the server deployment method is adopted, the data transmission and reception delay and cost will be greatly increased. Therefore, considering the cost reduction, the embodiment of the present disclosure adopts the deployment of the server terminal in a certain master device 21. Since the controlled devices 20 belong to the same functional devices, they have the same system environment, and a certain controlled device 20 can be randomly selected to be designated as the server terminal device of the MQTT protocol (i.e., the master device 21). At the same time, the system of the master device 21 will also start the client process to connect to the server terminal of the local machine. Therefore, any controlled device 20 connected to the server terminal can obtain the coordinate data sent by the device 1.

[0095] Specifically, after the user determines the main device 21 in the device list, the control device 10 sends a startup instruction to the main device 21 to start the MQTT service, and the main device 21 starts the corresponding MQTT service according to the startup instruction. At this time, the main device 21 can provide a message forwarding function as a server-side device. Considering that the main device 21, as one of the controlled devices 20, also needs to generate mouse instructions according to touch events and perform operations according to mouse instructions. Therefore, the main device 21 is a device on the client side while serving as a server-side device. That is, the client side of the main device 21 and the server side establish a communication connection through the MQTT protocol, so as to generate mouse instructions according to touch events and perform operations according to mouse instructions.

[0096] In addition, after the master device 21 successfully starts the MQTT service, it will return a response message to the control device 10, notifying the control device 10 that the MOTT service has been successfully started. After that, the control device 10 sends a connection instruction to the slave device 22 selected by the user to establish a communication connection with the master device 21 through the MQTT protocol. The slave device 22 establishes a communication connection between its own client and the server of the master device 21 through the MQTT protocol; and subscribes to coordinate messages from the master device 21 through the MQTT protocol.

[0097] In addition, when the control device 10 is a system-customizable device, the server of the MQTT protocol can be deployed in the control device 10, so that all controlled devices 20 are connected to the server of the control device 10 to perform MQTT communication.

[0098] In this implementation, the control device and the master device, and the master device and the slave device are connected through the MQTT protocol, and coordinate messages are subscribed to, thereby realizing the sending and receiving of touch events. By using the MQTT protocol, the information transmission efficiency is higher, and the communication connection between the touch devices can be quickly established, thereby saving the user's time cost.

[0099] Furthermore, in order to monitor whether the master device 21 successfully receives the touch event sent by the control device 10 , the control device 10 is further configured to subscribe to the coordinate message from the master device 21 .

[0100] In this way, by subscribing the coordinate message from the control device 10 to the master device 21, when the control device 10 sends a touch event to the master device 21, the master device 21 will return the touch event to the control device 10 again, so that it can be determined that the control device 10 has successfully sent the touch event to the master device 21. In this way, the master device 21 can better monitor the situation of receiving the information sent by the control device 10, so as to take timely countermeasures.

[0101] In another implementation, considering that different operating systems have different ways of processing touch events, the present disclosure provides a method for the controlled device 20 to process touch events for three operating systems: Linux, Android, and Windows. Figure 2A When the operating system of the controlled device 20 is a Linux operating system or an Android operating system, the controlled device 20 includes an input uinput driver module 221 and an operating system module 221-1. Figure 2B When the operating system of the controlled device 20 is a Windows operating system, the controlled device 20 includes an input Windows operating system application program interface sendinput WinAPI module 222 and an operating system module 222-1.

[0102] Specifically, when the operating system of the controlled device 20 is a Linux operating system or an Android operating system, the uinput driver module 221 is configured to determine an operation instruction corresponding to a touch event, and send the operation instruction to the operating system module 221-1. The operating system module 221-1 is configured to receive the operation instruction from the uinput driver module 221, and in response to the operation instruction, execute the operation corresponding to the operation instruction.

[0103] When the operating system of the controlled device 20 is a Windows operating system, the sendinput WinAPI module 222 is configured to determine an operation instruction corresponding to the touch event and send the operation instruction to the operating system module 222-1. The operating system module 222-1 is configured to receive the operation instruction from the sendinput WinAPI module 222, and in response to the operation instruction, perform the operation corresponding to the operation instruction.

[0104] In this implementation, when the operating system is a Linux operating system or an Android operating system, the input uinput driver module of the controlled device determines the operation instruction corresponding to the touch event and sends the touch event to the operating system module. When the operating system is a Windows operating system, the controlled device determines the operation instruction corresponding to the touch event by calling the sendinput WinAPI module and sends the touch event to the operating system module. In this way, the operating system module can determine the operation instruction based on the touch event under different operating systems and execute the operation corresponding to the operation instruction.

[0105] Reference Figure 3A, the disclosed embodiment takes into account the situation that there is more than one type of controlled device 20 that needs to be controlled. In order to improve the work efficiency of the user, the touch screen 11 of the control device 10 is split to obtain at least one sub-screen 111. In this way, the user can control the corresponding type of controlled device 20 to perform a corresponding operation through a touch operation in the sub-screen 111 at the same time. Therefore, the control device 10 includes a touch screen 11; the touch screen 11 includes at least one sub-screen 111; one sub-screen 111 corresponds to a type of controlled device 20; wherein the control device 10 is specifically configured to generate a touch event in response to the user's touch operation in the sub-screen 111.

[0106] For a better understanding, combine Figure 3B , the split screen situation is exemplarily described. The user can trigger the split screen control 11a on the touch screen 11 of the control device 10 to split the touch screen 11 of the control device 10, so as to obtain at least one sub-screen 111 (such as Figure 3B 1111, 1112, 1113, and 1114 are shown in FIG. 111 ); each sub-screen 111 corresponds to controlling a type of controlled device 20 (including a master device 21 and a slave device 22). Figure 3B The sub-screen 1111 shown in the figure corresponds to controlling the controlled device 21a and at least one slave device 22a. Similarly, the sub-screen 1112 corresponds to controlling the controlled device 21b and at least one slave device 22b; the sub-screen 1113 corresponds to controlling the controlled device 21c and at least one slave device 22c; the sub-screen 1114 corresponds to controlling the controlled device 21d and at least one slave device 22d. It should be noted that each sub-screen 111 controls the corresponding type of controlled device 20 to achieve the corresponding operation, which can be referred to above and will not be repeated here.

[0107] In one example, combining Figure 3A , the user can obtain at least one sub-screen 111 by triggering a split-screen control 11a for splitting the touch screen 11 of the control device 10; in one implementation, after the user triggers the split-screen control 11a, a drop-down menu may appear, and the menu displays the number of selectable sub-screens. After the user triggers the control corresponding to the number of sub-screens, the sub-screens corresponding to the number of sub-screens are displayed on the touch screen. In another implementation, the number of sub-screens is a default number; when the user triggers the split-screen control 11a, the default sub-screen (such as 2 split screens or 4 split screens. 2 split screens correspond to 2 sub-screens; 4 split screens correspond to 4 sub-screens) is directly displayed.

[0108] Further, when the number of sub-screens 111 is greater than the type of controlled devices, the redundant sub-screens 111 may display the following information: Figure 3C Of course, other information may also be displayed, such as displaying black in the redundant sub-screen 111.

[0109] In one possible implementation, the user can adjust the size of the sub-screen arbitrarily within a preset size range. For example, the user touches the touch screen 11 according to a preset gesture to adjust the size, such as inserting two fingers into the touch screen 11 and sliding in opposite directions. The larger the distance between the two fingers, the larger the size of the sub-screen; the smaller the distance between the two fingers, the smaller the size of the sub-screen. For another example, a default size control is provided. After the user triggers the size control, at least one size is displayed through a drop-down menu interface for the user to select. The disclosed embodiments do not impose any limitation on the method of adjusting the size of the sub-screen.

[0110] The disclosed embodiments can provide at least one sub-screen in the touch screen of the control device so that multiple types of controlled devices can be controlled through the multiple sub-screens, thereby improving the work efficiency of the user.

[0111] In addition, combined Figure 3B , taking the four-split screen as an example, the coordinate conversion method is explained. Among them, the four sub-screens can be divided into lower left, upper left, lower right, and upper right according to their relative positions. Specifically, each sub-screen 11 corresponds to controlling a type of controlled device 21 (including master device 21 and slave device 22). If the screen resolution of the control device 10 is (Wa, Ha), the size range values ​​of the four sub-screens 11 are lower left (0-Wa / 2, 0-Ha / 2), lower right (Wa / 2-Wa, 0-Ha / 2), upper left (0-Wa / 2, Ha / 2-Ha), and upper right (Wa / 2-Wa, Ha / 2-Ha). Therefore, the coordinate values ​​sent by the control device 10 under the four-split screen are lower left (Xa, Ya), lower right (Xa-Wa / 2, Ya), upper left (Xa, Ya-Ha / 2), and upper right (Xa-Wa / 2, Ya-Ha / 2). After receiving the screen resolution of the sub-screen 11 of the control device 10, the master device 21 and the slave device 22 calculate the conversion resolution (Wd = Wb / Wa, Hd=Hb / Ha) in combination with their own screen resolutions. When the master device 21 and the slave device 22 receive the first touch coordinates (Xa, Ya) of the control device 10, if it is the lower left sub-screen 11, combined with the conversion resolution, the coordinate conversion is performed according to the coordinate conversion formula (Xb=Xa*Wd, Yb=Ya*Hd) to obtain the second coordinates (Xb=Xa*Wb / Wa, Yb=Ya*Hb / Ha). Similarly, if it is the lower right sub-screen 11, combined with the conversion resolution, the coordinate conversion is performed according to the coordinate conversion formula (Xb=Xa*Wd, Yb=Ya*Hd) to obtain the second coordinates (Xb=(Xa-Wa / 2)*Wb / Wa, Yb=(0-Ha / 2)*Hb / Ha). For other sub-screens, please refer to the above and will not be repeated here.

[0112] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of the system. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed in this article, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0113] Here, the system architecture described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0114] Reference Figure 4 Some embodiments of the present disclosure provide a remote control method, which is applied to the above-mentioned remote control system, and the remote control system includes: a control device and at least one type of controlled device; the controlled device includes a master device and a slave device.

[0115] S31. In response to a user's touch operation on a control device, a touch event is generated, and the touch event is sent to a master device via a message queue telemetry transmission MQTT protocol; the touch event is used to characterize click data and a touch position generated by the user's touch on the control device.

[0116] In one implementation, the control device includes a touch screen; the touch screen includes at least one sub-screen; one sub-screen corresponds to a type of controlled device; in response to a user's touch operation on the control device, a touch event is generated, specifically including: in response to a user's touch operation in the sub-screen of the control device, a touch event is generated.

[0117] S32, receiving the touch event from the control device through the master device, determining the operation instruction corresponding to the touch event, and executing the operation corresponding to the operation instruction in response to the operation instruction, and sending the touch event to the slave device through the MQTT protocol.

[0118] Optionally, the operation instruction is used to instruct touch simulation of a touch event; in response to the operation instruction, an operation corresponding to the operation instruction is executed, including: in response to the operation instruction, information corresponding to the operation instruction is displayed.

[0119] Furthermore, after displaying the information corresponding to the operation instruction, the method further includes: feeding back the displayed information corresponding to the operation instruction to the control device through the controlled device. The information fed back by the controlled device is displayed through the control device.

[0120] Optionally, the operation instruction is used to instruct mouse simulation of a touch event, and in response to the operation instruction, an operation corresponding to the operation instruction is performed, including: in response to the operation instruction, controlling a local mouse pointer to perform a mouse operation corresponding to the operation instruction.

[0121] Furthermore, in response to a user's touch operation on the control device, before generating a touch event, the method further includes: acquiring preset coordinates through the controlled device, and controlling the local mouse pointer to move to a preset starting position based on the preset coordinates.

[0122] S33: receiving the touch event of the master device through the slave device, determining an operation instruction, and executing an operation corresponding to the operation instruction in response to the operation instruction.

[0123] Specifically, determining the operation instruction corresponding to the touch event, and executing the operation corresponding to the operation instruction in response to the operation instruction, specifically includes the following two methods:

[0124] Method 1: When the controlled device includes an input uinput driver module and an operating system module; and the operating system is a Linux operating system or an Android operating system, the uinput driver module determines the operation instruction corresponding to the touch event, and sends the operation instruction to the operating system module. The operating system module receives the operation instruction from the uinput driver module, and in response to the operation instruction, performs the operation corresponding to the operation instruction.

[0125] Method 2: When the controlled device includes a sendinput WinAPI module and an operating system module, and the operating system is a Windows operating system, the sendinput WinAPI module determines the operation instruction corresponding to the touch event, and sends the operation instruction to the operating system module. The operating system module receives the operation instruction from the sendinput WinAPI module, and in response to the operation instruction, executes the operation corresponding to the operation instruction.

[0126] Optional, combined Figure 4 , refer to Figure 5 , before S31, it also includes:

[0127] S301. Obtain and display identification information of at least one alternative device by a control device, determine a master device in response to a user's first selection operation on at least one alternative device, and determine at least one slave device in response to a user's second selection operation on other alternative devices; the other alternative devices include at least one alternative device that does not include the master device among the slave devices.

[0128] Optional, see Figure 5 , before S31, it also includes:

[0129] S302: Send a start instruction of the control device to the main device through the control device.

[0130] S303, receiving the startup instruction from the control device through the main device, starting the MQTT service according to the startup instruction, subscribing to the coordinate message, and generating a response message and sending it to the control device; the response information is used to indicate that the main device successfully starts the MQTT service; the subscribed coordinate message is used to obtain the touch event.

[0131] S304, receiving the response information of the master device through the control device, and establishing a communication connection with the master device through the MQTT protocol according to the response information.

[0132] S305: Enable the control device to send a connection instruction to the slave device according to the response information.

[0133] S306, receiving the connection instruction from the control device through the slave device, establishing a communication connection with the master device through the MQTT protocol according to the connection instruction, and subscribing to the coordinate message from the master device.

[0134] Optional, see Figure 5 , S304 and later also include:

[0135] S3041. Send a subscription coordinate message of the control device to the master device.

[0136] It should be noted that the exemplary description and beneficial effects corresponding to the above method can all refer to the above calibration verification system, which will not be described in detail here.

[0137] It should be pointed out that the various embodiments of the present disclosure may learn from or refer to each other, for example, the same or similar steps, method embodiments, and system embodiments may refer to each other without limitation.

[0138] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) having computer program instructions stored therein. When the computer program instructions are executed on a processor, the processor executes one or more steps of the remote control method as in any of the above-mentioned embodiments.

[0139] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0140] Some embodiments of the present disclosure further provide a computer program product. The computer program product includes computer program instructions, and when the computer program instructions are executed on a computer, the computer program instructions cause the computer to execute one or more steps in the remote control method described in the above embodiments.

[0141] Some embodiments of the present disclosure further provide a computer program. When the computer program is executed on a computer, the computer program enables the computer to execute one or more steps in the method described in the above embodiments.

[0142] The beneficial effects of the above-mentioned computer-readable storage medium, computer program product and computer program are the same as the beneficial effects of the remote control method described in some of the above-mentioned embodiments, and will not be repeated here.

[0143] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be thought of by any person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A remote control system, comprising: A controlling device and at least one type of controlled device; The controlled device includes a master device and at least one slave device; The control device is configured to generate a touch event in response to a touch operation of a user, and send the touch event to the master device via a message queue telemetry transmission MQTT protocol; The touch event is used to represent click data and a touch position generated by the user touching the control device; The master device is configured to receive the touch event from the control device, determine an operation instruction corresponding to the touch event, and in response to the operation instruction, control a local mouse of the master device to perform a mouse operation corresponding to the operation instruction, and send the touch event to the slave device through the MQTT protocol; The slave device is configured to receive the touch event from the master device, determine the operation instruction, and in response to the operation instruction, control the local mouse of the slave device to perform the mouse operation corresponding to the operation instruction; wherein the pointer displacement data of the local mouse of the controlled device is calculated based on the resolution of the controlled device and the resolution of the control device.

2. The system according to claim 1, wherein the operation instruction is used to instruct to perform touch simulation on the touch event, wherein: The master device and the slave device are both specifically configured to display information corresponding to the operation instruction in response to the operation instruction.

3. The system according to claim 2, wherein: The controlled device is further configured to feed back the displayed information corresponding to the operation instruction to the control device; The control device is configured to display the information fed back by the controlled device.

4. The system according to claim 1, wherein: The controlled device is further configured to obtain preset coordinates, and control the local mouse pointer to move to a preset starting position based on the preset coordinates.

5. According to the system according to any one of claims 1 to 4, the control device is further configured to: Obtaining and displaying identification information of at least one candidate device; In response to a user's first selection operation on the at least one candidate device, the master device is determined; in response to a user's second selection operation on other candidate devices, at least one slave device is determined; the other candidate devices include candidate devices among the at least one candidate device that do not include the master device.

6. The system according to any one of claims 1 to 5, wherein: The control device is further configured to send a start instruction to the main device; The master device is configured to receive the start instruction from the control device, start the MQTT service according to the start instruction, subscribe to the coordinate message, and generate a response message to send to the control device; The response information is used to indicate that the master device successfully starts the MQTT service; The subscription coordinate message is used to obtain the touch event; The control device is configured to receive response information from the master device, and establish a communication connection with the master device through the MQTT protocol according to the response information; The control device is further configured to send a connection instruction to the slave device according to the response information; The slave device is configured to receive the connection instruction from the control device, establish a communication connection with the master device through the MQTT protocol according to the connection instruction, and subscribe to coordinate messages from the master device.

7. The system according to claim 6, wherein: The control device is further configured to subscribe to coordinate messages from the master device.

8. The system according to any one of claims 1 to 7, wherein the controlled device comprises an input uinput driver module and an operating system module; the operating system is a Linux operating system or an Android operating system; wherein, The uinput driver module is configured to determine an operation instruction corresponding to the touch event, and send the operation instruction to the operating system module; The operating system module is configured to receive the operation instruction from the uinput driver module, and in response to the operation instruction, execute the operation corresponding to the operation instruction.

9. The system according to any one of claims 1 to 7, wherein the controlled device comprises an input Windows operating system application program interface sendinput WinAPI module and an operating system module; the operating system is a Windows operating system; wherein, The sendinput WinAPI module is configured to determine an operation instruction corresponding to the touch event and send the operation instruction to the operating system module; The operating system module is configured to receive the operation instruction from the sendinput WinAPI module, and execute the operation corresponding to the operation instruction in response to the operation instruction.

10. The system according to any one of claims 1 to 9, wherein the control device comprises a touch screen; the touch screen comprises at least one sub-screen; one sub-screen corresponds to one type of controlled device; wherein, The control device is specifically configured to generate the touch event in response to a user's touch operation on the sub-screen.

11. A remote control method, applied to a remote control system; The remote control system comprises: a control device and at least one type of controlled device; the controlled device comprises a master device and a slave device; In response to a user's touch operation on the control device, a touch event is generated, and the touch event is sent to the master device through a message queue telemetry transmission MQTT protocol; the touch event is used to characterize click data and a touch position generated by the user's touch on the control device; The touch event from the control device is received by the master device, and an operation instruction corresponding to the touch event is determined, and in response to the operation instruction, a local mouse of the master device is controlled to perform a mouse operation corresponding to the operation instruction, and the touch event is sent to the slave device through the MQTT protocol; The touch event from the master device is received by the slave device, and the operation instruction is determined, and in response to the operation instruction, the local mouse of the slave device is controlled to perform the mouse operation corresponding to the operation instruction; wherein the pointer displacement data of the local mouse of the controlled device is calculated based on the resolution of the controlled device and the resolution of the control device.

12. The method according to claim 11, wherein the operation instruction is used to instruct to perform touch simulation on the touch event; and the step of responding to the operation instruction and executing the operation corresponding to the operation instruction comprises: In response to the operation instruction, information corresponding to the operation instruction is displayed.

13. The method according to claim 12, after displaying the information corresponding to the operation instruction, the method further comprises: Feedback displayed information corresponding to the operation instruction to the control device through the controlled device; The information fed back by the controlled device is displayed through the control device.

14. The method according to claim 11, before generating a touch event in response to a user's touch operation on the control device, the method further comprises: The preset coordinates are acquired through the controlled device, and the local mouse pointer is controlled to move to a preset starting position based on the preset coordinates.

15. The method according to any one of claims 11 to 14, before generating a touch event in response to a user's touch operation on the control device, the method further comprises: The identification information of at least one slave device is acquired and displayed by the control device, the master device is determined in response to the user's first selection operation on the at least one slave device, and at least one slave device is determined in response to the user's second selection operation on other alternative devices; the other alternative devices include alternative devices among the at least one slave device that do not include the master device.

16. The method according to any one of claims 11 to 15, before generating a touch event in response to a user's touch operation on the control device, the method comprises: Sending a start instruction of the control device to the main device through the control device; The start instruction from the control device is received by the master device, and the MQTT service is started according to the start instruction, and the coordinate message is subscribed, and response information is generated and sent to the control device; The response information is used to indicate that the master device successfully starts the MQTT service; The subscription coordinate message is used to obtain the touch event; The control device receives the response information from the master device, and establishes a communication connection with the master device through the MQTT protocol according to the response information; enabling the control device to send a connection instruction to the slave device according to the response information; The connection instruction from the control device is received by the slave device, and a communication connection is established with the master device through the MQTT protocol according to the connection instruction, and a coordinate message is subscribed to the master device.

17. The method according to claim 16, after receiving the response information from the master device through the control device and establishing a communication connection with the master device through the MQTT protocol according to the response information, the method further comprises: Sending a subscription coordinate message of the control device to the master device.

18. The method according to any one of claims 11-17, wherein the controlled device comprises an input uinput driver module and an operating system module; the operating system is a Linux operating system or an Android operating system; The determining the operation instruction corresponding to the touch event, and executing the operation corresponding to the operation instruction in response to the operation instruction, specifically includes: Determine an operation instruction corresponding to the touch event through the uinput driver module, and send the operation instruction to the operating system module; The operating system module receives the operating instruction from the uinput driver module, and in response to the operating instruction, executes the operation corresponding to the operating instruction.

19. The method according to any one of claims 11 to 18, wherein the controlled device comprises an input Windows operating system application program interface sendinput WinAPI module and an operating system module; the operating system is a Windows operating system; The determining the operation instruction corresponding to the touch event, and executing the operation corresponding to the operation instruction in response to the operation instruction, specifically includes: Determine an operation instruction corresponding to the touch event through the sendinput WinAPI module, and send the operation instruction to the operating system module; The operating instruction from the sendinput WinAPI module is received through the operating system module, and in response to the operating instruction, an operation corresponding to the operating instruction is executed.

20. The method according to any one of claims 11 to 19, wherein the control device comprises a touch screen; the touch screen comprises at least one sub-screen; one sub-screen corresponds to one type of controlled device; and the generating of a touch event in response to a user's touch operation on the control device comprises: The touch event is generated in response to a user's touch operation on the sub-screen of the control device.

21. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed on a computer, the computer is enabled to execute the remote control method according to any one of claims 11 to 20.

Images