A remote control method and related device

Abstract

Embodiments of the present application provide a remote control method and related devices, and relate to the technical field of computers. The remote control method comprises: a control device receiving and displaying content data of a controlled window sent by a controlled device, the controlled window being a program window running in the background of the controlled device. The control device receives a user operation on the controlled window and determines a control instruction according to the user operation. The control device sends the control instruction to the controlled device, so that the controlled device performs a control operation on the controlled window according to the control instruction. In the embodiments of the present application, the controlled window does not need to be displayed on the front screen. The controlled device can perform other operations in the foreground when being remotely controlled, thereby improving the running efficiency of the controlled device.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technical field, and in particular, to a remote control method and related devices. BACKGROUND

[0002] With the development of mobile Internet, remote control technology is applied more and more widely in enterprise office. Remote control technology is a technology that a network control device remotely controls another device. In the prior art, remote education, remote maintenance, remote assistance, and device remote control are typical application scenarios of remote control.

[0003] Remote control is mainly realized through remote control software. Specifically, a controlled device intercepts the content on its screen and sends the content to a control device. The control device responds to the intercepted content, determines and sends a control instruction to the controlled device. Finally, the controlled device executes relevant operations on the screen according to the control instruction, and completes the entire remote control process.

[0004] In the above process, since the controlled device intercepts the screen content displayed in the foreground, the control device must control the controlled window in the foreground. Moreover, the controlled device must complete the control operation on the controlled window in the foreground screen. This will cause the controlled device to only execute the control operation when being remotely controlled, and cannot execute other user operations. This will seriously affect the running efficiency of the controlled device. SUMMARY

[0005] Therefore, the present application provides a remote control method and related devices. The control device determines a control instruction by acquiring the content of a controlled window running in the background, and then sends the control instruction to the controlled device, so that the controlled device executes the operation corresponding to the control instruction. In this way, the controlled window does not need to be displayed on the foreground screen. The controlled device can also execute other operations in the foreground when being remotely controlled.

[0006] The first aspect of the embodiment of the present application provides a remote control method, which includes:

[0007] The control device receives the content data of the controlled window sent by the controlled device, and the controlled window is a program window running in the background of the controlled device.

[0008] The control device receives a control operation for the controlled window, and determines a control instruction according to the control operation.

[0009] The control device sends the control instruction to the controlled device, so that the controlled device executes the control operation for the controlled window according to the control instruction.

[0010] In an optional implementation, the method further includes:

[0011] The control device receives an identification of the controlled window sent by the controlled device.

[0012] The control device sends a window capture request to the controlled device according to the identification of the controlled window, the window capture request being used to instruct the controlled device to capture a screenshot of content data of the controlled window in the background.

[0013] In an optional implementation, the control device sends the control instruction to the controlled device, including:

[0014] The control device displays the content data of the controlled window in the foreground.

[0015] The control device receives a control operation for the controlled window and determines the control instruction according to the control operation, including:

[0016] The control device receives a user control operation for the controlled window in the foreground and determines the control instruction according to the user control operation.

[0017] The second aspect of the embodiments of the present application provides another remote control method, including:

[0018] The controlled device determines the controlled window in at least one window running in the background.

[0019] The controlled device captures a screenshot of the controlled window in the background to obtain content data of the controlled window.

[0020] The controlled device sends the content data of the controlled window to the control device.

[0021] The controlled device receives a control instruction sent by the control device, the control instruction corresponding to the content data of the controlled window.

[0022] The controlled device performs a control operation for the controlled window according to the control instruction.

[0023] In an optional implementation, the method further includes:

[0024] The controlled device sends an identification of the controlled window to the control device.

[0025] The controlled device receives a window capture request sent by the control device, the window capture request corresponding to the identification of the controlled window.

[0026] The controlled device captures a screenshot of the controlled window in the background, including:

[0027] The controlled device captures a screenshot of the controlled window in the background according to the window capture request.

[0028] In an optional implementation, the controlled device sends the content data of the controlled window to the control device, including:

[0029] The controlled device compressively encodes content data of the controlled window.

[0030] The controlled device sends the compressively encoded content data of the controlled window to the control device.

[0031] In an optional implementation, the controlled device receives the control instruction sent by the control device, including:

[0032] The controlled device sends the control instruction to the controlled window, and the controlled window includes a message interface.

[0033] The controlled device executes the control operation for the controlled window according to the control instruction, including:

[0034] The controlled device executes the control operation for the controlled window in the background according to the control instruction.

[0035] In an optional implementation, the controlled device receives the control instruction sent by the control device, including:

[0036] The controlled device sends the control instruction to a task queue running in the foreground.

[0037] The controlled device executes the control operation for the controlled window according to the control instruction, including:

[0038] The controlled device controls the controlled window to run in the foreground.

[0039] The controlled device executes the control operation for the controlled window in the foreground according to the control instruction.

[0040] In an optional implementation, the method further includes:

[0041] The controlled device queries a task queue running in the foreground to obtain all control instructions corresponding to the controlled window.

[0042] The controlled device executes the control operation for the controlled window in the foreground according to the control instruction, including:

[0043] The controlled device executes the control operation corresponding to each control instruction in all control instructions in the foreground in sequence.

[0044] In an optional implementation, after the controlled device executes the control operation for the controlled window in the foreground, the method further includes:

[0045] The controlled device switches the controlled window to the background and controls the controlled window to run in the background.

[0046] The third aspect of the embodiments of the present application provides a control device, including:

[0047] The receiving unit is configured to receive content data of a controlled window sent by the controlled device, the controlled window being a program window running in the background of the controlled device.

[0048] The receiving unit is further configured to receive a control operation for the controlled window.

[0049] The determining unit is configured to determine a control instruction according to the control operation.

[0050] The sending unit is configured to send the control instruction to the controlled device, so that the controlled device performs the control operation for the controlled window according to the control instruction.

[0051] In an optional implementation, the receiving unit is further configured to receive an identifier of the controlled window sent by the controlled device.

[0052] The sending unit is further configured to send a window screenshot request to the controlled device according to the identifier of the controlled window, the window screenshot request being used to instruct the controlled device to take a screenshot of the content data of the controlled window in the background.

[0053] In an optional implementation,

[0054] The receiving unit is further configured to display the content data of the controlled window in the foreground. The user control operation for the controlled window is received in the foreground.

[0055] The determining unit is specifically configured to determine the control instruction according to the user control operation.

[0056] The fourth aspect of the embodiments of the present application provides a controlled device, which comprises:

[0057] The determining unit is configured to determine the controlled window from at least one window running in the background.

[0058] The processing unit is configured to take a screenshot of the controlled window in the background to obtain the content data of the controlled window.

[0059] The sending unit is configured to send the content data of the controlled window to the control device.

[0060] The receiving unit is configured to receive a control instruction sent by the control device, the control instruction corresponding to the content data of the controlled window.

[0061] The executing unit is configured to perform a control operation for the controlled window according to the control instruction.

[0062] In an optional implementation, the sending unit is further configured to send an identifier of the controlled window to the control device.

[0063] The receiving unit is further configured to receive a window screenshot request sent by the control device, the window screenshot request corresponding to the identifier of the controlled window.

[0064] The processing unit is specifically configured to perform screen capturing on the controlled window in the background according to the window capturing request.

[0065] In an optional implementation, the sending unit is specifically configured to perform compression encoding processing on the content data of the controlled window. The content data of the controlled window after the compression encoding processing is sent to the control device.

[0066] In an optional implementation, the receiving unit is specifically configured to send the control instruction to the controlled window, and the controlled window comprises a message interface.

[0067] The execution unit is specifically configured to perform the control operation on the controlled window in the background according to the control instruction.

[0068] In an optional implementation, the receiving unit is specifically configured to send the control instruction to the task queue running in the foreground.

[0069] The execution unit is specifically configured to control the controlled window to run in the foreground. The execution unit is specifically configured to perform the control operation on the controlled window in the foreground according to the control instruction.

[0070] In an optional implementation, the controlled device further comprises an acquisition unit.

[0071] The acquisition unit is specifically configured to query the task queue running in the foreground, and acquire all control instructions corresponding to the controlled window.

[0072] The execution unit is specifically configured to sequentially perform the control operation corresponding to each control instruction in all control instructions in the foreground.

[0073] In an optional implementation, the execution unit is further configured to switch the controlled window to the background, and control the controlled window to run in the background.

[0074] The fifth aspect of the embodiment of the present application further provides a control device, comprising a memory and a processor, the memory and the processor being coupled.

[0075] The memory is configured to store one or more computer instructions.

[0076] The processor is configured to execute the one or more computer instructions to implement the remote control method in the first aspect.

[0077] The sixth aspect of the embodiment of the present application further provides a controlled device, comprising a memory and a processor, the memory and the processor being coupled.

[0078] The memory is configured to store one or more computer instructions.

[0079] The processor is configured to execute one or more computer instructions to implement the remote control method of the second aspect.

[0080] The seventh aspect of the embodiments of the present application further provides a computer readable storage medium, which stores one or more computer instructions, and the instructions are executed by a processor to implement the remote control method of any of the technical solutions.

[0081] The technical solutions provided by the embodiments of the present application require the controlled device to perform a background screenshot on the background running controlled window to obtain the content data of the controlled window and send the content data to the control device. Then the control device receives the user operation on the controlled window, converts the user operation into a control instruction and sends the control instruction to the controlled device. Then the controlled device performs the control operation on the controlled window according to the control instruction. In this way, the controlled window can be remotely controlled without running in the foreground of the controlled device, and the controlled device can run other application programs or perform other operations in the foreground even if it is remotely controlled. At the same time, since the control device sends the control instruction to execute on a single independent window, multiple control devices can simultaneously control multiple background running windows of the same controlled device. In this way, the control efficiency and the running efficiency of the controlled device can be greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0083] Figure 1 A network architecture diagram of a remote control system provided by the embodiments of the present application;

[0084] Figure 2 A flowchart of a remote control method provided by the embodiments of the present application;

[0085] Figure 3 A structural diagram of a control device provided by the embodiments of the present application;

[0086] Figure 4 A structural diagram of a controlled device provided by the embodiments of the present application;

[0087] Figure 5 Another structural diagram of a control device provided by the embodiments of the present application;

[0088] Figure 6 Another structural diagram of a controlled device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0089] The application provides a remote control method and related devices. A control device determines a control instruction by obtaining the content of a controlled window running in the background, and then sends the control instruction to a controlled device, so that the controlled device performs an operation corresponding to the control instruction. In this way, the controlled window does not need to be displayed on the front screen. The controlled device can also perform other operations in the foreground when being remotely controlled.

[0090] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application. However, the present application can be implemented in many other ways different from the above description, therefore, based on the embodiments provided by the present application, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

[0091] It should be noted that the terms "first", "second", "third" and the like in the claims, specification and drawings of the present application are used to distinguish similar objects and are not intended to describe a specific order or sequence. The data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include", "have" and their variants are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0092] With the development of mobile Internet, remote control technology is increasingly widely used in enterprise office. Remote control technology is a technology that a network control device remotely controls another device. In the existing field, remote education, remote maintenance, remote assistance, and device remote control are typical application scenarios of remote control.

[0093] Currently, common remote control software mainly includes Sunflower, TeamViewer, etc. Some software such as Steam, QQ, and Nvidia graphics card driver also provides remote control function. The basic scheme of the current remote control includes two parts: one is that the control end device obtains the real-time state of the controlled end device, which usually includes screen image and sound. The other is that the control end device transmits control instructions such as mouse movement, keyboard input, and other operation instructions to the controlled end device through the network, instructing the controlled end device to perform control operations.

[0094] Remote control is primarily achieved through remote control software. Specifically, the controlled device captures content from its screen and sends it to the controlling device. The controlling device responds to the captured content, confirms it, and sends control commands to the controlled device. Finally, the controlled device executes the relevant operations on its screen according to the control commands, completing the entire remote control process.

[0095] In the above process, since the controlled device captures the screen content displayed in the foreground, the controlled window must be displayed in the foreground for the controlling device to control it. Furthermore, the controlled device must perform control operations on the controlled window on the foreground screen. This results in the controlled device only being able to perform control operations when remotely controlled, and being unable to perform other user operations. This will severely impact the operating efficiency of the controlled device.

[0096] To address the aforementioned problems, this application provides a remote control method and related equipment. The controlled device can intercept the content data of a controlled window running in the background and then send this content data to the controlling device. The controlling device determines control instructions for the controlled window based on the content data of the controlled window. Upon receiving the control instructions, the controlled device executes the control operation for the controlled window. In this way, the controlled window does not need to be displayed on the foreground screen, so the controlled device can perform other operations in the foreground even when being remotely controlled. The method, apparatus, terminal, and computer-readable storage medium of this application will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0097] Before introducing the embodiments of this application, a brief introduction to the remote control system will be given first:

[0098] Figure 1 This is a network architecture diagram of a remote control system provided in an embodiment of this application. Figure 1 As shown, the control device and the controlled device are connected via a communication network. The control device can then use remote control software to remotely control the controlled device. In this embodiment, multiple control devices can simultaneously control the same controlled device. This is because each control device can control one program window of the controlled device, and each controlled program window can run in the background. Thus, the control operation required by each control device is executed only in a single program window. Since each program window is independent and runs in the background, multiple control devices can control the same controlled device simultaneously. Furthermore, normal user operations can be performed in the foreground of the controlled device, greatly improving the operating efficiency of the controlled device.

[0099] Combination Figure 1 The network architecture diagram of the remote control system shown is as follows: Figure 2A flowchart of a remote control method is provided in the embodiments of the present application. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer-executable instructions, and in some cases, the steps shown can be executed in a logical order different from that shown in the flowchart.

[0100] As shown in Figure 2 the remote control method includes the following steps:

[0101] 201. The controlled device determines the controlled window in at least one background running window.

[0102] When the controlled device needs to be remotely controlled, it first needs to establish an effective network connection with the control device. For example, the controlled device receives a user operation, and according to the connection mode input by the user, the network identifier of the control device, the access password, and other information, sends a connection request to the control device. The network identifier of the control device can be the IP address of the control device, the computer name or login username of the control device, etc., which is not limited herein.

[0103] In the present embodiment, the control device is used to control a background running window on the controlled device, therefore, the controlled device also needs to determine the controlled window that needs to be controlled by the control device in the background running window, and then sends the identifier of the controlled window to the control device. It can be understood that one control device remotely controls one background running window on the controlled device, therefore, the controlled device can allow multiple control devices to simultaneously control different background running windows thereof. This can greatly improve the remote control efficiency.

[0104] After the control device receives the connection request sent by the controlled device, it can verify the user's authority according to the access password, etc., and establish a connection according to the verification result. Then the control device can identify the controlled window in the background running window of the controlled device according to the unique identifier of the target window specified by the controlled device.

[0105] 202. The controlled device sends the identifier of the controlled window to the control device.

[0106] For example, the controlled device can send the identifier of the controlled window to the control device in the above connection request, or can send the identifier of the controlled window separately. The specific implementation is not limited.

[0107] 203. The control device sends a window interception request to the controlled device according to the identifier of the controlled window.

[0108] After receiving the identification of the controlled window, the control device needs to send a window capture request to the controlled device, instructing the controlled device to perform a screenshot operation on the background running controlled window to obtain the content data of the controlled window. It can be understood that the function of the window capture request is to instruct the controlled device to start the remote control process, and the controlled device needs to respond to the window capture request to provide the control device with the window content of the controlled window.

[0109] 204、The controlled device performs a screenshot on the content data of the controlled window in the background according to the window capture request.

[0110] The controlled device can use the window API to continuously capture the controlled window to obtain real-time content data of the controlled window. Then encode the real-time content data according to a specific encoding / decoding format specification, and send the encoded content data to the control device. The encoding / decoding format specification can be selected according to actual needs, which can be JPEG picture stream, H.264 video stream, H.265 video stream specification, etc., and is not limited in particular.

[0111] The control device can also control the screenshot frequency of the controlled device. For example, the window capture request includes a preset time period. The controlled device periodically captures the content data of the controlled window according to the preset time period after receiving the window capture request.

[0112] The controlled device can use the window API to continuously capture the controlled window to obtain real-time content data of the controlled window. For example, for a background controlled window displayed using the GDI method under the Windows system, the background screenshot can be performed through the gdigrab method.

[0113] It can be understood that in the existing remote control process, the controlled device outputs a screenshot of the entire screen displayed in the foreground, but the control device often only needs part of the effective business data. Therefore, in order to improve the data processing rate and reduce the waste of transmission bandwidth, the window API needs to capture the content of the most effective area in the background window as much as possible. To ensure efficient use of transmission bandwidth, while also reducing data volume and improving data processing speed of the control device.

[0114] In the embodiments of the present application, the controlled device no longer intercepts the screen content displayed in the foreground, but intercepts the window content of the controlled window running in the background. In this way, the controlled window does not need to be displayed in the foreground and occupy the foreground. When being remotely controlled, the controlled device can also use the foreground to perform other operations, thereby improving the operation efficiency of the controlled device. Meanwhile, since the controlled window is running in the background, multiple control devices can simultaneously control different background running windows of one controlled device. Since the operation of each control device is only performed in a single background window of the controlled device, the multiple control devices will not conflict with each other, so that concurrent control of the controlled device by multiple control devices can be realized.

[0115] 205. The controlled device sends the content data of the controlled window to the control device.

[0116] After the controlled device intercepts the screenshot of the controlled window, the content data of the screenshot needs to be sent to the control device. For example, the real-time content data needs to be encoded according to a specific encoding / decoding format specification before being sent, so that the data transmission rate can be improved while ensuring the data security of the content data.

[0117] 206. The control device displays the content data of the controlled window sent by the controlled device.

[0118] After the control device receives the screen image encoding data returned from the controlled device, the control device needs to decode the information according to the agreed encoding / decoding mode, and then display the content data of the controlled window. It can be understood that remote control refers to how to operate on the control device, and the same operation also needs to be performed on the controlled device. Therefore, the control device needs to display the content data of the controlled window to receive the user operation on the controlled window.

[0119] 207. The control device receives the user operation on the controlled window.

[0120] The control device needs to receive the user operation on the controlled window, and then convert the user operation into a control instruction and send it to the controlled device. In this way, the controlled device can execute the same control operation as the user operation on the controlled window according to the control instruction, so as to achieve the purpose of remote control of the controlled window by the user.

[0121] 208. The control device determines the control instruction according to the user operation.

[0122] The control device needs to convert the user operation into a control instruction and send it to the controlled device. In this way, the controlled device can execute the relevant operation according to the control instruction and complete the remote control.

[0123] 209. The control device sends the control instruction to the controlled device.

[0124] For example, if the controlled window supports the message API, the control device can send the control instruction directly to the controlled window. In this way, the controlled window can perform the control operation in the background according to the control instruction. In this way, the controlled window does not need to run in the foreground of the controlled device, and the controlled device can run other applications or perform other operations in the foreground even if it is remotely controlled. At the same time, since the control instruction is executed directly in an independent background window, multiple control devices can simultaneously control multiple background running windows of the same controlled device. Thus, the running efficiency of the controlled device is greatly improved.

[0125] If the controlled window does not support the message API, the control operation still needs to be performed with the help of the foreground of the controlled device. Therefore, the control device needs to send the control instruction to the task queue running in the foreground. In this way, the controlled device needs to perform the control operation corresponding to the control instruction in the foreground to complete the remote control of the controlled window.

[0126] 210、The controlled device performs the control operation for the controlled window according to the control instruction.

[0127] The controlled device needs to perform the control operation according to the state of the controlled window. In the above example, if the controlled window supports the message API, the controlled device performs the control operation in the background running controlled window according to the control instruction. In this way, the controlled window does not need to run in the foreground of the controlled device, and the controlled window is remotely controlled in the background.

[0128] If the controlled window does not support the message API and still needs to be executed with the help of the foreground of the controlled device, the controlled device needs to call the controlled window in the foreground first, then get the control instruction in the task queue running in the foreground, and perform the control operation for the controlled window in the foreground according to the control instruction to complete the remote control of the controlled window.

[0129] In the above embodiment, the controlled device needs to perform a background screenshot of the background running controlled window to obtain the content data of the controlled window and send it to the control device. Then the control device receives the user operation for the controlled window, converts it into a control instruction and sends it to the controlled device, and then the controlled device performs the control operation for the controlled window according to the control instruction. In this way, the controlled window can be remotely controlled without running in the foreground of the controlled device, and the controlled device can run other applications or perform other operations in the foreground even if it is remotely controlled. At the same time, since the control instruction sent by the control device is only executed in a single independent window, multiple control devices can simultaneously control multiple background running windows of the same controlled device. Thus, the control efficiency and running efficiency of the controlled device are greatly improved.

[0130] The following will be described in detail for the scenario that the controlled window does not support the message API:

[0131] It can be understood that if the controlled window supports the message API, the control device only needs to send the control instruction directly to the controlled window. In this way, the controlled window can execute the control operation in the background of the controlled window according to the control instruction. The controlled window does not need to run in the foreground of the controlled device, and the controlled window can be remotely controlled in the background. In this way, the foreground can be liberated, and even in the remote control mode, the foreground of the controlled device also runs other application programs or performs other operations. At the same time, since the control instruction is executed directly in the independent background window, multiple control devices can simultaneously control multiple background running windows of the same controlled device.

[0132] If the controlled window does not support the message API, it needs to rely on the foreground of the controlled device to execute the control operation corresponding to the control instruction. At this time, the control device needs to send the control instruction to the message queue running in the foreground. When the message queue receives the control instruction of the controlled window, the controlled device calls the controlled window to execute and display in the foreground, and then executes the operation for the controlled window in the foreground by using the control instruction in the message queue. It can be understood that multiple background running windows of the controlled device can be controlled by different control devices at the same time, so the message queue will receive control instructions of multiple controlled windows. At this time, the controlled window needs to run multiple controlled windows in the foreground alternately to complete the control operation corresponding to multiple controlled windows.

[0133] It can be understood that the foreground running order of the controlled window can be determined according to the time sequence of receiving the control instruction by the message queue. Specifically, the controlled device determines the control instruction to be executed according to the receiving order of the control instruction in the message queue, then determines the controlled window corresponding to the control instruction, then calls the controlled window to run in the foreground, and finally executes the control operation for the controlled window in the foreground according to the control instruction. After the control operation is executed, the controlled window is switched to run in the background again. The next control instruction in the message queue is executed.

[0134] In order to reduce the operation delay, the controlled device can also reduce the foreground switching times by rearranging and the like. For example, there are multiple control instructions for the controlled window A and multiple control instructions for the controlled window B in the task queue, and the execution order of these control instructions is alternating after being arranged according to the message receiving time. At this time, the controlled device can execute all control instructions for the controlled window A first, and then execute all control instructions for the controlled window B. In this way, the foreground only needs to switch the controlled window once, which can reduce the operation delay and thus reduce the data processing burden of the controlled device.

[0135] The following describes the entire remote control process with a specific example:

[0136] The control device A remotely controls the program window 1 of the controlled device, the control device A and the program window 1 of the controlled device establish a remote connection, and the controlled device continuously sends the content data of the window 1 to the control device A.

[0137] The control device B remotely controls the program window 2 of the controlled device, the control device B and the program window 2 of the controlled device establish a remote connection, and the controlled device continuously sends the content data of the window 2 to the control device B.

[0138] Similarly, the control device N remotely controls the program window n of the controlled device, the control device N and the program window n of the controlled device establish a remote connection, and the controlled device continuously sends the content data of the window n to the control device N.

[0139] The control device A receives the remote operation action 1 issued by the user, converts the remote operation action 1 into a control instruction 1, and sends it to the controlled device, the controlled device receives the control instruction 1 and puts it into the instruction execution queue. Then the controlled device foregrounds the program window 1, sends the control instruction 1 to the operating system of the controlled device, and instructs the operating system of the controlled device to execute the operation on the program window 1 in the foreground according to the control instruction 1.

[0140] The control device B receives the remote operation action 2 issued by the user, converts the remote operation action 2 into a control instruction 2, and sends it to the controlled device, the controlled device receives the control instruction 2 and puts it into the instruction execution queue. Then the controlled device foregrounds the program window 2, sends the control instruction 2 to the operating system of the controlled device, and instructs the operating system of the controlled device to execute the operation on the program window 2 in the foreground according to the control instruction 1.

[0141] Similarly, the control device N receives the remote operation action n issued by the user, converts the remote operation action n into a control instruction n, and sends it to the controlled device, the controlled device receives the control instruction n and puts it into the instruction execution queue. Then the controlled device backgrounds the program window n-1, foregrounds the program window n, sends the control instruction n to the operating system of the controlled device, and instructs the operating system of the controlled device to execute the operation on the program window n in the foreground according to the control instruction n.

[0142] In the process of alternatingly switching the background window to the foreground to achieve remote control, in order to avoid frequent window switching, reduce operation delay and reduce CPU load, all operation instructions in the instruction queue for the current window can be executed in priority instead of following the time sequence of the operation instructions in the instruction execution queue.

[0143] In the embodiment, the controlled device needs to alternate the foreground running of multiple background running controlled windows when performing remote control operations on a controlled window that does not support the message API. The controlled device determines the control instruction to be executed according to the receiving order of the control instructions in the message queue, determines the controlled window corresponding to the control instruction, calls the controlled window to the foreground, performs the control operation on the controlled window in the foreground according to the control instruction, and switches the controlled window to the background after the control operation is performed. In order to reduce the operation delay, the controlled device can also reduce the foreground switching times by rearranging and the like. In this way, multiple control devices can simultaneously control multiple background running windows of the same controlled device. In this way, the control efficiency and the running efficiency of the controlled device are greatly improved.

[0144] Figure 3 A structural schematic diagram of a control device provided in the embodiment is shown in FIG. 1. The control device includes a receiving unit 301, a determining unit 302, and a sending unit 303. Figure 3 The embodiment provided in the embodiment is described in detail. The embodiments described below are used to explain the technical solutions of the present application, and are not limited by the actual use.

[0145] The control device includes:

[0146] The receiving unit 301 is configured to receive and display the content data of the controlled window sent by the controlled device. The controlled window is a program window running in the background of the controlled device.

[0147] The receiving unit 301 is further configured to receive the user operation on the controlled window.

[0148] The determining unit 302 is configured to determine the control instruction according to the user operation.

[0149] The sending unit 303 is configured to send the control instruction to the controlled device, so that the controlled device performs the control operation on the controlled window according to the control instruction.

[0150] In an optional embodiment, the receiving unit 301 is further configured to receive the identifier of the controlled window sent by the controlled device.

[0151] The sending unit 303 is further configured to send a window interception request to the controlled device according to the identifier of the controlled window. The window interception request is used to instruct the controlled device to take a screenshot of the content data of the controlled window in the background.

[0152] In an optional embodiment, the sending unit 303 is specifically configured to send the control instruction directly to the controlled window. Alternatively, the sending unit 303 is configured to send the control instruction to the task queue running in the foreground of the controlled device.

[0153] Figure 4A structural schematic diagram of a controlled device provided by an embodiment of the present application is shown in the following Figure 4 The embodiment provided by the present application is described in detail. The embodiments described below are used to explain the technical solutions of the present application, and are not limited to the actual use

[0154] The controlled device comprises:

[0155] A determination unit 401 is configured to determine a controlled window in at least one background running window.

[0156] A processing unit 402 is configured to perform screenshot on the controlled window in the background to obtain content data of the controlled window.

[0157] A sending unit 403 is configured to send the content data of the controlled window to a control device.

[0158] A receiving unit 404 is configured to receive a control instruction sent by the control device, the control instruction corresponding to the content data of the controlled window.

[0159] An execution unit 405 is configured to perform a control operation on the controlled window according to the control instruction.

[0160] In an optional implementation, the sending unit 403 is further configured to send an identifier of the controlled window to the control device.

[0161] The receiving unit 404 is further configured to receive a window screenshot request sent by the control device, the window screenshot request corresponding to the identifier of the controlled window.

[0162] The processing unit 402 is specifically configured to perform screenshot on the controlled window in the background according to the window screenshot request.

[0163] In an optional implementation, the sending unit 403 is specifically configured to perform compression and encoding processing on the content data of the controlled window. The content data of the controlled window after the compression and encoding processing is sent to the control device.

[0164] In an optional implementation, the receiving unit 404 is specifically configured to send the control instruction to the controlled window, the controlled window comprising a message interface.

[0165] The execution unit 405 is specifically configured to perform the control operation on the controlled window in the background according to the control instruction.

[0166] In an optional implementation, the receiving unit 404 is specifically configured to send the control instruction to a task queue running in the foreground.

[0167] The execution unit 405 is specifically configured to control the controlled window to run in the foreground. The control operation on the controlled window is performed in the foreground according to the control instruction.

[0168] In an optional implementation, the controlled device further includes an acquisition unit 406.

[0169] The acquisition unit 406 is specifically used to query the task queue running in the foreground and obtain all control instructions corresponding to the controlled window.

[0170] The execution unit 405 is specifically used to execute the control operation corresponding to each control instruction in the foreground in sequence.

[0171] In an optional implementation, the execution unit 405 is further configured to switch the controlled window to the background and control the controlled window to run in the background.

[0172] It should be noted that the information interaction and execution process between the various modules / units in the device are different from those in this application. Figures 1-2 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0173] In the above embodiment, the controlled device needs to take a background screenshot of the controlled window running in the background to obtain the content data of the controlled window and send it to the controlling device. Then, the controlling device receives user operations on the controlled window, converts them into control commands, and sends them to the controlled device. The controlled device then executes the control operations on the controlled window according to the control commands. In this way, the controlled window does not need to run in the foreground of the controlled device to be remotely controlled, and the controlled device can still run other applications or perform other operations in the foreground even when remotely controlled. At the same time, since the control device sends control commands only in a single independent window, multiple control devices can simultaneously control multiple background running windows of the same controlled device. This greatly improves the control efficiency and operating efficiency of the controlled device.

[0174] The following describes a control device provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of a control device provided in an embodiment of this application. The control device 800 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, a monitoring data processing device, or a radar data processing device, etc., and is not limited thereto. The control device 800 may be equipped with... Figure 3 The control device described in the corresponding embodiment is used to implement Figures 1-2 The functions correspond to those in the embodiments. Specifically, the control device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the number of processors 803 in the execution device 800 can be one or more). Figure 5The processor 803 can include, for example, an application processor 8031 and a communication processor 8032. In some embodiments of the present application, the receiver 801, the transmitter 802, the processor 803, and the memory 804 can be connected through a bus or other means.

[0175] The memory 804 can include read-only memory and random access memory, and provide the processor 803 with instructions and data. A portion of the memory 804 can also include non-volatile random access memory (NVRAM). The memory 804 stores processor and control operations, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the control operations can include various control operations for implementing various operations.

[0176] The processor 803 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, which can include a data bus in addition to a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, all kinds of buses are referred to as a bus system in the figure.

[0177] The method disclosed in the embodiments of the present application can be applied to the processor 803 or implemented by the processor 803. The processor 803 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the method can be completed by hardware integrated logic circuits in the processor 803 or by instructions in the form of software. The processor 803 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 803 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 804, and the processor 803 reads the information in the storage 804 and combines the hardware to complete the steps of the method.

[0178] The receiver 801 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 802 can be used to output digital or character information through the first interface; the transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 802 can also include a display device such as a display screen.

[0179] In the embodiments of the present application, the application processor 8031 in the processor 803 is used to execute the remote control method in the corresponding embodiments. It should be noted that the specific manner in which the application processor 8031 executes each step is based on the same concept as the corresponding method embodiments in the present application, which brings the same technical effects as the corresponding method embodiments in the present application, and the specific content can be referred to the description in the method embodiments disclosed in the present application, which will not be described here. Figure 2 In the embodiments of the present application, the application processor 8031 in the processor 803 is used to execute the remote control method in the corresponding embodiments. It should be noted that the specific manner in which the application processor 8031 executes each step is based on the same concept as the corresponding method embodiments in the present application, which brings the same technical effects as the corresponding method embodiments in the present application, and the specific content can be referred to the description in the method embodiments disclosed in the present application, which will not be described here. Figure 2 In the embodiments of the present application, the application processor 8031 in the processor 803 is used to execute the remote control method in the corresponding embodiments. It should be noted that the specific manner in which the application processor 8031 executes each step is based on the same concept as the corresponding method embodiments in the present application, which brings the same technical effects as the corresponding method embodiments in the present application, and the specific content can be referred to the description in the method embodiments disclosed in the present application, which will not be described here. Figure 2 In the embodiments of the present application, the application processor 8031 in the processor 803 is used to execute the remote control method in the corresponding embodiments. It should be noted that the specific manner in which the application processor 8031 executes each step is based on the same concept as the corresponding method embodiments in the present application, which brings the same technical effects as the corresponding method embodiments in the present application, and the specific content can be referred to the description in the method embodiments disclosed in the present application, which will not be described here.

[0180] The embodiments of the present application also provide a controlled device, please refer to Figure 6 , Figure 6is a structural schematic diagram of a controlled device provided by an embodiment of the present application, and the controlled device 900 can be deployed with Figure 4 corresponding embodiments of the controlled device described in the embodiments of the present application, for implementing Figure 2 corresponding functions of the controlled device in the embodiments of the present application. Specifically, the controlled device 900 is implemented by one or more servers, and the controlled device 900 can be greatly different due to different configurations or performances, and can include one or more central processing units (CPUs) 922 (for example, one or more processors) and a memory 932, one or more storage media 930 (for example, one or more mass storage devices) for storing an application program 942 or data 944. The memory 932 and the storage media 930 can be temporary storage or persistent storage. The program stored in the storage media 930 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the training device. Further, the central processing unit 922 can be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the controlled device 900.

[0181] The controlled device 900 can further include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958, and / or one or more operating systems 941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0182] In an embodiment of the present application, the central processing unit 922 is configured to execute Figure 2 the steps performed by the controlled device described in the corresponding embodiments of the present application. The specific manner in which the central processing unit 922 executes the foregoing steps is based on the same concept as the corresponding method embodiments of the present application, and the technical effects brought by the corresponding method embodiments of the present application are the same, and the specific content can be referred to the description of the method embodiments of the present application, which will not be repeated here. Figure 2 Figure 2 The corresponding method embodiments of the present application are based on the same concept, and the technical effects brought by the corresponding method embodiments of the present application are the same, and the specific content can be referred to the description of the method embodiments of the present application, which will not be repeated here.

[0183] An embodiment of the present application provides a computer readable storage medium, which includes computer instructions, and the computer instructions, when executed by a processor, are used to implement the technical solutions of any one of the remote control methods in the embodiments of the present application.

[0184] An embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to perform the steps in the method described in the foregoing Figure 2 embodiments. ​

[0185] The control device and the controlled device provided by the embodiments of the present application can be a chip, which comprises a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin, a circuit, or the like. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip executes the method described in the embodiments. Figure 2 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, or the like. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), or the like.

[0186] Although the present application is disclosed with the preferred embodiments, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be subject to the scope defined by the claims of the present application.

Claims

1. A remote control method characterized by, The remote control method comprises: The control device receives the content data of the controlled window sent by the controlled device; the controlled window is a program window running in the background of the controlled device; The control device receives the control operation for the controlled window and determines the control instruction according to the control operation; The control device sends the control instruction to the controlled device; so that the controlled device sends the control instruction to the task queue running in the foreground in the case that the controlled window does not support message interface, controls the controlled window to run in the foreground, queries the task queue running in the foreground, obtains all control instructions corresponding to the controlled window, and executes the control operation corresponding to each control instruction in the all control instructions in the foreground.

2. The method of claim 1, wherein, The method further comprises: The control device receives the identification of the controlled window sent by the controlled device; The control device sends a window interception request to the controlled device according to the identification of the controlled window; the window interception request is used to instruct the controlled device to take a screenshot of the content data of the controlled window in the background.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: The control device displays the content data of the controlled window in the foreground; The control device receives the control operation for the controlled window and determines the control instruction according to the control operation, comprising: The control device receives the user control operation for the controlled window in the foreground, and determines the control instruction according to the user control operation.

4. A remote control method characterized by, The remote control method comprises: The controlled device determines a controlled window in at least one window running in the background; The controlled device takes a screenshot of the controlled window in the background to obtain the content data of the controlled window; The controlled device sends the content data of the controlled window to a control device; The controlled device receives the control instruction sent by the control device; the control instruction corresponds to the content data of the controlled window; The controlled device executes the control operation for the controlled window according to the control instruction; The controlled device receives the control instruction sent by the control device, comprising: The controlled device sends the control instruction to the task queue running in the foreground in the case that the controlled window does not support message interface; The controlled device executes the control operation for the controlled window according to the control instruction, comprising: The controlled device controls the controlled window to run in the foreground; The controlled device executes the control operation for the controlled window in the foreground according to the control instruction; The method further comprises: The controlled device queries the task queue running in the foreground to obtain all control instructions corresponding to the controlled window; The controlled device executes the control operation for the controlled window in the foreground according to the control instruction, comprising: The controlled device executes the control operation corresponding to each control instruction in the all control instructions in the foreground.

5. The method of claim 4, wherein, The method further comprises: The controlled device sends the identification of the controlled window to the control device; The controlled device receives a window interception request sent by the control device; the window interception request corresponds to the identification of the controlled window; The controlled device intercepts the controlled window in the background, including: The controlled device intercepts the controlled window in the background according to the window interception request.

6. The method according to any one of claims 4 to 5, characterized in that, The controlled device sends the content data of the controlled window to the control device, including: The controlled device compresses and encodes the content data of the controlled window; The controlled device sends the content data of the controlled window after compression and encoding to the control device.

7. The method according to any one of claims 4 to 6, characterized in that, The controlled device receives a control instruction sent by the control device, including: The controlled device sends the control instruction to the controlled window; wherein the controlled window includes a message interface; The controlled device performs a control operation on the controlled window according to the control instruction, including: The controlled device performs the control operation on the controlled window in the background according to the control instruction.

8. The method of claim 4, after the controlled device performs the control operation on the controlled window in the foreground, the method further comprises: The controlled device switches the controlled window to the background and controls the controlled window to run in the background.

9. A control device characterized by comprising: The control device includes: A receiving unit for receiving content data of a controlled window sent by a controlled device; the controlled window is a program window running in the background of the controlled device; The receiving unit is also used to receive a control operation on the controlled window; A determination unit for determining a control instruction according to the control operation; A sending unit for sending the control instruction to the controlled device; so that the controlled device sends the control instruction to a task queue running in the foreground in the case that the controlled window does not support a message interface, controls the controlled window to run in the foreground, queries the task queue running in the foreground, obtains all control instructions corresponding to the controlled window, and executes the control operation corresponding to each control instruction in the foreground.

10. A controlled device, characterized by The controlled device includes: A determination unit for determining a controlled window in at least one window running in the background; A processing unit for intercepting the controlled window in the background to obtain content data of the controlled window; A sending unit for sending the content data of the controlled window to a control device; A receiving unit for receiving a control instruction sent by the control device; the control instruction corresponds to the content data of the controlled window; An execution unit for performing a control operation on the controlled window according to the control instruction; The controlled device receives a control instruction sent by the control device, including: The controlled device sends the control instruction to a task queue running in the foreground in the case that the controlled window does not support a message interface; The controlled device performs a control operation on the controlled window according to the control instruction, including: The controlled device controls the controlled window to run in the foreground; The controlled device executes the control operation corresponding to the controlled window in the foreground according to the control instruction; The controlled device is further configured to: query the task queue running in the foreground to obtain all control instructions corresponding to the controlled window; The controlled device executes the control operation corresponding to each control instruction of the all control instructions in the foreground according to the control instruction. The controlled device executes the control operation corresponding to each control instruction of the all control instructions in the foreground.

11. A control device, characterized by comprising: a memory and a processor, the memory and the processor being coupled; the memory is configured to store one or more computer instructions; the processor is configured to execute the one or more computer instructions to implement the remote control method according to any one of claims 1-3.

12. A controlled device, characterized by comprising: a memory and a processor, the memory and the processor being coupled; the memory is configured to store one or more computer instructions; the processor is configured to execute the one or more computer instructions to implement the remote control method according to any one of claims 4-8.

13. A computer-readable storage medium having stored thereon one or more computer instructions, wherein, The instructions are executed by the processor to implement the remote control method according to any one of claims 1-8.

Images