Remote control method and device, electronic equipment and storage medium

Abstract

The invention provides a remote control method and device, electronic equipment and a storage medium, and relates to the technical field of remote control. A cloud server obtains a remote control instruction by monitoring a remote control request sent by a mobile terminal to a remote terminal and analyzing the remote control request; then the remote control instruction is sent to an identification program of the remote terminal through the proxy connection, and then the identification program can determine a target application, a target element of the target application and a control action for the target application from the to-be-controlled applications according to the remote control instruction; and finally, the cloud server controls the identification program to execute a control action on the target element.

Description

Technical Field

[0001] The present invention relates to the field of remote control technology, and in particular to a remote control method, device, electronic equipment and storage medium. Background Art

[0002] At present, the method of remotely controlling personal computer desktop applications usually relies on the traditional Remote Desktop Protocol (RDP) or is implemented through third-party remote control software. Although these methods can realize remote operation, they have several problems and shortcomings.

[0003] First, traditional remote control software usually requires users to perform complex configurations on both remote and local devices, including network settings, firewall configuration, port forwarding, etc., which is not user-friendly. Second, traditional remote control software requires a high bandwidth when transmitting graphic data, especially in the case of high resolution and high frame rate. Network instability can cause operation to freeze or fail, seriously affecting the efficiency of use. In addition, the existing technology rarely makes full use of mainstream social platforms such as WeChat for convenient messaging and remote control, which limits the application scenarios of mobile terminals. Therefore, the existing technology limits the application scenarios and practicality of remote operation. Summary of the invention

[0004] In view of this, the present invention aims to propose a remote control method, device, electronic device and storage medium to solve the problem that the application scenarios and practicality of current remote operations are greatly limited, resulting in a poor user experience of remotely controlling a remote terminal on a mobile terminal.

[0005] According to a first aspect of the present invention, there is provided a remote control method, which is applied to a cloud server, wherein the cloud server is connected to a mobile terminal and a remote terminal, a corresponding proxy connection is configured between the cloud server and the remote terminal, and the remote terminal includes an identification program and a plurality of applications to be controlled, and the method comprises:

[0006] monitoring a remote control request sent by the mobile terminal to the remote terminal, and parsing the remote control request to obtain a remote control instruction;

[0007] sending the remote control instruction to an identification program of the remote terminal through the proxy connection, the identification program being used to determine a target application, a target element of the target application, and a control action for the target application from the application to be controlled according to the remote control instruction;

[0008] The recognition program is controlled to perform the control action on the target element.

[0009] Optionally, the proxy connection includes an encrypted channel and a target proxy connection, and the identification procedure for sending the remote control instruction to the remote terminal through the proxy connection includes:

[0010] encrypting the remote control instruction through the encryption channel;

[0011] The encrypted remote control instruction is sent to the identification program of the remote terminal through the target proxy connection.

[0012] Optionally, the cloud server includes a first encryption algorithm list, the remote terminal includes a second encryption algorithm list, and the encryption channel is established in the following manner:

[0013] Sending a connection request to the remote terminal, and receiving a response message from the remote terminal in response to the connection request;

[0014] Establishing an initial connection with the remote terminal according to the response message;

[0015] exchanging the first encryption algorithm list and the second encryption algorithm list, and selecting a target encryption algorithm from the first encryption algorithm list and the second encryption algorithm list;

[0016] Generate a shared key of the target encryption algorithm through a password exchange protocol;

[0017] The encryption channel is established in combination with the initial connection, the target encryption algorithm, and the shared key.

[0018] According to another aspect of the present invention, there is also provided a remote control method, which is applied to an identification program, the identification program is located in a remote terminal, the remote terminal further includes a plurality of applications to be controlled, the remote terminal is connected to a cloud server, and a corresponding proxy connection is configured between the remote terminal and the cloud server, the method comprising:

[0019] Receiving a remote control instruction sent by the cloud server through the proxy connection;

[0020] Determining a target application and a control action for the target application from the applications to be controlled according to the remote control instruction;

[0021] Loading the image file of the target application from the image recognition library of the remote terminal;

[0022] Creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application;

[0023] The control action is performed on the target element.

[0024] Optionally, before receiving the remote control instruction sent by the cloud server through the proxy connection, the method includes:

[0025] Taking a screenshot of the application to be controlled to obtain an image file of the application to be controlled;

[0026] The image file of the application to be controlled is stored to generate an image recognition library.

[0027] Optionally, the proxy connection includes an encrypted channel and a target proxy connection, a proxy server is connected between the mobile terminal and the cloud server, and the target proxy connection is established in the following manner:

[0028] Obtaining a proxy request sent by the application to be controlled;

[0029] Constructing a data packet, and sending the data packet and the proxy request to the proxy server for verification, wherein the data packet includes the identity information of the application to be controlled;

[0030] Receiving a verification result of the identity information by the proxy server;

[0031] When the verification result is that the verification is passed, the target proxy connection is established.

[0032] Optionally, the identification program is configured with a connection class, the address of the connection class is the address of the identification program, and after obtaining the proxy request sent by the application to be controlled, the method includes:

[0033] Acquire configuration information from the connection class, the configuration information including connection configuration, sending configuration and receiving configuration;

[0034] Rewrite the connection configuration, sending configuration and receiving configuration to generate new configuration information;

[0035] Creating a new connection class according to the new configuration information, where the address of the new connection class is the address of the proxy server;

[0036] The step of constructing a data packet and sending the data packet and the proxy request to the proxy server for verification includes:

[0037] A data packet is constructed, and the data packet and the proxy request are sent to the proxy server for verification according to the address of the new connection class.

[0038] According to another aspect of the present invention, there is also provided a remote control device, which is applied to a cloud server, wherein the cloud server is connected to a mobile terminal and a remote terminal, a corresponding proxy connection is configured between the cloud server and the remote terminal, and the remote terminal includes an identification program and a plurality of applications to be controlled, and the device includes:

[0039] A first monitoring module, configured to monitor a remote control request sent by the mobile terminal to the remote terminal, and parse the remote control request to obtain a remote control instruction;

[0040] A first sending module, configured to send the remote control instruction to an identification program of the remote terminal through the proxy connection, wherein the identification program is configured to determine a target application, a target element of the target application, and a control action for the target application from the application to be controlled according to the remote control instruction;

[0041] The first control module is used to control the recognition program to perform the control action on the target element.

[0042] Optionally, the proxy connection includes an encrypted channel and a target proxy connection, and the first sending module includes:

[0043] An instruction encryption module, used for encrypting the remote control instruction through the encryption channel;

[0044] The first sending submodule is used to send the encrypted remote control instruction to the identification program of the remote terminal through the target proxy connection.

[0045] Optionally, the cloud server includes a first encryption algorithm list, the remote terminal includes a second encryption algorithm list, and the encryption channel is established through the following modules:

[0046] A connection request transmission module, used for sending a connection request to the remote terminal and receiving a response message from the remote terminal in response to the connection request;

[0047] An initial connection establishing module, used for establishing an initial connection with the remote terminal according to the response message;

[0048] an encryption algorithm exchange module, configured to exchange the first encryption algorithm list and the second encryption algorithm list, and select a target encryption algorithm from the first encryption algorithm list and the second encryption algorithm list;

[0049] A shared key generation module, used to generate a shared key of the target encryption algorithm through a password exchange protocol;

[0050] The encryption channel generation module is used to establish the encryption channel in combination with the initial connection, the target encryption algorithm and the shared key.

[0051] According to another aspect of the present invention, there is also provided a remote control device, which is applied to an identification program, wherein the identification program is located in a remote terminal, wherein the remote terminal further comprises a plurality of applications to be controlled, wherein the remote terminal is connected to a cloud server, and a corresponding proxy connection is configured between the remote terminal and the cloud server, wherein the device comprises:

[0052] A first receiving module, configured to receive a remote control instruction sent by the cloud server through the proxy connection;

[0053] A first determining module, configured to determine a target application and a control action for the target application from the applications to be controlled according to the remote control instruction;

[0054] A first loading module, used for loading the image file of the target application from the image recognition library of the remote terminal;

[0055] A first matching module, used for creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application;

[0056] The first execution module is used to execute the control action on the target element.

[0057] Optionally, before the first receiving module, the device includes:

[0058] A screenshot module, used for taking a screenshot of the application to be controlled to obtain an image file of the application to be controlled;

[0059] The generating module is used to store the image file of the application to be controlled and generate an image recognition library.

[0060] Optionally, the proxy connection includes an encrypted channel and a target proxy connection, a proxy server is connected between the mobile terminal and the cloud server, and the target proxy connection is established through the following modules:

[0061] A proxy request acquisition module, used to acquire a proxy request sent by the application to be controlled;

[0062] An identity information verification module, used to construct a data packet, and send the data packet and the proxy request to the proxy server for verification, wherein the data packet includes the identity information of the application to be controlled;

[0063] A verification result receiving module, used for receiving the verification result of the identity information by the proxy server;

[0064] The target proxy connection establishing module is used to establish the target proxy connection when the verification result is verification passed.

[0065] Optionally, the identification program is configured with a connection class, the address of the connection class is the address of the identification program, and after the proxy request acquisition module, the device includes:

[0066] A configuration information acquisition module, used to acquire configuration information from the connection class, wherein the configuration information includes connection configuration, sending configuration and receiving configuration;

[0067] A configuration information rewriting module, used to rewrite the connection configuration, the sending configuration and the receiving configuration to generate new configuration information;

[0068] A new connection class creation module, used to create a new connection class according to the new configuration information, the address of the new connection class being the address of the proxy server;

[0069] The identity information verification module is specifically used to construct a data packet, and send the data packet and the proxy request to the proxy server for verification according to the address of the new connection class.

[0070] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0071] processor;

[0072] a memory for storing instructions executable by the processor;

[0073] The processor is configured to execute the instructions to implement the remote control method as described above.

[0074] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the remote control method described above are implemented.

[0075] In the remote control method provided by the embodiment of the present invention, the cloud server monitors the remote control request sent by the mobile terminal to the remote terminal, and parses the remote control request to obtain the remote control instruction; then the remote control instruction is sent to the recognition program of the remote terminal through the proxy connection, and then the recognition program can determine the target application, the target element of the target application and the control action for the target application from the application to be controlled according to the remote control instruction; finally, the cloud server controls the recognition program to perform the control action on the target element. The present invention simplifies the process of users remotely controlling desktop applications, expands the application scenarios, and realizes efficient scheduling of remote terminals through the combination of lightweight proxy connection and image recognition operations, thereby improving the user experience.

[0076] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0078] Figure 1 is a flowchart of the steps of a remote control method provided by an embodiment of the present invention;

[0079] Figure 2 is a flowchart of the steps of a remote control method provided by an embodiment of the present invention;

[0080] Figure 3 is a structural schematic diagram of a remote control device provided by an embodiment of the present invention;

[0081] Figure 4 is a schematic structural diagram of another remote control device provided by an embodiment of the present invention;

[0082] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it can be understood by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable readers to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present invention can also be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0084] Currently, remote control of personal computer desktop applications mainly relies on traditional remote desktop protocols or third-party remote control software. Although these technologies can achieve remote operation, their inherent limitations seriously restrict the expansion of user experience and application scenarios.

[0085] On the one hand, traditional remote control software requires users to perform cumbersome configurations on both remote and local devices, including network settings, firewall configuration, port forwarding, etc. This is too difficult for ordinary users to easily get started, which greatly limits the popularity of remote control technology. In addition, when transmitting graphic data, traditional remote control software requires a high bandwidth, especially in the case of high resolution and high frame rate. Network fluctuations can easily cause operation jams, delays, and even connection interruptions, seriously affecting usage efficiency and experience.

[0086] On the other hand, existing technologies rarely make full use of mainstream social platforms such as WeChat for convenient messaging and remote control, which limits the application scenarios of mobile terminals. Users cannot easily access and control their personal computer desktops through mobile devices such as mobile phones anytime and anywhere, making it difficult to meet the growing demand for mobile office and remote collaboration. Some remote control software has hidden dangers in terms of security and is easily attacked by hackers, resulting in user privacy leakage and data security risks.

[0087] Therefore, an embodiment of the present invention provides a remote control method applied to a cloud server, referring to Figure 1 , showing a flowchart of the steps of the method, the cloud server of the present invention is connected to the mobile terminal and the remote terminal, a corresponding proxy connection is configured between the cloud server and the remote terminal, the remote terminal includes an identification program and a plurality of applications to be controlled, and the method includes:

[0088] Step 101: monitor a remote control request sent by a mobile terminal to a remote terminal, and parse the remote control request to obtain a remote control instruction.

[0089] In one embodiment of the present invention, it is necessary to provide the user with a special account for remote control. The user can add the special account to the address book by scanning a QR code or searching for the account in the search box. The special account can be in the form of a public account or a one-to-one customer service account. The present invention does not specifically limit this.

[0090] For example, after the user applies for the remote control service dedicated to the present invention, the user obtains the QR code provided by the present invention, and first uses the APP (Application) of the mobile terminal to scan the QR code. The mobile terminal of the present invention can be a smart phone, a tablet computer, a smart watch, or other products that are easy for the user to carry. After scanning the QR code, the special account provided by the present invention can be seen. The user needs to follow or add the special account. The present invention can also send a unique identification code to the user who applies for the remote control service. After the user follows or adds the special account, the user needs to send the unique identification code to the special account and wait for the special account to verify the unique identification code. After the verification is passed, the binding of the user's account and the special account is completed. In an embodiment of the present invention, it is also necessary to input the IP address of the remote terminal into the bound special account, so as to complete the binding between "mobile terminal-cloud server-remote terminal".

[0091] The user needs to send a remote control message to the bound special account, such as "open the browser and search for the weather for the next week". After the mobile terminal recognizes the remote control message sent by the user, it encapsulates the remote control message into a remote control request and sends the remote control request to the cloud server.

[0092] After receiving the remote control request from the mobile terminal, the cloud server needs to parse the remote control request to obtain the remote control instruction contained in the remote control request. The remote control instruction is an instruction for controlling the remote terminal, which contains information about the remote operation.

[0093] Step 102 : sending the remote control instruction to the identification program of the remote terminal through the proxy connection, the identification program is used to determine the target application, the target element of the target application and the control action for the target application from the application to be controlled according to the remote control instruction.

[0094] In an embodiment of the present invention, a proxy connection is configured between the cloud server and the identification program of the remote terminal. The proxy connection of the present invention is a Socks5 connection, and the identification program can be a Python application. First, a corresponding proxy server needs to be configured for the Socks5 connection. The proxy server is another service end that does not belong to the mobile terminal, the cloud server, and the remote terminal, and the proxy server has an IP address and a port number.

[0095] The proxy server, as an intermediate node, is responsible for forwarding the remote control instructions sent by the cloud server to the identification program of the remote terminal. Under normal circumstances, the cloud server directly sends the remote control instructions to the identification program, and the present invention establishes a direct Socks5 connection between the cloud server and the remote terminal, and configures a proxy server, and the remote control instructions are sent from the cloud server to the proxy server, and the proxy server forwards the remote control instructions to the identification program. By forwarding instructions by the proxy server, the real IP addresses of the cloud server and the remote terminal can be hidden, anonymity is increased, and the risk of being attacked is reduced. In some network environments, there may be firewalls or other network restrictions that prevent direct remote control instruction transmission. Through the Socks5 proxy, these restrictions can be bypassed to ensure that the instructions can smoothly reach the identification program.

[0096] Step 103: Control the recognition program to perform a control action on the target element.

[0097] After the recognition program completes the control action on the target element, that is, after the remote control operation ends, the cloud server receives the remote operation feedback from the recognition program, such as "Completed", "Error! Please check the network connection", etc. The feedback message is used to indicate the execution status of this remote operation. After receiving the remote operation feedback sent by the recognition program, the cloud server will resend the remote operation instructions corresponding to the remote operation feedback with errors to the recognition program, and collect new remote operation feedback. If errors are still reported after multiple repetitions, the cloud server will send feedback messages to the mobile terminal for the user to check.

[0098] In one embodiment, the proxy connection includes an encrypted channel and a target proxy connection, and the identification program of sending the remote control instruction to the remote terminal through the proxy connection includes:

[0099] The remote control command is encrypted through an encrypted channel; the encrypted remote control command is sent to the identification program of the remote terminal through a target proxy connection.

[0100] In one embodiment of the present invention, it is necessary to establish an encrypted channel, which is used to ensure that the data transmitted between the cloud server and the recognition program is encrypted data, so as to prevent a series of hazards caused by attacks and tampering of remote control instructions by a third party when controlling the remote terminal.

[0101] In one embodiment, the cloud server includes a first encryption algorithm list, the remote terminal includes a second encryption algorithm list, and the encryption channel is established in the following manner:

[0102] Sending a connection request to a remote terminal and receiving a response message from the remote terminal in response to the connection request;

[0103] Establishing an initial connection with the remote terminal according to the response message;

[0104] Exchange the first encryption algorithm list and the second encryption algorithm list, and select a target encryption algorithm from the first encryption algorithm list and the second encryption algorithm list;

[0105] Generate a shared key for the target encryption algorithm through a password exchange protocol;

[0106] An encrypted channel is established by combining the initial connection, the target encryption algorithm, and the shared key.

[0107] When establishing an encrypted channel, the present invention first needs to initialize the context of the security protocol. The present invention does not specifically limit the selection of the security protocol. For example, SSL (Secure Sockets Layer) / TLS (Transport Layer Security) can be selected. SSL encrypts data by using encryption algorithms (such as AES, RSA, etc.) to ensure that the data will not be stolen or tampered with during transmission. At the same time, a message authentication code can be used to verify the integrity of the data. During the data transmission process, any tampering with the data will be detected, thereby ensuring the integrity of the data. Only the recipient with the correct key can decrypt and read the data.

[0108] The cloud server first sends a connection request to the identification program. The connection request contains information such as the supported security protocol version and encryption algorithm list. The identification program selects a security protocol version and encryption algorithm, and sends its digital certificate to the cloud server. The cloud server verifies the digital certificate of the identification program. If the verification is successful, the cloud server generates a random symmetric key, encrypts it with the public key of the identification program, and sends it to the identification program. The identification program uses its own private key to decrypt the key sent by the cloud server. The two parties complete the handshake and establish an encrypted communication channel.

[0109] After the handshake is completed, the cloud server and the identification program use the negotiated symmetric key to encrypt and decrypt the data to ensure the security of data transmission.

[0110] After the communication is completed, both the cloud server and the recognition program securely close the SSL connection by sending a close message, ensuring that any unfinished data transmission is properly processed.

[0111] In one embodiment of the present invention, a trust manager is set in the cloud server. The trust manager is responsible for verifying the digital certificate of the identification program in the cloud server. The trust manager loads the trusted certificate from the specified trust store (TrustStore). The trust store usually contains a set of trusted root certificate authority (CA) certificates. The trust manager verifies whether the digital certificate provided by the identification program is issued by a trusted CA and the certificate chain is valid. If the certificate verification fails, the SSL / TLS connection cannot be established. The trust manager can also be customized to add additional verification logic, such as checking the expiration time of the certificate, the subject name, etc.

[0112] In one embodiment of the present invention, a key manager is set in the cloud server, and the key manager is responsible for managing the digital certificate and private key of the identification program in the cloud server. The key manager loads the digital certificate and private key of the identification program from a specified key store (KeyStore), and the key store usually contains the digital certificate of the identification program and its corresponding private key.

[0113] In the remote control method provided by the embodiment of the present invention, the cloud server monitors the remote control request sent by the mobile terminal to the remote terminal, and parses the remote control request to obtain the remote control instruction; then the remote control instruction is sent to the recognition program of the remote terminal through the proxy connection, and then the recognition program can determine the target application, the target element of the target application and the control action for the target application from the application to be controlled according to the remote control instruction; finally, the cloud server controls the recognition program to perform the control action on the target element. The present invention simplifies the process of users remotely controlling desktop applications, expands the application scenarios, and realizes efficient scheduling of remote terminals through the combination of lightweight proxy connection and image recognition operations, thereby improving the user experience.

[0114] Further, see Figure 2 , showing a remote control method provided by the present invention, which is applied to an identification program, the identification program is located in a remote terminal, the remote terminal also includes a number of applications to be controlled, the remote terminal is connected to a cloud server, and a corresponding proxy connection is configured between the remote terminal and the cloud server, and the method includes:

[0115] Step 201, receiving a remote control instruction sent by a cloud server through a proxy connection;

[0116] The recognition program on the remote terminal listens for control commands from the cloud server. Since the commands are sent through the Socks5 proxy connection, the recognition program needs to be configured to receive commands through the proxy connection.

[0117] Step 202, determining a target application and a control action for the target application from the applications to be controlled according to the remote control instruction;

[0118] After the recognition program receives the instruction, it can parse the instruction content to obtain the target application required for this remote control and the control action for the target application. The present invention uses image recognition technology to locate and identify the desktop of the remote terminal, such as application icons, menu items, text boxes, etc. First, it is necessary to use a Python library (such as a computer vision library) to capture the image of the current remote terminal screen, and then use the template matching method to detect the features of the target application in the captured image, and determine the position and size of the target application on the screen based on the detected features. After determining the target application, the recognition program further locates the target element in the target application. This process involves parsing the user interface (UI) structure of the application, or locating the element through a specific UI automation tool (such as Selenium, Appium, etc.).

[0119] Step 203, loading the image file of the target application from the image recognition library of the remote terminal;

[0120] Step 204, creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application;

[0121] Step 205: Execute a control action on the target element.

[0122] In one example, for example, "search the weather for the next week in the browser", after analysis, it can be determined that the target application is the "browser", and the control action for the target application is "search", and the essence of the search function can be broken down into entering the search content in the search bar and pressing the search button. The search content is "weather for the next week". At the same time, it can be seen that in order to use the search function in the browser, the "search bar" and "input method" are required, and the search bar belongs to the target element in the browser, and the input method belongs to the associated application of the browser. Therefore, when the present invention performs a control action on the target element, it can also trigger the associated application of the target application. This associated application must be the associated application that needs to be used when performing the control action on the target element this time. Associated applications that are associated with the browser but are not used in this remote operation will not be triggered.

[0123] Once the target application and target element are determined, the recognition program performs the corresponding operation according to the control action in the instruction. For example, if the control action is "click", the recognition program will simulate the user clicking the target element; if the control action is "enter text", the recognition program will simulate the user entering text in the target element.

[0124] After executing the control action, the recognition program can feed back the execution results to the cloud server so that the cloud server can understand whether the operation is successful. During the whole process, the recognition program can record the execution of each step, including the received instructions, parsing results, positioning results, and execution results. These logs can be used for monitoring and debugging. If an exception occurs during the execution process (such as the target application is not found, the target element is not located, etc.), the recognition program can capture the exception and record the log, and feed back the exception information to the cloud server.

[0125] In one embodiment of the present invention, before step 201, the method includes:

[0126] A screenshot of the application to be controlled is taken to obtain an image file of the application to be controlled; the image file of the application to be controlled is stored to generate an image recognition library.

[0127] The present invention needs to take screenshots of all the applications to be controlled on the screen of the remote terminal, obtain the image files corresponding to each application to be controlled, record the name of each application to be controlled, associate the name with the image file and store them together to generate an image recognition library. In addition, the image recognition library needs to be updated regularly, re-capture all the applications to be controlled on the screen of the remote terminal, and record whether there are control applications with thumbnails changed or names changed. For applications to be controlled with thumbnails changed or names changed, re-association is required, and the old association in the image recognition library is replaced with the new association. For applications to be controlled that have not undergone any changes, there is no need to re-store.

[0128] In one embodiment, the proxy connection includes an encrypted channel and a target proxy connection, a proxy server is connected between the mobile terminal and the cloud server, and the target proxy connection is established in the following manner:

[0129] Get the proxy request sent by the application to be controlled;

[0130] Construct a data packet and send the data packet and proxy request to the proxy server for verification. The data packet contains the identity information of the application to be controlled.

[0131] Receive the verification result of the identity information by the proxy server;

[0132] If the verification result is passed, a target proxy connection is established.

[0133] The Socks5 proxy link of the present invention includes a target proxy connection of an encrypted channel. When the recognition program and the cloud server send data to each other, the sent data first needs to be encrypted in the encrypted channel and then the encrypted data is sent through the target proxy connection.

[0134] In one embodiment of the present invention, the identification program is configured with a connection class, the address of the connection class is the address of the identification program, and after obtaining the proxy request sent by the application to be controlled, the method includes:

[0135] Get configuration information from the connection class, which includes connection configuration, sending configuration, and receiving configuration;

[0136] Rewrite the connection configuration, sending configuration, and receiving configuration to generate new configuration information;

[0137] Create a new connection class according to the new configuration information, and the address of the new connection class is the address of the proxy server;

[0138] Construct a data packet and send the data packet and proxy request to the proxy server for verification, including:

[0139] Construct a data packet and send the data packet and proxy request to the proxy server for verification according to the address of the new connection class.

[0140] When the cloud server of the present invention sends instructions to the identification program, in order to hide the real IP addresses of the cloud server and the remote terminal, increase anonymity, and reduce the risk of being attacked, it is necessary to change the connection class in the identification program. The connection class is configured with the address of the identification program. The address of the identification program in the connection class is changed to the address of the proxy server to obtain a new connection class. Therefore, when the cloud server sends instructions to the identification program, it will send the instructions to the proxy server according to the new address in the new connection class, and the proxy server will then forward the instructions to the identification program.

[0141] The remote control method provided by the embodiment of the present invention comprises a recognition program receiving a remote control instruction sent by a cloud server through a proxy connection, and determining a target application and a control action for the target application from the application to be controlled according to the remote control instruction; further loading an image file of the target application from an image recognition library of a remote terminal; creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application; and finally, executing a control action on the target element. The present invention utilizes image recognition technology to simulate user operations, such as clicking an icon to start an application, operating a menu, etc., so as to complete the scheduling of the target desktop application.

[0142] Reference Figure 3 , shows a remote control device provided by an embodiment of the present invention, which is applied to a cloud server, the cloud server is connected to a mobile terminal and a remote terminal, a corresponding proxy connection is configured between the cloud server and the remote terminal, the remote terminal includes an identification program and several applications to be controlled, and the device includes:

[0143] The first monitoring module 301 is used to monitor the remote control request sent by the mobile terminal to the remote terminal, and parse the remote control request to obtain the remote control instruction;

[0144] A first sending module 302, configured to send the remote control instruction to the identification program of the remote terminal through a proxy connection, wherein the identification program is configured to determine the target application, the target element of the target application, and the control action for the target application from the application to be controlled according to the remote control instruction;

[0145] The first control module 303 is used to control the recognition program to perform a control action on the target element.

[0146] Optionally, the proxy connection includes an encrypted channel and a target proxy connection, and the first sending module 302 includes:

[0147] A command encryption module, used to encrypt remote control commands through an encrypted channel;

[0148] The first sending submodule is used to send the encrypted remote control instruction to the identification program of the remote terminal through the target proxy connection.

[0149] Optionally, the cloud server includes a first encryption algorithm list, the remote terminal includes a second encryption algorithm list, and the encryption channel is established through the following modules:

[0150] A connection request transmission module, used to send a connection request to a remote terminal and receive a response message from the remote terminal in response to the connection request;

[0151] An initial connection establishment module, used for establishing an initial connection with a remote terminal according to a response message;

[0152] An encryption algorithm exchange module, used to exchange the first encryption algorithm list and the second encryption algorithm list, and select a target encryption algorithm from the first encryption algorithm list and the second encryption algorithm list;

[0153] A shared key generation module, used to generate a shared key of a target encryption algorithm through a password exchange protocol;

[0154] The encryption channel generation module is used to establish an encryption channel by combining the initial connection, the target encryption algorithm and the shared key.

[0155] The above steps 101 to 103 refer to the previous steps and will not be repeated here.

[0156] In the remote control method provided by the embodiment of the present invention, the cloud server monitors the remote control request sent by the mobile terminal to the remote terminal, and parses the remote control request to obtain the remote control instruction; then the remote control instruction is sent to the recognition program of the remote terminal through the proxy connection, and then the recognition program can determine the target application, the target element of the target application and the control action for the target application from the application to be controlled according to the remote control instruction; finally, the cloud server controls the recognition program to perform the control action on the target element. The present invention simplifies the process of users remotely controlling desktop applications, expands the application scenarios, and realizes efficient scheduling of remote terminals through the combination of lightweight proxy connection and image recognition operations, thereby improving the user experience.

[0157] Reference Figure 4 , shows a remote control device of the present invention, which is applied to the recognition program, the recognition program is located in the remote terminal, the remote terminal also includes a number of applications to be controlled, the remote terminal is connected to the cloud server, and a corresponding proxy connection is configured between the remote terminal and the cloud server, and the device includes:

[0158] The first receiving module 401 is used to receive the remote control instruction sent by the cloud server through the proxy connection;

[0159] A first determination module 402, configured to determine a target application and a control action for the target application from the applications to be controlled according to the remote control instruction;

[0160] A first loading module 403, used to load an image file of a target application from an image recognition library of a remote terminal;

[0161] A first matching module 404 is used to create an element template corresponding to the remote control instruction, match the element template with the image file of the target application, and obtain a target element corresponding to the target application;

[0162] The first execution module 405 is used to execute a control action on a target element.

[0163] Optionally, before the first receiving module 401, the device includes:

[0164] A screenshot module, used to take a screenshot of the application to be controlled to obtain an image file of the application to be controlled;

[0165] The generation module is used to store the image files of the application to be controlled and generate an image recognition library.

[0166] Optionally, the proxy connection includes an encrypted channel and a target proxy connection, a proxy server is connected between the mobile terminal and the cloud server, and the target proxy connection is established through the following modules:

[0167] A proxy request acquisition module, used to acquire a proxy request sent by the application to be controlled;

[0168] An identity information verification module is used to construct a data packet and send the data packet and the proxy request to the proxy server for verification, the data packet contains the identity information of the application to be controlled;

[0169] A verification result receiving module, used for receiving the verification result of the proxy server on the identity information;

[0170] The target proxy connection establishment module is used to establish a target proxy connection when the verification result is that the verification is passed.

[0171] Optionally, the identification program is configured with a connection class, the address of the connection class is the address of the identification program, and after the proxy request acquisition module, the device includes:

[0172] The configuration information acquisition module is used to obtain configuration information from the connection class. The configuration information includes connection configuration, sending configuration and receiving configuration;

[0173] The configuration information rewriting module is used to rewrite the connection configuration, the sending configuration and the receiving configuration to generate new configuration information;

[0174] A new connection class creation module is used to create a new connection class according to the new configuration information, and the address of the new connection class is the address of the proxy server;

[0175] The identity information verification module is specifically used to construct a data packet and send the data packet and the proxy request to the proxy server for verification according to the address of the new connection class.

[0176] The above steps 201 to 205 are described in the foregoing description and will not be repeated here.

[0177] The remote control method provided by the embodiment of the present invention comprises a recognition program receiving a remote control instruction sent by a cloud server through a proxy connection, and determining a target application and a control action for the target application from the application to be controlled according to the remote control instruction; further loading an image file of the target application from an image recognition library of a remote terminal; creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application; and finally, executing a control action on the target element. The present invention utilizes image recognition technology to simulate user operations, such as clicking an icon to start an application, operating a menu, etc., so as to complete the scheduling of the target desktop application.

[0178] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0179] A processor 501, a memory 503 for storing processor executable instructions;

[0180] The processor 501 is configured to execute instructions to implement the above remote control method, which is applied to the cloud server:

[0181] Monitor the remote control request sent by the mobile terminal to the remote terminal, and parse the remote control request to obtain the remote control instruction;

[0182] A recognition program for sending a remote control instruction to a remote terminal through a proxy connection, wherein the recognition program is used to determine a target application, a target element of the target application, and a control action for the target application from the application to be controlled according to the remote control instruction;

[0183] The control recognition procedure performs control actions on the target element.

[0184] In addition, the processor 501 is configured to execute instructions and can also implement the above remote control method, which is applied to the identification program:

[0185] Receive remote control instructions sent by the cloud server through the proxy connection;

[0186] Determine a target application and a control action for the target application from the application to be controlled according to the remote control instruction;

[0187] Load the image file of the target application from the image recognition library of the remote terminal;

[0188] Creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application;

[0189] Performs a control action on the target element.

[0190] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0191] The communication interface is used for communication between the above terminal and other devices.

[0192] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0194] In another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the remote control method described in any one of the above embodiments is implemented.

[0195] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0196] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0197] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0198] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A remote control method, characterized in that: Applied to a cloud server, the cloud server is connected to a mobile terminal and a remote terminal, a corresponding proxy connection is configured between the cloud server and the remote terminal, the remote terminal includes an identification program and a plurality of applications to be controlled, the method comprises: monitoring a remote control request sent by the mobile terminal to the remote terminal, and parsing the remote control request to obtain a remote control instruction; sending the remote control instruction to an identification program of the remote terminal through the proxy connection, the identification program being used to determine a target application, a target element of the target application, and a control action for the target application from the application to be controlled according to the remote control instruction; The recognition program is controlled to perform the control action on the target element.

2. The method according to claim 1, characterized in that: The proxy connection includes an encrypted channel and a target proxy connection, and the identification procedure for sending the remote control instruction to the remote terminal through the proxy connection includes: encrypting the remote control instruction through the encryption channel; The encrypted remote control instruction is sent to the identification program of the remote terminal through the target proxy connection.

3. The method according to claim 2, characterized in that The cloud server includes a first encryption algorithm list, the remote terminal includes a second encryption algorithm list, and the encryption channel is established in the following manner: Sending a connection request to the remote terminal, and receiving a response message from the remote terminal in response to the connection request; Establishing an initial connection with the remote terminal according to the response message; exchanging the first encryption algorithm list and the second encryption algorithm list, and selecting a target encryption algorithm from the first encryption algorithm list and the second encryption algorithm list; Generate a shared key of the target encryption algorithm through a password exchange protocol; The encryption channel is established in combination with the initial connection, the target encryption algorithm, and the shared key.

4. A remote control method, characterized in that: The method is applied to an identification program, the identification program is located in a remote terminal, the remote terminal further includes a plurality of applications to be controlled, the remote terminal is connected to a cloud server, and a corresponding proxy connection is configured between the remote terminal and the cloud server. The method includes: Receiving a remote control instruction sent by the cloud server through the proxy connection; Determining a target application and a control action for the target application from the applications to be controlled according to the remote control instruction; Loading the image file of the target application from the image recognition library of the remote terminal; Creating an element template corresponding to the remote control instruction, matching the element template with the image file of the target application, and obtaining a target element corresponding to the target application; The control action is performed on the target element.

5. The method according to claim 4, characterized in that Before receiving the remote control instruction sent by the cloud server through the proxy connection, the method includes: Taking a screenshot of the application to be controlled to obtain an image file of the application to be controlled; The image file of the application to be controlled is stored to generate an image recognition library.

6. The method according to claim 4, characterized in that The proxy connection includes an encrypted channel and a target proxy connection. A proxy server is connected between the mobile terminal and the cloud server. The target proxy connection is established in the following manner: Obtaining a proxy request sent by the application to be controlled; Constructing a data packet, and sending the data packet and the proxy request to the proxy server for verification, wherein the data packet includes the identity information of the application to be controlled; Receiving a verification result of the identity information by the proxy server; When the verification result is that the verification is passed, the target proxy connection is established.

7. The method according to claim 6, characterized in that The identification program is configured with a connection class, the address of the connection class is the address of the identification program, and after obtaining the proxy request sent by the application to be controlled, the method includes: Acquire configuration information from the connection class, the configuration information including connection configuration, sending configuration and receiving configuration; Rewrite the connection configuration, sending configuration and receiving configuration to generate new configuration information; Creating a new connection class according to the new configuration information, where the address of the new connection class is the address of the proxy server; The step of constructing a data packet and sending the data packet and the proxy request to the proxy server for verification includes: A data packet is constructed, and the data packet and the proxy request are sent to the proxy server for verification according to the address of the new connection class.

8. A remote control device, characterized in that: Applied to a cloud server, the cloud server is connected to a mobile terminal and a remote terminal, a corresponding proxy connection is configured between the cloud server and the remote terminal, the remote terminal includes an identification program and a plurality of applications to be controlled, the device includes: A first monitoring module, configured to monitor a remote control request sent by the mobile terminal to the remote terminal, and parse the remote control request to obtain a remote control instruction; A first sending module, configured to send the remote control instruction to an identification program of the remote terminal through the proxy connection, wherein the identification program is configured to determine a target application, a target element of the target application, and a control action for the target application from the application to be controlled according to the remote control instruction; The first control module is used to control the recognition program to perform the control action on the target element.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the remote control method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the remote control method according to any one of claims 1 to 7 is implemented.

Images