Equipment picture capturing and remote control method and system

Abstract

The invention discloses a method and a system for capturing and remotely controlling an equipment picture, in the system, a webpage end and an equipment end respectively establish TCP (Transmission Control Protocol) connection with a server end; the webpage side initiates an RTC connection request to the server side, and the server side forwards the request; the device end obtains a screen video stream, initializes RTC, configures video transmission parameters, and transmits the video stream to the webpage end; the webpage end and the equipment end carry out P2P connection negotiation through SDP and CAN information exchange, and after RTC connection is successfully established, the webpage end receives and displays a video stream of the equipment end; the webpage end monitors the control instruction and the function instruction and transmits the control instruction and the function instruction to the equipment end, and the equipment end executes a corresponding screen response according to the control instruction or the function instruction and sends an execution result to the webpage end to be displayed. Through the technical scheme of the invention, stable and timely interaction experience is provided, the pressure of data transmission and exchange is reduced, the migration cost of multiple types of equipment ends is reduced, and the development and maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of remote control technology, and in particular to a method for device screen capture and remote control and a system for device screen capture and remote control. Background Art

[0002] At present, remote control technology mainly uses FFmpeg to push images, establishes socket connections to achieve end-to-end data exchange, and binds key characters and key positions as data objects. After the controlled end parses, it executes actions to achieve limited control of the device. For example, some remote desktop software can achieve remote control of computers, or remote control of mobile devices.

[0003] The current technology of computer remote control equipment is relatively mature, but in the development process of remote human-machine interaction, the management and control of equipment usually adopts traditional methods, such as remote connection and work through specific software and tools. However, there are many devices and they are widely distributed geographically. At the same time, for developers, ordinary remote connection tools have relatively simple functions and relatively high latency. Moreover, the existing remote control technology only focuses on basic operation control, lacks comprehensive monitoring of device status and lacks rich usage functions, such as intuitive current device status management, command line console, remote file management, and real-time screen UI element acquisition. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and system for device screen capture and remote control, which realizes the communication connection among the web page, server and device through TCP technology, and on this basis realizes end-to-end stable and fast transmission through RTC technology, so that the overall connection tends to be highly stable and low-latency, and can provide a stable and timely interactive experience. The RTC video stream and data channel are separated and do not interfere with each other, reducing the pressure of frequent data transmission and exchange. The input and output data structure based on the unified operation of the shell reduces the migration cost of various types of device ends, and reduces the development and maintenance costs.

[0005] To achieve the above object, the present invention provides a method for device screen capture and remote control, which is applied to a web page, and includes:

[0006] Establish a TCP long connection with the server;

[0007] Initiate a remote control request to the server according to the device selection instruction to obtain real-time image data of the selected device;

[0008] Realize RTC connection with the device side through message interaction with the server side, receive the screen data stream of the device side, so as to display the screen content of the device side;

[0009] Monitor the user's control instructions and transmit them to the device end, so as to perform corresponding operations on the device end;

[0010] Sending operation instructions to the device to realize remote control function;

[0011] Obtain the return data from the device and display the corresponding control result;

[0012] According to the end remote control instruction, the connection with the device is disconnected and the remote operation is stopped.

[0013] In the above technical solution, preferably, the RTC connection with the device side is realized through message interaction with the server side, and the screen data stream of the device side is received, and the specific process includes:

[0014] After receiving the online information of the selected device from the server, static resources are loaded and the RTC connection is initialized;

[0015] Under the coordination of the server, create a PeerConnection instance and initialize the video stream receiving function;

[0016] The screen video stream of the device is obtained according to the onTrack callback method of PeerConnection.

[0017] In the above technical solution, preferably, in the process management, application management and file management interface, a request is sent to the device end through the RTC data channel to obtain the current process, application list or file list, and the corresponding user operation is performed after receiving the response from the device end;

[0018] Send a log request to the device and receive the returned log data, and display the log information according to the set log level and filtering conditions

[0019] A view node request is sent to the device end to obtain a screenshot and node information of the device end for display.

[0020] The present invention also proposes a method for capturing and remotely controlling a device screen, which is applied to a device end and includes:

[0021] Establish a TCP connection with the server and send device status information to the server;

[0022] According to the received RTC connection request from the server, the screen video stream is obtained and the RTC is initialized;

[0023] Initiate data stream transmission based on the received remote control request and send screen image data to the web page;

[0024] According to the control instruction received from the web page, perform corresponding operations according to the control instruction;

[0025] Responding to the control request received from the web page, and implementing the corresponding control function;

[0026] According to the received end control instruction, the connection with the web page and the server is disconnected, and the data stream transmission is stopped.

[0027] In the above technical solution, preferably, the method of device screen capture and remote control specifically includes:

[0028] Start the daemon process and establish a TCP connection with the server, and send device status information to the server through a heartbeat request;

[0029] After receiving the remote control signal from the server, the main process is called up and enters the waiting control state;

[0030] According to the RTC connection request received from the server, the screen video stream is obtained and the RTC is initialized, the video stream transmission parameters are configured, and the video stream is transmitted to the web page;

[0031] Monitor the control instructions in the RTC data channel, parse the mouse and keyboard events sent by the web page, and perform corresponding operations;

[0032] According to the received shell command sent by the web page, after parsing and executing, the execution result is returned to the web page through the RTC data channel;

[0033] In process management and application management, respond to the process and application requests of the web page, perform related operations, and send the results to the web page;

[0034] Execute the file management operation according to the received file management request, and return the result to the web page through the RTC data channel;

[0035] Responding to the log management request, calling the log tool to obtain the log information of the specified conditions, and returning the result to the web page;

[0036] After receiving the view node request, obtain the interface node information through the accessibility node, OCR and object detection, and return the screenshot and node information to the web page;

[0037] According to the received end control instruction of the web page or the server, the RTC connection is closed, the main process resources are destroyed, and the daemon process is recycled.

[0038] The present invention also proposes a method for capturing and remotely controlling a device screen, which is applied to a server and includes:

[0039] Establish TCP connections with the device and web page respectively;

[0040] Receiving and storing the status information of the device end;

[0041] Responding to the remote control request of the web page, and coordinating the web page and the device to establish an RTC connection;

[0042] Necessary connection information is transmitted between the web page end and the device end to establish data stream transmission;

[0043] When the remote control ends, the connection between the web page and the device is disconnected.

[0044] In the above technical solution, preferably, the method of device screen capture and remote control specifically includes:

[0045] Establishing TCP connections with the device and the web page respectively, receiving the heartbeat request and device configuration information from the device, and storing the device status;

[0046] After receiving the remote control request from the web page, confirming the online status of the requested device and notifying the web page;

[0047] After the web page initiates an RTC connection request, an RTC initialization signal is sent to the target device, and after the device responds and returns a confirmation message, the confirmation message is sent to the web page;

[0048] Coordinate the web page to establish a P2P connection with the device, and implement RTC connection negotiation by exchanging SDP and candidate information;

[0049] When the heartbeat detection of the device ends, or the user ends the control, a closing notification is sent to the web page and the device respectively to terminate the connection between the two parties;

[0050] After the remote control operation is completed, the connection resources with the device end and the web page end are recovered according to the system instructions.

[0051] The present invention also proposes a system for device screen capture and remote control, which applies the device screen capture and remote control method disclosed in any one of the above technical solutions, including: a web page end, a device end and a server end;

[0052] The web page end and the device end respectively establish a TCP connection with the server end;

[0053] The web page initiates an RTC connection request to the server, and the server forwards the request to the device after confirming that the device is online;

[0054] The device end obtains the screen video stream and initializes the RTC according to the received RTC connection request, configures the video transmission parameters, and transmits the video stream to the web page end;

[0055] The web page and the device exchange SDP and candidate information to perform P2P connection negotiation. After the RTC connection is successfully established, the web page receives the video stream of the device and displays the screen image of the device;

[0056] The web page monitors the user's control instructions and function instructions, and transmits them to the device through the RTC data channel, and the device executes a corresponding screen response according to the control instructions or the function instructions;

[0057] The device sends the execution result to the web page for display.

[0058] In the above technical solution, preferably, the device side obtains the screen video stream and initializes RTC according to the received RTC connection request, configures video transmission parameters, and transmits the video stream to the web page side. The web page side and the device side exchange SDP and candidate information to perform P2P connection negotiation and successfully establish an RTC connection. The specific process includes:

[0059] The device end obtains the screen video stream and initializes the RTC configuration according to the received RTC connection request;

[0060] Configure video stream encoding and transmission parameters, set RTC configuration, create and initialize PeerConnection data channel;

[0061] Media negotiation is performed based on the PeerConnection data channel, and the device end and the web page end complete the P2P negotiation connection by exchanging SDP information and candidate information.

[0062] In the above technical solution, preferably, the web page monitors the user's control instructions and function instructions, and transmits them to the device through the RTC data channel. The device performs a corresponding screen response according to the control instructions or the function instructions. The specific process includes:

[0063] The web page captures the user's mouse events and keyboard events, and transmits them to the device through the RTC data channel. The device performs corresponding screen responses according to the event type and coordinates or key codes;

[0064] The web page enters the console interface and sends a shell initialization command to the device. The device executes the command and returns the initialization result. According to the command input by the user, the web page sends the command to the device through the RTC data channel. The device executes the command and returns the result to the web page for display.

[0065] The web page sends a process management request to the device, and the device executes the top command and returns the current process information; according to the process closing instruction selected by the user, the web page sends a closing command containing the process ID, and the device executes the command and updates the process information and returns;

[0066] The web page initiates an application list request to the device, the device obtains application information of the current device and returns it, and the web page displays all application items so that the user can perform application operations;

[0067] The web page requests the device file list and carries the path information, and the device executes the command to return the file directory structure; according to the download file selected by the user, the device converts the file into a byte array and sends it to the web page, and according to the deletion operation performed by the user, the device executes the corresponding deletion command;

[0068] The web page sends a log request to the device through the RTC data channel, the device calls the log tool to execute the command and returns the log data, and the web page filters and displays the log data according to the log level, application selection and regular expression set by the user;

[0069] The web page requests the view node information of the device, and the device obtains the current device interface element information based on accessibility nodes, OCR text recognition and YOLO object detection, and encapsulates the screenshot and node information and sends them to the web page for display.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] (1) Strong real-time interactivity: Through RTC technology, remote screen capture and control of devices are achieved. End-to-end connection reduces screen and operation delays, allowing users to control and monitor devices in real time, improving development efficiency and the ability to handle emergencies. Whether in short-distance LAN, medium- and long-distance domestic, or global device control, it can provide a stable and timely interactive experience.

[0072] (2) It is rich in functions and provides many functions that are helpful to software developers, such as console, process management, and application logs, which make it easier for script developers to use node acquisition functions to assist in development. Multiple functional modules meet various needs in the development process, reduce dependence on different tools, and improve work efficiency.

[0073] (3) High compatibility: It can adapt to different platforms and devices, including devices running Windows, Mac, Android, iOS and other systems, allowing developers to manage and control different types of devices on a unified platform, reducing development and maintenance costs.

[0074] (4) High stability: A variety of measures are adopted to protect process connections, ensuring that video streams and control commands can be transmitted in a timely and correct manner. At the same time, heartbeat requests and scheduled connection maintenance ensure the stability and reliability of the system.

[0075] (5) Convenient operation: The user interface is simple and clear, and the operation process is intuitive, so developers can quickly get started to control and manage the device. Whether it is screen control, console operation or other functions, it has a good user experience.

[0076] (6) Flexible configuration: The streaming settings can adjust parameters such as bit rate, FPS, frame size, horizontal and vertical screen window size, etc. to adapt to different network environments and usage requirements. At the same time, the various functions of log filtering conditions and task logs facilitate developers to troubleshoot and debug problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a three-terminal network architecture of a method for capturing and remotely controlling device images disclosed in an embodiment of the present invention;

[0078] Figure 2 A schematic diagram of the functional architecture of a method for capturing and remotely controlling device screens disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0081] like Figure 1 and Figure 2 As shown, a method for capturing and remotely controlling a device screen according to the present invention is applied to a web page, comprising:

[0082] Establish a TCP long connection with the server;

[0083] According to the device selection instruction, a remote control request is initiated to the server to obtain the real-time image data of the selected device;

[0084] Through message interaction with the server, the RTC connection with the device is realized, and the screen data stream of the device is received to display the screen content of the device.

[0085] Monitor the user's control instructions and transmit them to the device to perform corresponding operations on the device;

[0086] Send operation instructions to the device to realize remote control function;

[0087] Get the return data from the device and display the corresponding control results;

[0088] According to the end remote control instruction, disconnect from the device and stop remote operation.

[0089] In this implementation, TCP technology is used to achieve communication connections among the web page, server, and device ends. On this basis, RTC technology is used to achieve end-to-end stable and fast transmission, so that the overall connection tends to be highly stable and low-latency, and can provide a stable and timely interactive experience. The RTC video stream and data channel are separated and do not interfere with each other, reducing the pressure of frequent data transmission and exchange. The input and output data structure based on the unified operation of the shell reduces the migration cost of various types of device ends, and reduces development and maintenance costs.

[0090] Specifically, the web page is the web page remote control terminal, which serves as the user's entry point for using the device screen capture and remote control functions. The device is the controlled smart device terminal, including mainstream devices that can use the Java environment, such as devices with Window, Mac, Android, iOS and other systems. The server is the server that processes the central business data, which is used to connect the web page and the device, establish business connections and data transfer, and act as a signaling server.

[0091] In the above implementation, preferably, the RTC connection with the device is realized by message interaction with the server, and the screen data stream of the device is received. The specific process includes:

[0092] After receiving the online information of the selected device from the server, load the static resources and initialize the RTC connection;

[0093] Under the coordination of the server, create a PeerConnection instance and initialize the video stream receiving function;

[0094] Get the screen video stream of the device according to the onTrack callback method of PeerConnection.

[0095] In the above implementation, preferably, in the process management, application management and file management interface, a request is sent to the device side through the RTC data channel to obtain the current process, application list or file list, and the corresponding user operation is performed after receiving the response from the device side;

[0096] Send log requests to the device and receive the returned log data, displaying the log information according to the set log level and filter conditions

[0097] Send a view node request to the device to obtain the device's screenshot and node information for display.

[0098] The present invention also proposes a method for capturing and remotely controlling a device screen, which is applied to a device end and includes:

[0099] Establish a TCP connection with the server and send device status information to the server;

[0100] According to the RTC connection request received from the server, the screen video stream is obtained and the RTC is initialized;

[0101] Initiate data stream transmission based on the received remote control request and send screen image data to the web page;

[0102] According to the control instruction received from the web page, the corresponding operation is performed according to the control instruction;

[0103] Respond to the control request received from the web page and implement the corresponding control function;

[0104] According to the received end control instruction, the connection with the web page and the server is disconnected and the data stream transmission is stopped.

[0105] In the above implementation, preferably, the method for device screen capture and remote control specifically includes:

[0106] Start the daemon process and establish a TCP connection with the server, and send device status information to the server through heartbeat requests;

[0107] After receiving the remote control signal from the server, the main process is called up and enters the waiting state;

[0108] According to the RTC connection request received from the server, the screen video stream is obtained and the RTC is initialized, the video stream transmission parameters are configured, and the video stream is transmitted to the web page;

[0109] Listen to the control instructions in the RTC data channel, parse the mouse and keyboard events sent by the web page, and perform corresponding operations;

[0110] According to the shell command sent by the web page, the execution result is returned to the web page through the RTC data channel after parsing and executing;

[0111] In process management and application management, respond to process and application requests from the web page, perform related operations, and send the results to the web page;

[0112] Execute file management operations based on the received file management request, and return the results to the web page through the RTC data channel;

[0113] Respond to log management requests, call log tools to obtain log information of specified conditions, and return the results to the web page;

[0114] After receiving the view node request, the interface node information is obtained through accessibility nodes, OCR and object detection, and the screenshot and node information are returned to the web page;

[0115] According to the end control instruction received from the web page or server, the RTC connection is closed, the main process resources are destroyed, and the daemon process is recycled.

[0116] The present invention also proposes a method for capturing and remotely controlling a device screen, which is applied to a server and includes:

[0117] Establish TCP connections with the device and web page respectively;

[0118] Receive and store device status information;

[0119] Respond to remote control requests from the web page and coordinate the establishment of RTC connection between the web page and the device;

[0120] Pass necessary connection information between the web page and the device to establish data stream transmission;

[0121] When remote control ends, disconnect the web page and the device.

[0122] In the above implementation, preferably, the method for device screen capture and remote control specifically includes:

[0123] Establish TCP connections with the device and web page respectively, receive the heartbeat request and device configuration information from the device, and store the device status;

[0124] After receiving the remote control request from the web page, confirm the online status of the requested device and notify the web page;

[0125] After the web page initiates an RTC connection request, it sends an RTC initialization signal to the target device. After the device responds and returns a confirmation message, it sends the confirmation information to the web page.

[0126] Coordinate the establishment of a P2P connection between the web page and the device, and implement RTC connection negotiation through the exchange of SDP and candidate information;

[0127] When the heartbeat detection on the device side times out or the user ends control, a closing notification is sent to the web page side and the device side respectively to terminate the connection between the two sides;

[0128] After the remote control operation is completed, the connection resources with the device and web pages are recovered according to system instructions.

[0129] According to the device screen capture and remote control methods disclosed in the above embodiments, which are respectively applied to the web page side, the device side and the server side, the steps of establishing a TCP connection between the three sides include:

[0130] (1) The device starts a daemon process, which is used to communicate with the server and maintain the life cycle of the main process on the device. When receiving the remote control signal from the server, the main process is awakened. When the control ends, the main process is destroyed and resources are recycled. The main process and the daemon process use LocalSocket to establish inter-process communication.

[0131] (2) The device-side daemon establishes a TCP connection with the server, and sends a heartbeat request to the server every minute (the data sent includes information such as device ID, heartbeat interval, device model, etc.), and at the same time sends a request to the server to obtain the configuration file, which will obtain the TURN server information and the current IP address. When the TCP connection is successfully established, the daemon starts listening and waits for the server to transmit the task signal.

[0132] (3) The web page is similar to the device side. It will first establish a TCP long connection with the server. When the device is selected for remote control and the remote control page is entered, the static resources of the page will be loaded first, and then the RTC function will be initialized immediately, and the status of the PeerConnection connection and the video stream channel will be monitored and forwarded.

[0133] At this point, the web page, server, and device have successfully established a TCP connection, with the server as the center to coordinate the web page and device.

[0134] On this basis, the web page establishes an RTC connection with the device, and the web page displays the device screen. The specific process includes:

[0135] (1) The web page initiates an "openrtc" request (TCP) to the server. After the server finds the device that matches the web page request (determined by the device ID), it first sends a "ping" message to the device and then starts waiting. If the "ping" message is not returned to the server at this time, a timeout occurs, and the server sends an "openrtcerror" to the web page to inform it that the RTC opening failed. The web page waits for the listener, and after receiving the failed rest, it initiates the "openrtc" request again after an interval of 3 seconds until a connection is successfully established.

[0136] (2) When the device receives the "ping" message, it returns it to the server intact. Then the server sends the "openrtc" message to the device again, indicating that the device has started the RTC connection process.

[0137] (3) After receiving the "openrtc" message, the device needs to obtain the screen video stream first. Different device systems have different ways of obtaining it. Basically, you need to obtain the screen recording authorization first, and then obtain the video stream through the interface provided by the system. Taking Android system devices as an example, after obtaining the MEDIA_PROJECTION_SERVICE service, you can jump to the application page by obtaining the screen recording intent. After clicking Confirm, the system will attach the instance object of the screen video stream in the callback method.

[0138] (4) Get the screen video stream object, and then start RTC initialization. First, release all possible connections. This step is for thorough initialization. Then bind the message body identifier to use the WebRtcListener interface to listen to messages in the data stream and map functions by category. The above process implements message category monitoring for functions such as interface display, screen touch actions, ADB commands, clock calibration, screen information, file management, command line (shell) tools, log management, system operation status, and application management.

[0139] (5) Then create a hardware video source and add texture rendering. Since the screen is remotely monitored and controlled, when setting up RTC, only one-way video stream flow is considered, that is, the device transmits the video stream to the web page. Although the web page does not transmit media to the device, SDP exchange is also required. Here, the audio and video codec settings of the RTC Peer Connection are pre-set, the default codec is used, H.264 video encoding is enabled, hardware acceleration is performed, and the current video bit rate, frame rate and frame size are set.

[0140] (6) Set up RTC. ICE configuration, use the TRUN information and IP address carried in the aforementioned heartbeat request to configure the STUN server and TRUN server. More configurations are as follows: disable CPU overload detection; disable TCP candidate strategy (that is, only UDP transmission); set the continuous collection strategy in the RTC configuration to continuous collection mode, so that ICE candidates are continuously collected to improve connection stability when the network changes; set DtlsSrtp to enable, and encrypt transmission to improve security. Set media constraints: disable audio reception proposals; enable video reception proposals; turn on DTLS to ensure secure communication with the browser; turn off CPU automatic frame rate calculation to prevent screen freezes.

[0141] (7) Create a PeerConnection, pass in the aforementioned RTC configuration, media constraints, and callback interface implementation to monitor various state changes of the connection. While waiting for the event to arrive, set the previously acquired screen recording media stream (VideoTrack) to the PeerConnection.

[0142] (8) Then proceed with the initialization process of the PeerConnection data channel. Create a WebRtcListener interface to monitor the data channel. Use DataChannel.registerObserver to obtain the current data channel status in the onStateChange method and call back to the instance of the WebRtcListener interface (corresponding to (4) above). In the onMessag method, listen for RTC messages. The message is a byte array and is parsed according to the following protocol format: 0-5 bits are flags, starting with "HEARD"; 5-9 bits describe the data size (size); 9-size+9 is the data body; 9-size+9 bits describe the file data size (bytesFileSize); size+9+4-size+9+4+bytesFile bits are the file data body. After obtaining the message, judge based on the data size and the actual size, and parse it if they are consistent. Parse the data into json format and call back to the instance of the WebRtcListener interface (corresponding to (4) above).

[0143] (9) After PeerConnection is created, the media negotiation step is performed. PeerConnection is used to create Offer. After the creation is successful, the SDP (Session Description Protocol) information of this device is obtained. Then TCP is used to send the SDP to the server, named "offer", which the server will forward to the web page. At the same time, a new thread is opened to write the local SDP into the LocalDescription configuration.

[0144] (10) After receiving the "offer" message, the web page obtains the SDP information of the device and writes it into the RemoteDescription of the web page RTC. The web page creates a response, obtains the local SDP information, and writes it into the LocalDescription of the web page RTC. At the same time, the local SDP information is sent as data to the device via TCP as "answer".

[0145] (11) After receiving the "answer", the device writes the SDP information of the local web page into RemoteDescription, and then the onIceCandidate event listener set by PeerConnection obtains the candidate information and sends the "candidate" information to the web page. Similarly, after setting RemoteDescription, the web page will also listen to the onicecandidate event of PeerConnection, obtain the candidate information, and send the "candidate" to the device. In this way, both ends get the candidate information of the other party, and then add it to the local end through addIceCandidate, and the two sides establish a P2P connection.

[0146] (12) After the connection is successfully established, the web page obtains the screen video stream according to the onTrack callback method of PeerConnection, displays the screen in the Video tag, and automatically plays the streaming media.

[0147] At this point, the dual-end RTC is successfully established and the device screen is successfully displayed.

[0148] The present invention also provides a system for device screen capture and remote control, which applies the device screen capture and remote control method disclosed in any one of the above embodiments, including: a web page end, a device end, and a server end;

[0149] The web page and device establish TCP connections with the server respectively;

[0150] The web page initiates an RTC connection request to the server, and the server forwards the request to the device after confirming that the device is online;

[0151] The device obtains the screen video stream and initializes the RTC according to the received RTC connection request, configures the video transmission parameters, and transmits the video stream to the web page.

[0152] The web page and the device exchange SDP and candidate information to negotiate a P2P connection. After the RTC connection is successfully established, the web page receives the video stream from the device and displays the screen image of the device.

[0153] The web page monitors the user's control instructions and function instructions, and transmits them to the device through the RTC data channel. The device executes the corresponding screen response according to the control instructions or function instructions;

[0154] The device sends the execution results to the web page for display.

[0155] In the above implementation, preferably, the device side obtains the screen video stream and initializes the RTC according to the received RTC connection request, configures the video transmission parameters, and transmits the video stream to the web page side. The web page side and the device side exchange SDP and candidate information to perform P2P connection negotiation and successfully establish the RTC connection. The specific process includes:

[0156] The device obtains the screen video stream and initializes the RTC configuration based on the received RTC connection request;

[0157] Configure video stream encoding and transmission parameters, set RTC configuration, create and initialize PeerConnection data channel;

[0158] Media negotiation is performed based on the PeerConnection data channel. The device and the web page complete the P2P negotiation connection by exchanging SDP information and candidate information.

[0159] In the above implementation, preferably, the web page monitors the user's control instructions and function instructions, and transmits them to the device through the RTC data channel. The device performs a corresponding screen response according to the control instructions or function instructions. The specific process includes:

[0160] The web page captures the user's mouse events and keyboard events, and transmits them to the device through the RTC data channel. The device performs the corresponding screen response according to the event type and coordinates or key code;

[0161] The web page enters the console interface and sends the shell initialization command to the device. The device executes the command and returns the initialization result. According to the command entered by the user, the web page sends the command to the device through the RTC data channel. After the device executes the command, the result is returned to the web page for display.

[0162] The web page sends a process management request to the device, and the device executes the top command and returns the current process information. According to the process closing instruction selected by the user, the web page sends a closing command containing the process ID, and the device executes the command and updates the process information and returns.

[0163] The web page sends an application list request to the device. The device obtains the application information of the current device and returns it. The web page displays all application items so that users can perform application operations.

[0164] The web page requests the device file list and carries the path information. The device executes the command and returns the file directory structure. According to the download file selected by the user, the device converts the file into a byte array and sends it to the web page. According to the deletion operation performed by the user, the device executes the corresponding deletion command.

[0165] The web page sends a log request to the device through the RTC data channel. The device calls the log tool to execute the command and returns the log data. The web page filters and displays the log data according to the log level, application selection and regular expression set by the user.

[0166] The web page requests the view node information from the device. The device obtains the current device interface element information based on accessibility nodes, OCR text recognition, and YOLO object detection, and encapsulates the screenshot and node information and sends them to the web page for display.

[0167] According to the device screen capture and remote control method and system disclosed in the above embodiments, based on the three-terminal communication connection and data interaction mode, the specific implementation methods of the following functions include:

[0168] (1) Screen control: The web page monitors the mouse "press", "move", "lift", "leave" and "scroll wheel sliding" events and the keyboard key "press" and "lift" events in the video window. These events are sent to the device through the RTC data channel. The device performs consistent response operations based on the event type and value (coordinates, key code).

[0169] (2) Console implementation: The web page uses the xtermjs framework as the window for command line input and result display, and the device uses the terminal tool (Windows uses cmd, macOS and Linux directly use shell, Android uses termux, and iOS uses libmobiledevice) as the executor of the shell script. When the web page enters the console page, the web page sends the shell initialization command to the device. After the initialization is successful, the device informs the web page that shell operations can be performed. When the shell command is entered and the Enter key is pressed, the current command is sent to the device. The device parses the shell command and sends it to the executor for execution. The device sends the execution result (byte array) to the web page, and the web page parses the content and updates it to the xtermjs framework. All of the above data / message transmission is implemented through the RTC data channel.

[0170] (3) Process management: The implementation logic of process management is similar to the above (2) console implementation, and the top command in the shell is used. The web page sends a process management message to the device. After the device executes the top command, it sends the execution result to the web page for parsing. The web page displays the device status in a format. Since the top command carries the process id, if the web page wants to close the process, it can send a message to the device to close the process (kill-9 process id), carrying the selected process id. While the device executes the shell, it will execute the top command again and return the latest process information to the web page. All of the above data / message transmission is implemented through the RTC data channel.

[0171] (4) Application management: The web page sends a message to the device to obtain the application list. The device returns the package name and application name of all applications on the current device to the web page for display through the package management program. For overall application management, we provide the function options of "Get all applications", "Get background applications", "Get foreground running applications" and "Install applications". For the application list, we provide the function options of "Open application", "Open application settings", "Close application" and "Uninstall application" for the item. The above functions are all implemented by the device package management tool, and the two-way communication is implemented using the RTC data channel.

[0172] (5) File management: When the web page enters the file management page, it sends a message to the client to obtain the file list, and carries the target folder path. The device executes the command: ls-la "path". After execution, the result (byte array) is sent to the web page for parsing, and the name and permission creation time of the file and folder can be obtained. Each file item has added "download" (the device converts the obtained file into a byte array and sends it to the web page, refer to the RTC message data format) and "delete" (execute rm-rf "file path") functions. For mobile devices (Android and iOS) on the device side, the media file type (picture, audio and video) has an additional broadcast function. After clicking, this function notifies the system to scan the media file so that the device media library can recognize and display the file.

[0173] (6) Application log: The implementation principle of this function is similar to that of (2) console. The log tool on the device is used to output the log by calling the shell command, and the output result is converted into bytes and sent to the web page for parsing. Three filter conditions, "log level", "select application" and "regular expression", are added to the web page to filter the log. These three filter conditions are spliced ​​into the parameters of the shell command and sent to the device.

[0174] (7) Node elements: After the web page sends a message to the device to obtain the view node. The device encapsulates the node elements that can be obtained by the current device, returns the current device screenshot and the node (including class name, boundary information, width and height, text information, description, package name, visibility, etc.) information list to the web page for display. For node acquisition, take Android devices as an example: use accessibility nodes to obtain basic node information, use OCR text recognition to obtain text information on the screen, and use YOLO object detection to identify specific element nodes. Through boundary information and text description, the three types of node information obtained are sorted, deduplicated, and the parent-child relationship is reconstructed.

[0175] The method and system for capturing and remotely controlling device images disclosed in the above-mentioned implementation manner are applied to the digital human platform of our company to assist in the development of business scripts during implementation. The specific development process includes:

[0176] 1) After getting a new business script task, select a device and click on the remote connection to enter the remote control page. Display the current device screen in real time and in high definition. Use the mouse to click, slide, long press and other operations on the screen. The screen is transmitted smoothly and the interaction is updated smoothly and in time. Through manual use, find out the possible solutions for script execution. By clicking on the Get Node Information function, get the node information of the login button on the current screen, assist in writing business scripts, and publish tasks through the company's business system.

[0177] 2) After the task was executed, it was found that an Android device was running abnormally, and the console function was used to investigate. Using the process management function, you can view all processes running on the current device that are related to the company's operations. Use the application log to view the current log information, execute the task again, and observe the current task execution status and possible error information. After observation, it was found that the click event did not trigger the relevant UI interaction as expected. Use the shell to view the screen size (execute the wmsize command) and find that the screen size is not the expected size. After modification (execute the wm size1080x1920 command), it was executed again and the program ran successfully. Modify the business script to modify the screen to the specified size when the task is initialized, and the error is successfully fixed.

[0178] The example sets the image quality: the bit rate level is original, the frame rate (FPS) is original speed, and the frame size is 1.5 times.

[0179] Example 1: Remote control of short-distance devices in a local area network:

[0180] In the development of business scripts, there are development and testing devices in the company's internal LAN, and this technology is used for remote control. Open the browser, enter the web page, and successfully connect to the specified device. The average RTC delay of the current device is stable within 15ms (calculated as: the time it takes to send a ping from the web page to the device, and then the time it takes for the device to return to the web page / 2). When operating the device, you can experience smooth tracking, clear images, and timely responses.

[0181] Example 2: Domestic medium and long distance device control:

[0182] For remote connection of devices distributed throughout the country, TRUN server is used for relay, which stabilizes the average delay within 70ms. When operating the equipment, the connection is stable, the picture is clear, and there is a slight delay.

[0183] Example 3 Overseas global equipment control:

[0184] When connecting to devices around the world remotely, multiple TRUN servers are used for relaying, and the average delay is stable within 350ms. The connection is stable, the picture is clear, and the delay is slightly higher. The file management transfer speed is slightly slower than the previous two instances. After using the global network proxy, the delay approaches the delay speed of the proxy server. Taking Thailand as an example, a proxy server with a delay of 100ms located in Hong Kong, China is used for network proxy. The overall delay will drop to 120ms, which is similar to the experience of mainland devices.

[0185] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for capturing and remotely controlling a device screen, applied to a web page, characterized in that: include: Establish a TCP long connection with the server; Initiate a remote control request to the server according to the device selection instruction to obtain real-time image data of the selected device; Realize RTC connection with the device side through message interaction with the server side, receive the screen data stream of the device side, so as to display the screen content of the device side; Monitor the user's control instructions and transmit them to the device end, so as to perform corresponding operations on the device end; Sending operation instructions to the device to realize remote control function; Obtain the return data from the device and display the corresponding control result; According to the end remote control instruction, the connection with the device is disconnected and the remote operation is stopped.

2. The method for device screen capture and remote control according to claim 1, characterized in that: The RTC connection with the device side is realized by message interaction with the server side, and the screen data stream of the device side is received. The specific process includes: After receiving the online information of the selected device from the server, static resources are loaded and the RTC connection is initialized; Under the coordination of the server, create a PeerConnection instance and initialize the video stream receiving function; The screen video stream of the device is obtained according to the onTrack callback method of PeerConnection.

3. The method for device screen capture and remote control according to claim 1, characterized in that: In the process management, application management and file management interface, a request is sent to the device through the RTC data channel to obtain the current process, application list or file list, and a corresponding user operation is performed after receiving a response from the device; Send a log request to the device and receive the returned log data, and display the log information according to the set log level and filtering conditions A view node request is sent to the device end to obtain a screenshot and node information of the device end for display.

4. A method for capturing and remotely controlling a device screen, applied to a device, characterized in that: include: Establish a TCP connection with the server and send device status information to the server; According to the received RTC connection request from the server, the screen video stream is obtained and the RTC is initialized; Initiate data stream transmission based on the received remote control request and send screen image data to the web page; According to the control instruction received from the web page, perform corresponding operations according to the control instruction; Responding to the control request received from the web page, and implementing the corresponding control function; According to the received end control instruction, the connection with the web page and the server is disconnected, and the data stream transmission is stopped.

5. The method for device screen capture and remote control according to claim 4, characterized in that: Specifically include: Start the daemon process and establish a TCP connection with the server, and send device status information to the server through a heartbeat request; After receiving the remote control signal from the server, the main process is called up and enters the waiting control state; According to the RTC connection request received from the server, the screen video stream is obtained and the RTC is initialized, the video stream transmission parameters are configured, and the video stream is transmitted to the web page; Monitor the control instructions in the RTC data channel, parse the mouse and keyboard events sent by the web page, and perform corresponding operations; According to the received shell command sent by the web page, after parsing and executing, the execution result is returned to the web page through the RTC data channel; In process management and application management, respond to the process and application requests of the web page, perform related operations, and send the results to the web page; Execute the file management operation according to the received file management request, and return the result to the web page through the RTC data channel; Responding to the log management request, calling the log tool to obtain the log information of the specified conditions, and returning the result to the web page; After receiving the view node request, obtain the interface node information through the accessibility node, OCR and object detection, and return the screenshot and node information to the web page; According to the received end control instruction of the web page or the server, the RTC connection is closed, the main process resources are destroyed, and the daemon process is recycled.

6. A method for capturing and remotely controlling a device screen, applied to a server, characterized in that: include: Establish TCP connections with the device and web page respectively; Receiving and storing the status information of the device end; Responding to the remote control request of the web page, and coordinating the web page and the device to establish an RTC connection; Necessary connection information is transmitted between the web page end and the device end to establish data stream transmission; When the remote control ends, the connection between the web page end and the device end is disconnected.

7. The method for device screen capture and remote control according to claim 6, characterized in that: Specifically include: Establishing TCP connections with the device and the web page respectively, receiving the heartbeat request and device configuration information from the device, and storing the device status; After receiving the remote control request from the web page, confirming the online status of the requested device and notifying the web page; After the web page initiates an RTC connection request, an RTC initialization signal is sent to the target device, and after the device responds and returns a confirmation message, the confirmation message is sent to the web page; Coordinate the web page to establish a P2P connection with the device, and implement RTC connection negotiation by exchanging SDP and candidate information; When the heartbeat detection of the device ends, or the user ends the control, a closing notification is sent to the web page and the device respectively to terminate the connection between the two parties; After the remote control operation is completed, the connection resources with the device end and the web page end are recovered according to the system instructions.

8. A system for capturing and remotely controlling device images, characterized in that: The method for capturing and remotely controlling a device screen according to any one of claims 1 to 7 is applied, comprising: a web page end, a device end, and a server end; The web page end and the device end respectively establish a TCP connection with the server end; The web page initiates an RTC connection request to the server, and the server forwards the request to the device after confirming that the device is online; The device end obtains the screen video stream and initializes the RTC according to the received RTC connection request, configures the video transmission parameters, and transmits the video stream to the web page end; The web page and the device exchange SDP and candidate information to perform P2P connection negotiation. After the RTC connection is successfully established, the web page receives the video stream of the device and displays the screen image of the device; The web page monitors the user's control instructions and function instructions, and transmits them to the device through the RTC data channel, and the device executes a corresponding screen response according to the control instructions or the function instructions; The device sends the execution result to the web page for display.

9. The device screen capture and remote control system according to claim 8, characterized in that: The device side obtains the screen video stream and initializes RTC according to the received RTC connection request, configures video transmission parameters, and transmits the video stream to the web page side. The web page side and the device side exchange SDP and candidate information to perform P2P connection negotiation and successfully establish an RTC connection. The specific process includes: The device end obtains the screen video stream and initializes the RTC configuration according to the received RTC connection request; Configure video stream encoding and transmission parameters, set RTC configuration, create and initialize PeerConnection data channel; Media negotiation is performed based on the PeerConnection data channel, and the device end and the web page end complete the P2P negotiation connection by exchanging SDP information and candidate information.

10. The device screen capture and remote control system according to claim 9, characterized in that: The web page monitors the user's control instructions and function instructions, and transmits them to the device through the RTC data channel. The device performs a corresponding screen response according to the control instructions or the function instructions. The specific process includes: The web page captures the user's mouse events and keyboard events, and transmits them to the device through the RTC data channel. The device performs corresponding screen responses according to the event type and coordinates or key codes; The web page enters the console interface and sends a shell initialization command to the device. The device executes the command and returns the initialization result. According to the command input by the user, the web page sends the command to the device through the RTC data channel. The device executes the command and returns the result to the web page for display. The web page sends a process management request to the device, and the device executes the top command and returns the current process information; according to the process closing instruction selected by the user, the web page sends a closing command containing the process ID, and the device executes the command and updates the process information and returns; The web page initiates an application list request to the device, the device obtains application information of the current device and returns it, and the web page displays all application items so that the user can perform application operations; The web page requests the device file list and carries the path information, and the device executes the command to return the file directory structure; according to the download file selected by the user, the device converts the file into a byte array and sends it to the web page, and according to the deletion operation performed by the user, the device executes the corresponding deletion command; The web page sends a log request to the device through the RTC data channel, the device calls the log tool to execute the command and returns the log data, and the web page filters and displays the log data according to the log level, application selection and regular expression set by the user; The web page requests the view node information of the device, and the device obtains the current device interface element information based on accessibility nodes, OCR text recognition and YOLO object detection, and encapsulates the screenshot and node information and sends them to the web page for display.

Images