Remote desktop implementation method and network video recording equipment

Abstract

The invention relates to the technical field of embedded systems, in particular to a remote desktop implementation method and network video recording equipment, which are suitable for a server and a client in an embedded system environment. The server is responsible for responding to a control instruction of the remote desktop, and the client is responsible for initiating a control operation of the remote desktop. The server side registers a connection interface, a disconnection interface and an operation interface in advance so as to realize modular division of remote desktop functions. After the client and the server successfully establish network connection, the remote video stream is received by calling the connection interface, and the operation interface is called for remote control; and then the client can call the disconnection interface to close the network connection. According to the remote desktop implementation method and the network video recording equipment, a complete remote desktop control closed loop between the client and the server is constructed, so that the embedded equipment has the capability of independently completing a complex remote desktop control task, the dependence on a PC (Personal Computer) is reduced, and the hardware investment and the deployment space are greatly saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of embedded systems, and particularly relates to a remote desktop implementation method and a network video recording device. BACKGROUND

[0002] At present, remote desktop technology is widely used in communication between a personal computer (PC) and a PC, or between a PC and an embedded device. As a typical embedded device, a network video recording device (NVR) mainly has video acquisition, storage and management functions, but due to its relatively limited computing and interaction capabilities, it is difficult to independently complete a complex remote desktop control task. Therefore, the existing remote desktop implementation scheme of the NVR usually relies on a PC as a master control terminal. As a powerful master control device, the PC can run a remote desktop client software. A user inputs a unique identification number (device UID number), a username and a password of the NVR on the PC terminal, and then uses a peer-to-peer (P2P) protocol to establish a network connection with the NVR, so as to realize remote access and control of the NVR. The user can watch the video screen captured by the NVR in real time on the PC terminal, and operate the interface through a mouse and other input devices, so as to complete preview and configuration management of the front-end camera. However, the current remote desktop implementation scheme of the embedded device which generally relies on the PC as the master control device still has some deficiencies. The main problem is that the introduction of the additional PC device not only increases the hardware cost of the system, but also occupies additional deployment space, thereby limiting the flexibility and economy of the system. SUMMARY

[0003] In view of this, the present application provides a remote desktop implementation method and a network video recording device, aiming at solving the problem that the existing embedded device is difficult to independently complete a complex remote desktop control task.

[0004] To solve the above problems, the present application adopts the following technical scheme: In a first aspect, the present application provides a remote desktop implementation method suitable for a server and a client in an embedded system environment, wherein the server is responsible for responding to control instructions of the remote desktop, and the client is responsible for initiating control operations of the remote desktop, including: The server registers a connection interface, a disconnection interface and an operation interface, so as to be called by the client and realize the remote desktop function; The client actively establishes a network connection with the server and confirms that the network connection is successful; The client calls the connection interface, and the server synthesizes a local video stream and transmits the local video stream to the client. The client receives a remote video stream and decodes and renders the remote video stream. When the client calls the disconnection interface, the client actively disconnects the network connection with the server, and the server passively disconnects the network connection with the client and releases corresponding resources. When the client calls the operation interface, the client collects a local mouse operation instruction and sends the local mouse operation instruction to the server. The server receives the mouse operation instruction and executes a corresponding mouse operation event.

[0005] Further, the server synthesizes a local video stream and transmits the local video stream to the client, comprising: The server checks a state of an encoding channel and acquires a currently activated encoding channel ID. If it is checked that the state is an unstarted state, a video processing subsystem and a video encoder are created and initialized, and the video processing subsystem is bound with the video encoder and the video processing subsystem is bound with a video output module of the server. The video output module of the server can write current displayed video data back to a frame buffer and capture original video data from the frame buffer. A video stream acquisition and encoding thread is started, and a video frame callback function is registered. The video frame callback function is responsible for receiving encoding data output by the video encoder and encapsulating the encoding data into a standard streaming media format. When the video encoder generates a new encoded video frame, the video frame callback function is automatically called to process the encoding data, so as to complete encapsulation of the encoded video frame into the standard streaming media format. The encapsulated video data is transmitted to an upper network transmission module, and the video data is transmitted to the client based on a network transmission protocol. The state of the encoding channel is updated to a connected state.

[0006] Further, the server passively disconnects the network connection with the client and releases corresponding resources, comprising: The server checks the state of the encoding channel and confirms that the state is a connected state. A mouse operation instruction receiving thread is closed, the mouse operation instruction of the client is stopped from being received, and related resources are released. A network connection corresponding to the encoding channel is closed, and network communication related resources are released. A video stream acquisition and encoding thread is stopped, and internal buffer and thread resources of the video encoder are released. unbinding the video processing subsystem and the video encoder, and the video processing subsystem and the video output module of the server, and destroying the resources occupied by the video processing subsystem and the video encoder; revoking the registration of the video frame callback function by passing a NULL parameter; updating the state of the encoding channel to a closed state.

[0007] Further, the server receives the mouse operation instruction and executes the corresponding mouse operation event, including: The server receives the mouse operation instruction sent by the client through the signaling channel; wherein the mouse operation instruction includes a moving operation instruction, a clicking operation instruction and a scroll wheel operation instruction; Parsing the mouse operation instruction, determining the type of the mouse operation instruction, and extracting the normalized coordinates of the mouse operation instruction; Mapping the normalized coordinates to the actual screen resolution of the server and calculating the corresponding pixel coordinates; Calling the API of the GUI system of the server and simulating the mouse operation behavior corresponding to the mouse operation instruction based on the calculated pixel coordinates.

[0008] Further, the client actively establishes a network connection with the server and confirms that the network connection is successful, including: When the client receives the identity information input through the user interface, the client completes the creation of the network socket and establishes the underlying network connection with the server through the P2P protocol; wherein the identity information includes the username, password and device UID number of the server responsible for responding to remote desktop control instructions; The client calls the connection interface registered by the server and passes the identity information and connection parameters, and then the client and the server perform a stream media protocol handshake to establish an application layer connection; wherein the connection parameters include the encoding format, maximum resolution and frame rate limit; When the client receives the response information returned by the server after completing the identity verification based on the identity information, the client confirms that the network connection is successful, updates the connection state of the client to a connected state, and starts the video stream receiving thread and the mouse operation instruction sending thread; wherein the response information includes the encoding channel ID and the encoding parameters, and the network connection success is the application layer connection success.

[0009] Further, the client receives the remote video stream and decodes and renders it, including: The client starts the remote video stream receiving and decoding thread, creates and initializes the video decoder and renderer; receiving the video data in a standard streaming media format sent by the server by calling a connection interface registered by the server, and decoding the video data by calling an interface of the video decoder to obtain decoded YUV format frame data; scaling and adapting the YUV format frame data according to a difference between a resolution of the YUV format frame data and a video window size of the client by using a preset scaling algorithm; mapping the scaled and adapted YUV format frame data to a display layer of a video output module of the client, and calling a hardware acceleration interface of the renderer to perform rendering, so as to realize a remote desktop display function; wherein the rendering supports dynamic adjustment of a rendering resolution according to a change in the resolution of the input YUV format frame data or the video window size of the client.

[0010] Further, the client actively disconnects the network connection with the server, comprising: the client displays a disconnect button on a display screen of the client; when the disconnect button receives a trigger instruction, the client sends a disconnection request to the server by calling a disconnection interface registered by the server; stopping the remote video stream receiving and decoding thread, and closing a network socket; releasing resources of the video decoder and the renderer; stopping the mouse operation instruction sending thread, and stopping sending mouse operation instructions to the server through a signaling channel; updating a connection state of the client to a disconnected state.

[0011] Further, the client collects local mouse operation instructions and sends the mouse operation instructions to the server, comprising: the client listens to a local mouse operation event, and captures mouse position coordinates, mouse key states and mouse wheel scrolling amounts of the mouse operation event through an API of a GUI system of the client; wherein the mouse operation event comprises a moving operation event, a clicking operation event and a wheel operation event; converting the captured mouse position coordinates into normalized coordinates defined by the client; packaging mouse operation instructions; wherein the packaged mouse operation instructions comprise a type of the mouse operation event and corresponding normalized coordinates; sending the packaged mouse operation instructions to an operation interface registered by the server based on a TCP protocol or a UDP protocol.

[0012] Further, after the client actively establishes a network connection with the server and confirms that the network connection is successful, the client further comprises: The client continuously monitors the heartbeat packets sent by the server. If the heartbeat packet is detected to be lost, it is confirmed that the current network connection has been lost, and the client triggers the network automatic reconnection mechanism; wherein, the reconnection interval of the network automatic reconnection mechanism gradually increases; Once the network reconnection is confirmed to be successful, the video stream receiving thread and the mouse operation command sending thread are restored; however, when restoring the video stream receiving thread, the encoding parameters need to be renegotiated with the server.

[0013] Secondly, the present invention also proposes a network video recording device, operating in an embedded system environment, wherein the network video recording device is a server responsible for responding to remote desktop control commands or a client responsible for initiating remote desktop control operations, comprising: Memory, used to store program instructions; and A processor for executing the program instructions to implement the steps of the remote desktop implementation method as described in the first aspect.

[0014] Compared with existing technologies, the advantages of this invention are as follows: a remote desktop implementation method and a network video recording device are applicable to server and client environments in embedded systems. The server is responsible for responding to remote desktop control commands, and the client is responsible for initiating remote desktop control operations. The server pre-registers connection interfaces, disconnection interfaces, and operation interfaces to achieve modular division of remote desktop functions. After the client and server successfully establish a network connection, the client receives and decodes the remote video stream by calling the connection interface, and the server transmits the synthesized local video stream to the client through the connection interface. During the remote desktop session, the client sends the acquired local mouse operation commands to the server by calling the operation interface, and the server receives and executes the corresponding remote mouse operation commands through the operation interface. When the client actively disconnects the network connection with the server by calling the disconnection interface, the server passively disconnects the network connection with the client and releases the corresponding resources through the disconnection interface. This remote desktop implementation method and network video recording device construct a complete remote desktop control closed loop between the client and the server, enabling embedded devices to independently complete complex remote desktop control tasks, reducing dependence on PC devices, eliminating the need for additional PC devices, greatly saving hardware investment and deployment space, suitable for various application scenarios, and offering strong implementation flexibility. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not intended to limit the present application, the scope of which is limited only by the claims. Further, in the drawings, the same reference numbers are used to denote the same components. In the drawings: Figure 1 a flowchart of a first embodiment of a remote desktop implementation method of the present application; Figure 2 a flowchart of a second embodiment of a remote desktop implementation method of the present application. DETAILED DESCRIPTION

[0016] The embodiments of the present application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all the other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0017] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement condition, etc. between the components in a certain posture (as shown in the drawings). If the certain posture changes, the directional indications also change accordingly.

[0018] In addition, if the embodiments of the present application involve descriptions of "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features with "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of those of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0019] It should be noted that NVR (Network Video Recorder) is a device or system that receives, stores and manages video monitoring data through the network. It is responsible for recording, playing back and centrally managing video streams from network cameras, supporting remote access and multi-channel video synchronization processing, and is widely used in video monitoring and security systems. VPSS (Video Processing Subsystem) is a module responsible for pre-processing and image enhancement of video signals. VENC (Video Encoder) is a module that compresses and encodes video data into a specified format for storage and transmission. VO (Video Output Module) is a module that outputs processed video data to a display device. WBC (Write Back Controller) is responsible for writing processed image data back to memory or cache for subsequent processing or storage. Frame Buffer is a memory area used to store a frame of image data, supporting image display and processing. YUV is a video color space format commonly used in video data processing, which separates the brightness information (Y) and chrominance information (U, V) of an image to facilitate video data processing and compression. GUI system (Graphical User Interface System) is a system that allows users to interact with computers or embedded devices through graphical means. API (Application Programming Interface) is a set of predefined functions and specifications that software developers can call to implement specific functions. UI (User Interface) is a module in the system responsible for interacting with users, providing operation entry and information display to ensure that users can easily and intuitively control and use system functions. P2P (Peer-to-Peer) is a network communication method that allows two terminals to establish a direct connection for data transmission to reduce the load of intermediate servers and improve transmission efficiency and real-time performance. TCP (Transmission Control Protocol) is a connection-oriented transport layer protocol that provides reliable data transmission services to ensure that data packets arrive at the target in order and completeness. UDP (User Datagram Protocol) is a connectionless transport layer protocol with fast transmission speed but no guarantee of data reliability, suitable for scenarios that require high real-time performance and allow some data loss.RTSP (Real Time Streaming Protocol) is an application layer protocol, mainly used to control multimedia streams on streaming media servers to achieve control functions such as play, pause, stop, etc. RTP (Real-time Transport Protocol) is a transport layer protocol based on UDP, responsible for real-time transport of audio and video data streams to ensure data order and time synchronization, meeting the needs of real-time multimedia transmission. UID (Unique Identifier) is a string or number sequence used to uniquely identify a device, user or object. In remote control and network communication, UID number is used to distinguish different terminals or resources to ensure the accuracy and security of data transmission and operation.

[0020] Referring to Figure 1 Fig. 1 is a flowchart of a remote desktop implementation method according to an embodiment of the present application.

[0021] In this embodiment, the remote desktop implementation method is applicable to a server and a client in an embedded system environment, wherein the server is responsible for responding to control instructions of the remote desktop, and the client is responsible for initiating control operations of the remote desktop, including: Step S100: The server registers a connection interface, a disconnection interface and an operation interface for calling by the client and implementing remote desktop functions.

[0022] Specifically, when the server starts, it initializes the remote desktop service module and completes the registration of the connection interface, the disconnection interface and the operation interface. These interfaces are registered with the network library through the network protocol to expose them to the client for calling. The connection interface is used to establish a remote desktop session, the disconnection interface is used to safely close the remote desktop session, and the operation interface is used to transmit mouse operation instructions of the client. During the interface registration process, the server allocates necessary resources (such as video capture channels, video encoder instances, etc.) and sets the video frame callback function of the interface to ensure that the corresponding processing logic can be triggered when the client calls. This step establishes the remote desktop interaction infrastructure between the server and the client, and clearly defines the responsibilities and interface calling process of both parties.

[0023] Step S200: The client actively establishes a network connection with the server and confirms that the network connection is successful.

[0024] Specifically, after the user inputs the identity information in the client interface, the client initiates a network connection request to the server actively using the socket-based network communication technology and based on the P2P protocol or the traditional TCP protocol. During the network connection (i.e., the underlying connection based on the transport layer protocol such as TCP) establishment process, the client and the server perform handshaking and identity authentication to ensure the legality and security of the network connection. After the network connection is successful, the client receives the connection success response returned by the server, and at this time, the client is ready to call the interface registered by the server to complete the subsequent operation.

[0025] Step S300: The client calls the connection interface, and the server synthesizes the local video stream and transmits it to the client. The client receives the remote video stream and decodes and renders it.

[0026] Specifically, the connection interface is an interface at the level of the remote desktop protocol and has the functions of starting a remote desktop session and transmitting a video stream. Its call is performed after the network connection is established. When the client calls the connection interface registered by the server, the server starts the local video acquisition module to acquire the local video data stream. The acquired video data is subjected to format conversion and preprocessing by the video processing subsystem and then is transmitted to the video encoder for compression and encoding (the encoding format is H.264 or H.265). The encoded data is encapsulated into a standard streaming media format (the standard streaming media format includes a transmission protocol such as RTSP and RTP and an encoding format such as H.265 and H.264) and is then sent to the client through the network. After the client receives the synthesized video stream, the video decoder is started to decode the compressed video data into original image frames, and the renderer is used to display the images on the video window of the client. The entire process adopts the multi-threading and asynchronous callback mechanism to ensure the real-time performance and smoothness of the video stream and to realize efficient remote desktop picture transmission.

[0027] Step S400: When the client calls the disconnection interface, the client actively disconnects the network connection with the server, and the server passively disconnects the network connection with the client and releases the corresponding resources.

[0028] Specifically, the client initiates a network disconnection with the server, which is equivalent to the client sending a disconnection request to the server; the server passively disconnects the network connection with the client, which is equivalent to the server responding to the client's disconnection request. When the client calls the disconnection interface, it first sends a disconnection request to the server, notifying the server to prepare to close the remote desktop session. After receiving the disconnection request, the server stops the video stream collection and encoding thread, closes the network transmission channel, unregisters the video frame callback function, and releases all resources related to the client's remote desktop session (such as memory buffer, network socket, etc.). The client synchronously closes the remote video stream receiving and decoding thread, releases local resources, and updates the connection state to the disconnected state. Through this step, it can ensure that when the client and server disconnect the network connection, the resources of both parties are safely released, avoiding memory leakage and resource occupation, and ensuring the long-term stable operation of the system.

[0029] Step S500: When the client calls the operation interface, the client collects local mouse operation instructions and sends them to the server, and the server receives the mouse operation instructions and executes the corresponding mouse operation event.

[0030] Specifically, the client captures the real-time position coordinates and operation types of the mouse by listening to local mouse operation events (including movement operation events, click operation events, and scroll wheel operation events). The position coordinates are normalized and converted to a proportional value relative to the client display area (i.e., converted to normalized coordinates) to ensure operation consistency between devices of different resolutions. The client sends the encapsulated mouse operation instructions to the server through the operation interface. After receiving the mouse operation instructions, the server analyzes the operation type and position coordinates, and then converts the normalized coordinates to actual pixel coordinates of the local screen, i.e., pixel positions in the actual screen resolution coordinate system of the server. Subsequently, the API of the GUI system is called to simulate the mouse operation behavior corresponding to the mouse operation instructions, realizing the real operation effect of the remote mouse.

[0031] It should be noted that steps S400 (disconnection) and S500 (remote mouse operation control) are both executed during the remote desktop session, and there is no sequence between the two, which can be implemented in parallel or selectively.

[0032] Compared with the prior art, the remote desktop implementation method provided by the application can realize remote operation of one embedded device by another embedded device. In particular, when the NVR device can support both server function and client function and can flexibly switch roles according to requirements (only one role can be activated at the same time, and a display needs to be equipped when the client role is activated), only one NVR device and one display are needed to remotely control another NVR device. The controlled NVR device can be equipped with a local display or not, and only remote access is needed to complete the operation, thereby significantly simplifying the system architecture, effectively reducing the hardware cost and saving the deployment space.

[0033] In some embodiments of the application, the server synthesizes the local video stream and transmits it to the client, including: Step S311: The server checks the state of the encoding channel and obtains the currently activated encoding channel ID.

[0034] Specifically, the check of the encoding channel state depends on the global variable s_iChannel, which records the currently activated encoding channel ID. When called for the first time, if the value of s_iChannel is -1, it indicates that the corresponding encoding channel has not been activated, and the system will enter the initialization process.

[0035] Step S312: If it is checked that the state is the unstarted state, the video processing subsystem and the video encoder are created and initialized, and the video processing subsystem and the video encoder and the video output module of the server are bound.

[0036] The video output module of the server can write the currently displayed video data back to the frame buffer and capture the original video data from the frame buffer.

[0037] Step S313: Start the video stream acquisition and encoding thread and register the video frame callback function.

[0038] The video frame callback function is responsible for receiving the encoding data output by the video encoder and encapsulating the encoding data into a standard streaming media format.

[0039] Step S314: When the video encoder generates a new encoded video frame, the video frame callback function is automatically called to process the encoding data, so as to complete the encapsulation of the encoded video frame into a standard streaming media format.

[0040] Step S315: The encapsulated video data is transmitted to the upper network transmission module, and the video data is transmitted to the client based on the network transmission protocol.

[0041] Specifically, the video data sent to the client adopts the H.264 or H.265 encoding format, and the resolution, frame rate and other parameters are allowed to be partially configured by the user according to the needs. Of course, in order to ensure that the system performance is not affected, the related parameters need to be limited within the preset maximum value range.

[0042] Step S316: updating the state of the encoding channel to a connected state.

[0043] Specifically, the embodiment details the process of combining the local video stream by the server and transmitting it to the client, specifically including: dynamic checking and initialization of the encoding channel, binding of the video processing subsystem and the video encoder, starting of the video stream acquisition and encoding thread, and registration of the video frame callback function, as well as encapsulation and transmission of the encoded data. Through the embodiment, the server can efficiently and in real time acquire and encode the locally displayed video data, and then encapsulate it into a standard streaming media format, so as to ensure the continuity of the video stream and the stability of the transmission. The embodiment not only optimizes the utilization of system resources, but also improves the response speed and efficiency of the encoding process through the video frame callback mechanism, and at the same time, adopts the standard streaming media format, enhances the compatibility and network adaptability of the transmission, and can ensure the smoothness and reliability of the remote video transmission.

[0044] In some embodiments of the present application, the server passively disconnects the network connection with the client and releases the corresponding resources, including: Step S411: the server checks the state of the encoding channel and confirms that the state is a connected state.

[0045] Specifically, the checking of the encoding channel state is realized by judging the value of the global variable s_iChannel: when s_iChannel is -1, it means that the current encoding channel has not been activated, the server has not established the corresponding network connection, and there is no need to continue to execute the subsequent operation, at this time the server directly returns the information of successful operation to the client; when s_iChannel is not -1, it means that the current encoding channel has been activated, the network connection between the server and the client has been established, and the subsequent operation needs to be continued. The state checking mechanism can effectively prevent repeated operation and error release, and guarantee the long-term stable operation and reliability of the server.

[0046] Step S412: closing the mouse operation instruction receiving thread, stopping receiving the mouse operation instruction of the client, and releasing the related resources.

[0047] Step S413: closing the network connection corresponding to the encoding channel, and releasing the network communication related resources.

[0048] Step S414: stopping the video stream acquisition and encoding thread, releasing the internal buffer of the video encoder and the thread resources.

[0049] Step S415: unbind the video processing subsystem and the video encoder, and the video processing subsystem and the video output module of the server, and destroy the resources occupied by the video processing subsystem and the video encoder.

[0050] Step S416: cancel the registration of the video frame callback function by passing in a NULL parameter.

[0051] Specifically, to register a video frame callback function, a function pointer (such as RemoteDesktopFBCallback) needs to be passed in. If the registered video frame callback function needs to be canceled and no longer receives relevant notifications, the cancel operation can be implemented by calling the same registration interface and passing in a NULL parameter (null pointer). Passing in a NULL parameter indicates that the registered video frame callback function is canceled, and the system will stop calling any callback, thereby achieving efficient cancellation of the callback. After the network connection is disconnected, the registration of the video frame callback function should be canceled in a timely manner to avoid the video encoder continuing to call the invalid video frame callback function to pass video data, thereby ensuring the stability and security of the system.

[0052] Step S417: update the state of the encoding channel to a closed state.

[0053] Specifically, the state of the encoding channel is updated to a closed state, that is, the global variable s_iChannel of the encoding channel is set to -1, indicating that the encoding channel has failed and is no longer in an active state. Subsequently, if it needs to be activated again, the initialization process must be re-executed to restore the normal working state.

[0054] Specifically, the embodiment details the process of the server passively disconnecting the network connection with the client and releasing the corresponding resources, specifically including: encoding channel state checking, closing the network connection, closing the mouse operation instruction receiving thread, stopping the video stream acquisition and encoding thread, unbinding the video processing subsystem, canceling the registration of the video frame callback function, and state updating. Through the embodiment, the server can safely and completely clean up all resources related to the connection with the client, effectively avoiding resource leakage and system abnormalities.

[0055] In some embodiments of the present application, the server receives the mouse operation instruction and executes a corresponding mouse operation event, including: Step S511: the server receives the mouse operation instruction sent by the client through the signaling channel.

[0056] The mouse operation instruction includes a moving operation instruction, a clicking operation instruction, and a scroll wheel operation instruction.

[0057] Step S512: parsing the mouse operation instruction, determining the type of the mouse operation instruction, and extracting the normalized coordinates of the mouse operation instruction.

[0058] Step S513: mapping the normalized coordinates to the actual screen resolution of the server, and calculating the corresponding pixel coordinates.

[0059] Specifically, the actual screen resolution of the server is a key basic parameter in the video capture and encoding process, which is usually 1920x1080. In the video capture, encoding and transmission process, keeping the resolution of the video stream consistent with the actual screen resolution helps to improve the video quality and the overall performance of the system.

[0060] Step S514: calling the API of the GUI system of the server, and based on the calculated pixel coordinates, simulating the mouse operation behavior corresponding to the mouse operation instruction.

[0061] Specifically, the embodiment details the process of the server receiving the mouse operation instruction and executing the corresponding mouse operation event, which specifically includes receiving the mouse operation instruction through the signaling channel, analyzing the type and normalized coordinates of the mouse operation instruction, mapping to the actual screen pixel coordinates, and calling the GUI system API to simulate the mouse operation behavior corresponding to the mouse operation instruction. This embodiment supports multiple mouse operation types, enriches the interaction mode, and through the mapping of normalized coordinates, ensures the compatibility between devices with different resolutions. At the same time, directly calling the GUI system API makes the mouse operation response fast and real, realizing the accurate response of the server to the client mouse operation and the efficient interaction of remote control.

[0062] In some embodiments of the present application, step S200 includes: Step S201: when the client receives the identity information input through the user interface, the client completes the network socket creation and establishes a bottom-layer network connection with the server through the P2P protocol.

[0063] Wherein, the identity information includes username, password and device UID number of the server responsible for responding to remote desktop control instructions.

[0064] Step S202: the client calls the connection interface registered by the server, and passes the identity information and connection parameters, then the client and the server perform a stream media protocol handshake to establish an application layer connection.

[0065] Wherein, the connection parameters include encoding format, maximum resolution and frame rate limit.

[0066] Specifically, the encoding format is H.264 or H.265, and the maximum resolution and frame rate limit can be configured according to the client screen performance. Of course, to ensure that the system performance is not affected, the related parameters need to be limited within the preset maximum value range.

[0067] Step S203: When the client receives the response information returned by the server based on the identity information after completing the identity verification, the client confirms that the network connection is successful, updates the connection state of the client to the connected state, and starts the video stream receiving thread and the mouse operation instruction sending thread.

[0068] The response information includes an encoding channel ID and encoding parameters, and the network connection success is an application layer connection success.

[0069] Specifically, the embodiment details the process of the client actively establishing a network connection with the server and confirming the success of the network connection, which specifically includes user identity information input, network socket creation, P2P protocol connection, calling the registered connection interface of the server, and stream media protocol handshake and identity verification. Through this embodiment, the client can safely and reliably complete identity verification, successfully establish a network connection with the server, and prepare for receiving remote desktop video data and sending control instructions. The embodiment uses the P2P protocol, which can effectively reduce network delay and improve connection efficiency, and the identity verification mechanism can further enhance system security.

[0070] In some embodiments of the present application, the client receives and decodes the remote video stream, including: Step S321: The client starts the remote video stream receiving and decoding thread, creates and initializes the video decoder and renderer.

[0071] Step S322: Receive the video data in the standard stream media format sent by the server by calling the registered connection interface of the server, and decode the video data by calling the interface of the video decoder to obtain the decoded YUV format frame data.

[0072] Specifically, first, a video decoder (VDEC) is created, then the standard stream media format stream data (which is the video data after encoding and compression) sent by the server is received by calling the registered connection interface of the server, and then the H.265 or H.264 stream data is sent to the video decoder for decoding by calling the interface function VDEC_SendStream provided by the video decoder, and finally the decoded YUV format frame data is obtained.

[0073] Step S323: scaling and adapting the YUV format frame data according to the difference between the resolution of the YUV format frame data and the video window size of the client, by using a preset scaling algorithm.

[0074] Specifically, the resolution of the YUV format frame data is usually consistent with the resolution of the server video stream, and is also the actual resolution of the decoded video frame. The video window size of the client refers to the window size for displaying the video, which belongs to the display area at the software level. Since the resolution of the server video stream may not match the video window size of the client (for example, the resolution of the server video stream is 1920x1080, and the video window size of the client is 1280x720.), it is necessary to process the decoded YUV format frame data by using a preset scaling algorithm to adapt to the video window size of the client. By using the preset scaling algorithm to scale and adapt the YUV format frame data, the picture size is dynamically adjusted, which can effectively ensure the proportion and clarity of the picture and improve the overall viewing effect.

[0075] Step S324: mapping the scaled and adapted YUV format frame data to the display layer of the video output module of the client, and calling the hardware acceleration interface of the renderer to perform rendering, so as to realize the remote desktop display function.

[0076] Among them, the rendering supports dynamic adjustment of the rendering resolution according to the change of the resolution of the input YUV format frame data or the video window size of the client.

[0077] Specifically, the decoded YUV format frame data is directly mapped to the virtual address of the VO display layer, and then the VO_SendFrame interface is called to realize hardware acceleration rendering. This efficient rendering method by using the hardware acceleration interface not only can significantly improve the rendering efficiency, but also effectively reduces the CPU load. The system also supports dynamic adjustment of the resolution, such as switching from 1080P to 720P (i.e., switching from a resolution of 1920x1080 to a resolution of 1280x720), to adapt to different video window sizes and further optimize the display effect.

[0078] Specifically, the embodiment details the process of receiving and decoding the remote video stream by the client, which specifically includes: starting the remote video stream receiving and decoding thread, initializing the video decoder and the renderer, receiving the video data in the standard stream media format, decoding into the YUV format frame data, scaling and adapting, and rendering by using the hardware acceleration interface. Through this embodiment, the client can efficiently complete the decoding and rendering of the video stream, ensuring smooth and clear picture quality for the remote desktop display.

[0079] In some embodiments of the present application, the client actively disconnects the network connection with the server, including: Step S421: The client displays a disconnect button on the display screen of the client.

[0080] Step S422: When the disconnect button receives a triggering instruction, the client sends a disconnect request to the server by calling the disconnect interface registered by the server.

[0081] Step S423: The remote video stream receiving and decoding thread is stopped, and the network socket is closed.

[0082] Step S424: The resources of the video decoder and the renderer are released.

[0083] Step S425: The mouse operation instruction sending thread is closed, and the sending of mouse operation instructions to the server through the signaling channel is stopped.

[0084] Step S426: The connection state of the client is updated to a disconnected state.

[0085] Specifically, the embodiment describes in detail the process of the client actively disconnecting the network connection with the server, including displaying a disconnect button, sending a disconnect request, stopping the remote video stream receiving and decoding thread, releasing the resources of the video decoder and the renderer, closing the mouse operation instruction sending thread, and updating the connection state of the client. Through the embodiment, the client can safely and actively disconnect the remote desktop connection, release the relevant resources, avoid memory leakage, and ensure the stable operation of the system.

[0086] In some embodiments of the present application, the client collects local mouse operation instructions and sends them to the server, including: Step S521: The client listens to the local mouse operation event and captures the mouse position coordinates, mouse key state, and mouse wheel scrolling amount of the mouse operation event through the API of the GUI system of the client.

[0087] The mouse operation event includes a moving operation event, a clicking operation event, and a wheel operation event.

[0088] Specifically, the API of the GUI system is called to capture the mouse operation event, which can improve the compatibility and accuracy of the mouse operation event capture.

[0089] Step S522: The captured mouse position coordinates are converted into normalized coordinates defined by the client.

[0090] Specifically, the method of converting mouse position coordinates into normalized coordinates includes linear scaling, proportional mapping, etc. The value range of mouse position coordinates is determined by screen resolution (e.g. 1920x1080). Normalization of mouse position coordinates can eliminate position deviation caused by different resolutions and ensure consistency of cross-device operations.

[0091] Step S523: encapsulate the mouse operation instruction.

[0092] The encapsulated mouse operation instruction includes the type of the mouse operation event and the corresponding normalized coordinates.

[0093] Step S524: send the encapsulated mouse operation instruction to the operation interface registered by the server based on TCP or UDP protocol.

[0094] Specifically, the transmission of mouse operation instructions is based on two transmission protocols. Key mouse operation instructions (e.g. click operation) use TCP protocol for retransmission to ensure transmission reliability, while non-key mouse operation instructions (e.g. continuous movement operation) use UDP protocol for transmission to reduce transmission delay and achieve smoother response effect. This transmission strategy can adapt to different network environments and improve the flexibility and stability of the system.

[0095] Specifically, the embodiment details the process of capturing local mouse operation instructions by the client and sending them to the server, which includes listening to mouse operation events, capturing mouse position coordinates, state and scroll wheel scroll amount through the API of the GUI system, performing coordinate normalization, encapsulating mouse operation instructions, and sending mouse operation instructions to the operation interface registered by the server based on TCP or UDP protocol. This embodiment can capture local mouse operation events in real time and accurately transmit them to the server to achieve precise control of the remote mouse.

[0096] Please refer to Figure 2 The flowchart of the second embodiment of the remote desktop implementation method of the present application is shown.

[0097] In some embodiments of the present application, after step S200, the process further includes: Step S601: the client continuously monitors the heartbeat packet sent by the server.

[0098] Step S602: if the heartbeat packet is detected to be lost, it is confirmed that the current network connection has been disconnected, and the client triggers the network automatic reconnection mechanism.

[0099] The reconnection interval time of the network automatic reconnection mechanism is gradually increased.

[0100] Specifically, if a heartbeat packet is detected as missing, it indicates a network connection error or server crash. In this case, the automatic network reconnection mechanism should be triggered promptly to ensure the continuity and stability of the remote desktop service. The reconnection interval is increased incrementally, successively to 1 second, 2 seconds, or 4 seconds, and so on. By gradually increasing the retry waiting time, the success rate of network connection recovery can be improved, while reducing the system load.

[0101] Step S603: Once the network reconnection is confirmed to be successful, resume the video stream receiving thread and the mouse operation command sending thread.

[0102] When resuming the video stream receiving thread, the encoding parameters need to be renegotiated with the server.

[0103] Specifically, this embodiment provides supplementary explanations of the process by which the client actively establishes a network connection with the server. This includes: continuously monitoring the server's heartbeat packets to detect the network connection status; triggering an automatic network reconnection mechanism upon detecting a disconnection; and employing an incremental reconnection interval strategy to avoid network congestion and resource waste. After successful network reconnection, the client resumes its video stream receiving thread and mouse operation command sending thread, and renegotiates encoding parameters to ensure stable video quality and performance. This embodiment endows the client with automatic recovery capabilities in the event of network anomalies, significantly improving the stability and continuity of remote desktop connections while reducing user intervention and optimizing the overall user experience.

[0104] The remote desktop implementation method provided by this invention breaks through the limitations of traditional remote desktop implementation schemes that rely on PCs and dedicated application software, and directly controls embedded devices, significantly reducing the overall system cost and device deployment space requirements.

[0105] The following are embodiments of the network video recording device provided by the present invention. The embodiments of the network video recording device and the embodiments of the remote desktop implementation method described above belong to the same concept. Details not fully described in the embodiments of the network video recording device can be found in the embodiments of the remote desktop implementation method described above.

[0106] In this embodiment, a network video recording device, running in an embedded system environment, is a server responsible for responding to remote desktop control commands or a client responsible for initiating remote desktop control operations, comprising: Memory, used to store program instructions; and A processor for executing the program instructions to implement the remote desktop implementation method as described above.

[0107] It should be noted that the client is the user entrance of the remote control system, mainly responsible for establishing a network connection with the server, receiving and rendering a remote video stream, collecting and sending local mouse operation instructions, and processing network exceptions to ensure smooth and stable user experience. The server is mainly responsible for collecting and encoding local video streams, receiving and analyzing mouse operation instructions sent by the client, and continuously sending heartbeat packets to ensure the stability and continuity of remote control.

[0108] The network video recording device of the present application can realize the steps of the remote desktop implementation method described above, and therefore has all the beneficial effects brought by the technical solutions of the embodiments of the steps of the remote desktop implementation method described above, which will not be repeated here.

[0109] The above only describes some or preferred embodiments of the present application, and neither the text nor the drawings can limit the scope of protection of the present application. Any equivalent structural transformation made by using the content of the specification and drawings, or direct / indirect application in other related technical fields is included in the scope of protection of the present application.

[0110] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD ROM, optical storage, etc.) containing computer usable program code.

[0111] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0112] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices that implement the functions specified in the flowFigure 1 one or more processes and / or functions described in the flow Figure 1 one or more blocks or multiple blocks.

[0113] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable devices provide processes for implementing the flow Figure 1 one or more processes and / or functions described in the flow Figure 1 one or more blocks or multiple blocks.

[0114] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application have been described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the scope of protection of the claims of the present application.

Claims

1. A remote desktop implementation method, applicable to server and client in an embedded system environment, wherein, The server is responsible for responding to remote desktop control commands, and the client is responsible for initiating remote desktop control operations. The feature is that it includes: The server registers connection interfaces, disconnection interfaces, and operation interfaces for the client to call and implement remote desktop functionality. The client actively establishes a network connection with the server and confirms that the network connection is successful; When the client calls the connection interface, the server synthesizes a local video stream and transmits it to the client. The client receives the remote video stream and decodes and renders it. When the client calls the disconnect interface, the client actively disconnects the network connection with the server, and the server passively disconnects the network connection with the client and releases the corresponding resources. When the client calls the operation interface, the client collects local mouse operation instructions and sends them to the server. The server receives the mouse operation instructions and executes the corresponding mouse operation events.

2. The remote desktop implementation method according to claim 1, characterized in that, The server synthesizes a local video stream and transmits it to the client, including: The server checks the status of the encoding channel and obtains the ID of the currently active encoding channel; If the status is detected as not started, a video processing subsystem and a video encoder are created and initialized, and the video processing subsystem is bound to the video encoder and the video output module of the server. The video output module of the server can write back the currently displayed video data to the frame buffer and capture the original video data from the frame buffer. Start the video stream acquisition and encoding thread, and register a video frame callback function; wherein, the video frame callback function is responsible for receiving the encoded data output by the video encoder and encapsulating the encoded data into a standard streaming media format; When the video encoder generates a new encoded video frame, it automatically calls the video frame callback function to process the encoded data in order to complete the standard streaming media format encapsulation of the encoded video frame. The encapsulated video data is passed to the upper-layer network transmission module, and then the video data is transmitted to the client based on the network transmission protocol; Update the status of the encoding channel to "connected".

3. The remote desktop implementation method according to claim 2, characterized in that, The server passively disconnects the network connection with the client and releases the corresponding resources, including: The server checks the status of the encoding channel and confirms that the status is connected. Close the mouse operation command receiving thread, stop receiving mouse operation commands from the client, and release related resources; Close the network connection corresponding to the encoding channel and release network communication-related resources; Stop the video stream acquisition and encoding threads, and release the internal buffer and thread resources of the video encoder; Unbind the video processing subsystem from the video encoder, and the video output module of the server from the video processing subsystem, and destroy the resources occupied by the video processing subsystem and the video encoder; The registration of the video frame callback function can be cancelled by passing in a NULL parameter; Update the status of the encoding channel to the off state.

4. The remote desktop implementation method according to claim 1, characterized in that, The server receives the mouse operation command and executes the corresponding mouse operation event, including: The server receives mouse operation commands sent by the client through a signaling channel; wherein the mouse operation commands include movement operation commands, click operation commands, and scroll wheel operation commands. The mouse operation command is parsed to determine its type and the normalized coordinates of the mouse operation command are extracted. The normalized coordinates are mapped to the actual screen resolution of the server, and the corresponding pixel coordinates are calculated. The API of the server's GUI system is called, and based on the calculated pixel coordinates, the mouse operation behavior corresponding to the mouse operation command is simulated.

5. The remote desktop implementation method according to claim 1, characterized in that, The client actively establishes a network connection with the server and confirms the successful network connection, including: When the client receives the identity information entered through the user interface, the client completes the creation of the network socket and establishes an underlying network connection with the server through the P2P protocol; wherein, the identity information includes the username, password and the device UID number of the server responsible for responding to remote desktop control commands; The client calls the connection interface registered by the server and passes the identity information and connection parameters. Then, the client and the server perform a streaming media protocol handshake to establish an application layer connection. The connection parameters include encoding format, maximum resolution, and frame rate limit. When the client receives a connection success response from the server after completing authentication based on the identity information, the client confirms the successful network connection, updates the client's connection status to connected, and starts the video stream receiving thread and the mouse operation command sending thread; wherein, the response information includes the encoding channel ID and encoding parameters, and the successful network connection refers to the successful application layer connection.

6. The remote desktop implementation method according to claim 5, characterized in that, The client receives and decodes the remote video stream, including: The client starts a remote video stream receiving and decoding thread, and creates and initializes a video decoder and renderer. By calling the connection interface registered by the server, the system receives video data in standard streaming media format sent by the server, and calls the interface of the video decoder to decode the video data to obtain decoded YUV format frame data. Based on the difference between the resolution of the YUV format frame data and the size of the client's video window, a preset scaling algorithm is used to scale and adapt the YUV format frame data. The scaled and adapted YUV format frame data is mapped to the display layer of the client's video output module, and the hardware acceleration interface of the renderer is called for rendering to realize the remote desktop display function; wherein, the rendering supports dynamic adjustment of the rendering resolution according to the resolution of the input YUV format frame data or the change of the client's video window size.

7. The remote desktop implementation method according to claim 6, characterized in that, The client actively disconnecting from the network connection with the server includes: The client displays a disconnect button on its screen; When the disconnect button receives a trigger command, it sends a disconnect request to the server by calling the disconnect interface registered by the server. Stop the remote video stream receiving and decoding thread and close the network socket; Release the resources of the video decoder and the renderer; Close the mouse operation command sending thread and stop sending mouse operation commands to the server through the signaling channel; Update the client's connection status to disconnected.

8. The remote desktop implementation method according to claim 1, characterized in that, The client collects local mouse operation commands and sends them to the server, including: The client listens for local mouse operation events and captures the mouse position coordinates, mouse button states, and mouse wheel scrolling amount of the mouse operation events through the API of the client's GUI system; wherein, the mouse operation events include movement operation events, click operation events, and scroll wheel operation events; Convert the captured mouse position coordinates into normalized coordinates defined by the client; Encapsulate mouse operation instructions; wherein, the encapsulated mouse operation instructions include the type of the mouse operation event and the corresponding normalized coordinates; Based on the TCP or UDP protocol, the encapsulated mouse operation commands are sent to the operation interface registered on the server.

9. The remote desktop implementation method according to claim 5, characterized in that, After the client actively establishes a network connection with the server and confirms a successful network connection, the process further includes: The client continuously monitors the heartbeat packets sent by the server. If the heartbeat packet is detected to be lost, it is confirmed that the current network connection has been lost, and the client triggers the network automatic reconnection mechanism; wherein, the reconnection interval of the network automatic reconnection mechanism gradually increases; Once the network reconnection is confirmed to be successful, the video stream receiving thread and the mouse operation command sending thread are restored; however, when restoring the video stream receiving thread, the encoding parameters need to be renegotiated with the server.

10. A network video recording device, operating in an embedded system environment, wherein the network video recording device is a server responsible for responding to control commands from a remote desktop or a client responsible for initiating control operations from a remote desktop, characterized in that, include: Memory, used to store program instructions; as well as A processor for executing the program instructions to implement the steps of the remote desktop implementation method as described in any one of claims 1 to 9.

Images