A method and system for protecting the real image of controlled equipment in remote control

Abstract

A method and system for protecting the real image of a controlled device in remote control, comprising: injecting an injection program into the window manager DWM process of the controlled device; when the window manager merges the images and processes them into a final output image, the injection program automatically searches for the interface of the system that finally outputs the image to the display of the controlled end, backs up the interface, and replaces the interface with the interface of the control program that outputs the specified image; the injection program responds to the relevant instructions of the privacy screen image setting of the control program, performs image processing and replacement, and displays the privacy screen image on the display of the controlled end; the controlled end connects to the image acquisition interface of the controlled device system to acquire the real final output image. The present invention realizes the purpose of protecting the privacy of the real picture in the remote control process by injecting it into the window manager process, and does not affect the normal operation of the system and other programs, and has wide applicability and strong compatibility.

Description

Technical Field

[0001] The present invention relates to the technical field of remote control, and in particular to a method and system for protecting a real image of a controlled device in remote control. Background Art

[0002] In a remote control environment, the remotely controlled machine equipment may be in an unattended state, and the final picture displayed in real time on its physical display may be seen by other people or recorded, which poses a great risk of privacy or confidentiality leakage or security risks. In this application environment, the remote control program can display the real picture normally during remote control operation, and the physical display does not output the real picture (displays a static image or a dynamic screen saver), which can solve this problem well. The present invention is an effective solution to this problem. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method for protecting the real image of a controlled device in remote control with wider applicability and stronger compatibility, as well as a system for protecting the real image of a controlled device in remote control.

[0004] The present invention is achieved through the following technical solutions:

[0005] A method for protecting the real image of a controlled device in remote control comprises the following steps:

[0006] S1, the controlled end control program responds to the privacy screen opening instruction and injects the injection program into the window manager DWM process of the controlled device;

[0007] S2. Generally speaking, after the window manager DWM merges the graphics drawn by the windows of each application program on the controlled device into a final output image, it will be output to the display through an interface (i.e., the system interface) and finally displayed on the display. Before outputting the image to the display, the injection program automatically searches for the system interface of the controlled device system that finally outputs the image to the controlled end display, backs up the interface, and replaces the interface with the interface for the control program to output the specified image. The interface for the control program to output the specified image is the hook interface of the injection program, which is used for replacing the subsequent output image.

[0008] S3, the injection program responds to the relevant instructions of the privacy screen image setting of the control program, performs image processing and replacement, and displays the privacy screen image on the display of the controlled terminal;

[0009] S4. The controlled end is connected to the image acquisition interface of the controlled device system through the image acquisition module, acquires the real final output image processed by the window manager DWM, and transmits it to the remote control end, and displays it on the display of the remote control end.

[0010] Furthermore, the privacy screen image described in step S3 is the image obtained after dynamic processing of the final output image described in step S2, or other images or videos set by the control program. The privacy screen image needs to be processed by pixel matrix and obtained by directx texture conversion, and finally can be directly output to the display using the original real interface.

[0011] Furthermore, the dynamic processing includes blurring, watermarking or static scaling and cropping, which are performed by an injection program.

[0012] Furthermore, the privacy screen image described in step S3 is a 3D scene, which is obtained by real-time 3D rendering of preset materials and is finally output through a real interface.

[0013] Furthermore, the privacy screen image exists in the form of a file and supports cloud configuration. When the privacy screen image in the cloud needs to be called, it is synchronized to the local controlled device by the controlled end control program, and finally the injection program performs corresponding processing according to the control program instructions.

[0014] Furthermore, in step S3, when the controlled end has multiple displays, the control program specifies the display that needs to display the privacy screen image, and the DWM process performs filtering according to the identification number of each display, and displays the replaced privacy screen image on the specified display.

[0015] Furthermore, the controlled device system is a Windows 7 or higher system, such as Win7, Win8, Win10, etc.

[0016] A system for protecting the real image of a controlled device in remote control, comprising a controlled terminal and a remote control terminal;

[0017] The controlled end includes a controlled device and a controlled end display, the controlled device is installed with a controlled device system and a control program, the controlled device system is provided with a window manager DWM, and the window manager DWM controls the final presentation of the image on the controlled end display through a system interface;

[0018] The control program on the controlled end includes an image acquisition module and a privacy screen control module, which are used to: respond to the user's choice in real time to turn on or off the privacy screen; when the privacy screen is turned on, load the injection program into the window manager DWM process of the controlled device system, and control the privacy screen display policy of the controlled end display through the privacy screen control module, and the privacy screen display policy includes specifying the category of the privacy screen image and specifying the controlled end display that needs to display the privacy screen image; when the controlled device needs to be remotely controlled, connect the image acquisition interface of the controlled device system through the image acquisition module to acquire the real final output image processed by the window manager DWM; when the program exits or the privacy screen is closed, control the injection program to restore the output to the real image, and uninstall the injection program.

[0019] After the injection program is loaded into the controlled device system of the controlled device, it is used to: respond to the control program and inject itself into the window manager DWM process; automatically find the system interface of the controlled device system to finally output the image to the controlled end display; automatically hook the system interface of the controlled device system to output the image to the controlled end display; have partial image processing capabilities, including dynamic processing of the image merged and processed by the window manager DWM; at the hook interface, output the image specified by the control program or the dynamically processed image to the controlled end display; respond to the real-time exit of the control program, and restore the original state after the control program exits or receives the exit instruction of the control program.

[0020] The remote control terminal includes a control device and a control terminal display, the control terminal display is used to display the real image of the controlled terminal, the remote control device is installed with a remote control program, the remote control program is communicated with the control program on the controlled terminal, and is used to send relevant control instructions to the controlled terminal and receive the real image of the controlled terminal.

[0021] Furthermore, one of the controlled devices is connected to a plurality of the controlled end displays. The controlled device system is Windows 7 or above.

[0022] Furthermore, the image processing capabilities of the injection program include image blurring, watermarking, static scaling and cropping, and real-time 3D rendering.

[0023] The present invention uses an injection method in a window manager process. The injection program uses process injection, system internal interface hook, image processing, cross-process communication and other technologies to process special effects, add watermarks, or replace with other set images on the real image originally output to the physical display device of the controlled end, and then display it on the physical display device through a hook interface. At the same time, the normal operation and display of the computer program are not affected, and the remote tool or any other image acquisition software can acquire the real image, thereby achieving the purpose of protecting the privacy of the real picture and preventing information from being snooped and leaked during the remote control process of the computer. The present invention is suitable for existing operating systems above Windows 7. Compared with a layer solution that requires placing a top-level window on the system to display the set image, the present invention has wider applicability, stronger compatibility, and less impact on the operation of other programs on the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A general schematic diagram of the logic of program image output, image acquisition and output to the display.

[0025] Figure 2 The figure is a logic diagram of the system image output when a privacy screen is used in the present invention.

[0026] Figure 3 It is a logical structure diagram of the present invention. DETAILED DESCRIPTION

[0027] As we all know, before Windows Vista and Windows 7, Figure 1 As shown in the figure, window drawing is directly submitted to the graphics card's video memory for display, that is, the Windows desktop has only one corresponding image cache, and all programs are directly rendered in this cache. Vista has completely reconstructed the window system and introduced the window manager DWM. The application redirects the image drawn by the window to the off-screen Surface corresponding to the DWM, and then the DWM controls all the off-screen Surfaces, synthesizes them together, and then outputs them to the display.

[0028] In Windows 7 and above, each window has a separate image buffer. The system's window manager will eventually synthesize these layers, including special effects processing. The window manager determines the final output of this synthesized image (that is, the system image) at the application level. The present invention finds a way to intercept the screen interface output by the system to the display in the window manager, and replaces this screen with a preset image or performs special processing on the real image in this interface. Figure 2As shown, since all image acquisitions provided by the system are performed before the final output, the replacement here does not affect the call of the remote control software or other related system functions, and after the replacement, it does not affect the normal use of other software, so that the acquisition program can acquire the real image and the display shows the specified image, thereby achieving the purpose of protecting privacy. Figure 2 The image acquisition program in the image acquisition program refers to the use of the system image acquisition interface to acquire images. The system provides a general acquisition interface solution, and common ones include gdi and desktop duplication. Other image acquisition programs can also use this function, such as conventional screenshots such as paint, and screen video recording such as vlc, vnc, obs, etc. The specific technical solution of the present invention refers to the following discussion.

[0029] A method for protecting the real image of a controlled device in remote control, which is used when the system of the controlled device is Windows 7 and above, such as Win7, Win8, Win10, refer to Figure 2 , mainly including process injection, system internal interface hook, image processing, cross-process communication and other steps, specifically including the following steps:

[0030] S1. In response to the privacy screen opening instruction of the control program, the injection program is injected into the window manager DWM process of the controlled device. The control program is installed on the controlled terminal device, and the controlled terminal user makes relevant settings and turns the privacy screen function on or off. The controlled terminal can be set to automatically open, that is, when the remote control terminal is connected, it automatically enters the privacy screen protection mode. The controlled terminal user can also actively start this function as needed, and only turn it on when privacy screen protection is needed.

[0031] During remote control operations such as remote office and remote maintenance, when the privacy screen needs to be turned on, the user performs relevant operations on the corresponding control program of the controlled device, including issuing instructions to turn on privacy, setting up privacy screen related solutions, etc. The injection program on the controlled device responds to the instructions of the control program and injects itself into the window manager DWM process of the controlled device. On the Windows system platform, the injection program is a DLL (Dynamic Linkable Library) file. When the DLL module is called when the privacy screen operation is required, the system will load it into the memory space. The injection program of the present invention includes functions such as module injection and unloading, system output image to display interface hook, display output image dynamic replacement, and response to remote control instructions.

[0032] S2. After the window manager DWM merges the graphics drawn by the windows of various applications on the controlled device into a final output image (real image), and outputs the image to the controlled display through the system interface, the injection program loaded into the system window manager process automatically searches for the system interface (i.e., the real output interface, which is not open to the system), backs up the interface (the purpose of the backup is to use the real interface to output the processed image later), and replaces the interface with the interface for the control program to output the specified image. The interface is the hook interface of the injection program, which is used to replace the subsequent output image. This process is the hook process.

[0033] The conventional DWM process is that after the window manager DWM synthesizes the final output image, it is directly output to the controlled terminal display through the above system interface. However, the present invention intercepts the output operation before the image is output and replaces the interface by injecting a program.

[0034] S3. The injection program responds to the relevant instructions of the privacy screen image setting of the control program, performs image processing, replacement or other operations, and displays the privacy screen image on all displays of the controlled end or on one or several designated displays. The default is to replace on all displays, and the privacy screen image can also be displayed on a designated display according to the setting. Each display has a device identification number. The DWM process can filter according to the identification number set by the control program, that is, according to the setting, it is determined which displays need to replace the real image output, so as to display the replaced privacy screen image.

[0035] The privacy screen image is the image that is finally output to the display of the controlled end. It can be the image obtained after the final output image (real image) is dynamically processed, or it can be other images or videos set by the control program. The dynamic processing includes blur processing, watermarking, static scaling and cropping, etc. The privacy screen image can be set as needed and can be designed as follows:

[0036] 1. Static preset images, which can come from picture resources or dynamically generated images;

[0037] 2. Dynamic processing of real images, that is, the original system output image (real image) is processed, such as watermarking, blurring, static scaling and cropping;

[0038] 3. Video, which can come from the resource library;

[0039] 4.3D scene.

[0040] The first three can be processed into pixel matrices, converted into directx textures, and then directly output to the display using the original interface. The last one performs real-time 3D rendering based on preset materials, and is also output through the original interface. The above image resources will have default image files in the form of files, and this file supports cloud configuration. When the cloud privacy screen image needs to be called, it will be synchronized to the local by the controlled end control program, and finally processed according to the instructions in the injection program, and finally output to the real display. The injection program has image processing and 3D rendering capabilities.

[0041] When the device operating system finally outputs the image (real image) to the display, the output to the display interface (real output interface) is called, but because the interface has been replaced by the injection program with the interface (hook interface) that controls the program to output the specified image, the output interface that is actually called is the replaced output interface. At this time, the corresponding image will be dynamically processed (such as blurring, adding watermarks) or replaced with other pre-set images or videos (i.e., privacy screen images) as appropriate. After that, the real output interface is called to output these privacy screen images, that is, the display shows the specified image (privacy screen image) instead of the real image of the system, thereby protecting the privacy screen. This function can be used for all physical display devices connected to the computer, which can be understood as all displays or specified displays.

[0042] S4. The controlled end is connected to the image acquisition interface of the controlled device system through the image acquisition module, acquires the real final output image processed by the window manager DWM, and transmits it to the remote control end, which can be displayed on the display of the remote control end.

[0043] Since the image acquisition is completed before the output display, the above injection operation will not affect the image acquisition of the remote control end, nor will it affect the normal image acquisition process of the controlled device system, nor will it affect the normal image acquisition of any other program that calls the system interface for image acquisition. The final effect is that the control program can acquire the real system image, while the physical display shows the replaced picture, thereby achieving the goal of protecting the real image of the controlled device.

[0044] Compared with the layer solution (i.e., placing a top-level window on the system, which is used to display the set image in this window to cover the real image, and the acquisition program has two solutions to acquire the image behind this layer, thereby also achieving the same effect of protecting the privacy of the controlled image), the method of the present invention has a wider applicability, and because of the addition of the window, it may interfere with the operation of other programs in the system. The solution of the present invention has little impact on the system because it is an image interception at the system level, and more importantly, it will not affect the display and image acquisition of other programs, and has good compatibility. On the other hand, the supported systems are also more extensive, and win7 and later systems all support this solution, while the layer solution is only applicable to win10 systems after 2004 because it relies on a newly added feature of the windows system.

[0045] A system for protecting the real image of a controlled device in remote control, such as Figure 3 As shown, it includes the controlled end and the remote control end.

[0046] The controlled end includes a controlled device and a controlled end display. The controlled device can be a common PC. The controlled device is installed with a controlled device system and a control program. The controlled device system is provided with a window manager DWM. The controlled device system is a Windows 7 and above system, such as Win7, win8, win10. The window manager DWM controls the final presentation of the image on the controlled end display through a system interface. The controlled end display is a display screen that needs to be protected for privacy. One controlled device can be connected to several controlled end displays. The specific display images for which privacy protection is required can be set in the control program and enabled in time during implementation. The window manager DWM is used to redirect the images drawn by the windows of each application on the device to the off-screen Surface corresponding to the DWM, and then synthesize them into images that can be output in the end. The final output image is usually controlled by the controlled device system and displayed on the controlled end display. The final output image of the present invention is collected by the control program of the controlled end and can be transmitted to the control end, and finally displayed on the display of the control end. It can also be dynamically processed by the injection program according to the setting of the control program and then displayed on the controlled end display.

[0047] The control program on the controlled end includes an image acquisition module and a privacy screen control module, which has the following functions:

[0048] (1) Respond to the user's choice to enable or disable the privacy screen in real time;

[0049] (2) When the privacy screen is turned on, the injection program is loaded into the window manager DWM process of the controlled device system, and the privacy screen display policy of the controlled display is controlled through the privacy screen control module. The privacy screen display policy includes specifying the category of the privacy screen image and specifying the controlled display that needs to display the privacy screen image, so that the injection program can convert, process and replace the image according to the corresponding policy, and display it on the specified display.

[0050] (3) When the controlled device needs to be remotely controlled, the image acquisition module is connected to the image acquisition interface of the controlled device system to acquire the real final output image processed by the window manager DWM. The image can be displayed on the control end display to facilitate remote control operations.

[0051] (4) When the program exits or the privacy screen is closed, the injection program is controlled to resume outputting the real image, and the injection program is uninstalled.

[0052] The injection program is loaded into the controlled device system of the controlled device. When it needs to be called, the controlled device system will load it into the memory space for execution. It has the following functions:

[0053] (1) responding to the control program and injecting itself into the window manager DWM process;

[0054] (2) After the injection, automatically search for the undisclosed system interface of the controlled device system that finally outputs the image to the controlled terminal display;

[0055] (3) After injecting itself into the DWM process, it automatically hooks the system interface of the controlled device system outputting the image to the controlled end display, that is, the interface in the DWM process where the operating system finally submits the image to the display;

[0056] (4) It has partial image processing capabilities, including dynamic processing of the images merged and processed by the window manager DWM, and calling this function when the image needs to be processed, such as image special effects (blurring), watermarking, static zooming and cropping, real-time 3D rendering, etc.;

[0057] (5) at the hook interface, outputting the image specified by the control program or the dynamically processed image to the physical display of the controlled end;

[0058] (6) responding to the real-time exit of the control program, after the control program exits or receives the exit instruction of the control program, restoring the original state, including restoring the system to the original process of DWM, that is, the previously mentioned replacement and restoration to backup, and uninstalling the hook module;

[0059] (7) Can respond to instructions from multiple remote control programs.

[0060] The remote control terminal includes a control device and a control terminal display, and the control terminal display is used to display the real image of the controlled terminal (the final output image synthesized by DWM). A remote control program is installed on the remote control device, and the remote control program is connected to the control program on the controlled terminal for sending relevant control instructions to the controlled terminal and receiving the real image of the controlled terminal.

[0061] The above detailed description is a specific description of a feasible embodiment of the present invention. The embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or modification that does not deviate from the present invention should be included in the patent scope of this case.

Claims

1. A method for protecting the real image of a controlled device in remote control, characterized in that: A system for protecting the real image of a controlled device in remote control is adopted, which includes a controlled terminal and a remote control terminal; The controlled end includes a controlled device and a controlled end display, the controlled device is installed with a controlled device system and a control program, the controlled device system is provided with a window manager DWM, and the window manager DWM controls the final presentation of the image on the controlled end display through a system interface; The control program on the controlled end includes an image acquisition module and a privacy screen control module, which are used to: respond to the user's choice in real time to turn on or off the privacy screen; when the privacy screen is turned on, load the injection program into the window manager DWM process of the controlled device system, and control the privacy screen display strategy of the controlled end display through the privacy screen control module, wherein the privacy screen display strategy includes specifying the category of the privacy screen image and specifying the controlled end display that needs to display the privacy screen image; When the controlled device needs to be remotely controlled, the image acquisition interface of the controlled device system is connected through the image acquisition module to acquire the real final output image processed by the window manager DWM; when the program exits or the privacy screen is closed, the injection program is controlled to restore the output to the real image, and the injection program is uninstalled; After the injection program is loaded into the controlled device system of the controlled device, it is used to: respond to the control program and inject itself into the window manager DWM process; automatically find the system interface of the controlled device system to finally output the image to the controlled end display; automatically hook the system interface of the controlled device system to output the image to the controlled end display; have partial image processing capabilities, including dynamic processing of the image merged and processed by the window manager DWM; At the hook interface, the image specified by the control program or the image after dynamic processing is output to the controlled terminal display; in response to the real-time exit of the control program, the original state is restored after the control program exits or an exit instruction of the control program is received; The remote control terminal includes a remote control device and a control terminal display, wherein the control terminal display is used to display the real image of the controlled terminal, and the remote control device is installed with a remote control program, which is connected to the control program on the controlled terminal for sending relevant control instructions to the controlled terminal and receiving the real image of the controlled terminal; The method comprises the following steps: S1, the controlled end control program responds to the privacy screen opening instruction and injects the injection program into the window manager DWM process of the controlled device; the control program is installed on the controlled end device, and the controlled end user performs relevant settings and turns on or off the privacy screen function; S2. When the window manager DWM merges the graphics drawn by the windows of each application on the controlled device into a final output image and outputs the image to the controlled terminal display through the system interface, the injection program automatically searches for the system interface, backs up the interface, and replaces the interface with the interface for outputting the specified image by the control program. The interface for outputting the specified image by the control program is the hook interface of the injection program, which is used to replace the subsequent output image. S3, the injection program responds to the relevant instructions of the privacy screen image setting of the control program, performs image processing and replacement, and displays the privacy screen image on the display of the controlled terminal; S4, the controlled end connects to the image acquisition interface of the controlled device system through the image acquisition module to acquire the real final output image processed by the window manager DWM, and transmits it to the remote control end, and displays it on the display of the remote control end; The privacy screen image in step S3 is an image obtained after dynamic processing of the final output image in step S2, or other images or videos set by the control program. The privacy screen image needs to be obtained through pixel matrix processing and directx texture conversion; Alternatively, the privacy screen image in step S3 is a 3D scene, which is obtained by real-time 3D rendering of preset materials.

2. A method for protecting the real image of a controlled device in remote control according to claim 1, characterized in that: The dynamic processing includes blurring, watermarking or static scaling and cropping, which is performed by the injection program.

3. A method for protecting the real image of a controlled device in remote control according to claim 1, characterized in that: The privacy screen image exists in the form of a file and supports cloud configuration. When the privacy screen image in the cloud needs to be called, it is synchronized to the local controlled device by the controlled end control program, and finally the injection program performs corresponding processing according to the control program instructions.

4. A method for protecting the real image of a controlled device in remote control according to claim 1, characterized in that: In step S3, when the controlled end has multiple displays, the control program specifies the display that needs to display the privacy screen image, and the DWM process performs filtering according to the identification number of each display, and displays the replaced privacy screen image on the specified display.

5. A method for protecting the real image of a controlled device in remote control according to claim 1, characterized in that: The controlled device system is Windows 7 or above.

6. A method for protecting the real image of a controlled device in remote control according to claim 1, characterized in that: The image processing capabilities of the injection program include image blurring, watermarking, static scaling and cropping, and real-time 3D rendering.

Images