Remote control method, device, equipment and medium

Abstract

The present application discloses a remote control method, device, equipment and medium, and relates to the field of remote control technology. The remote control method applied to the controlled device includes: after establishing a remote control connection with the first control device, creating a first screen capture task based on the graphics device interface GDI mode and a second screen capture task based on the direct extension DirectX mode; calling the first screen capture task and the second screen capture task to perform screen capture at the same time at the target screen capture frequency; judging whether the first image is the same as the second image, wherein the second image is the previous frame image of the first image obtained by calling the second screen capture task to perform screen capture; if they are different, performing target processing on the image obtained by calling the first screen capture task to perform screen capture; if they are the same, reducing the target screen capture frequency. According to the embodiment of the present application, the resources occupied by remote control can be reduced.

Description

Technical Field

[0001] The present application belongs to the field of remote control technology, and in particular, relates to a remote control method, device, equipment and medium. Background Art

[0002] Remote control refers to connecting a control device to a controlled device, displaying the desktop environment of the controlled device on the control device, and controlling the controlled device through the control device (for example, configuration, software installation, modification, etc.).

[0003] During remote control, the controlled device needs to perform screen capture, image conversion, image encoding and data transmission on its desktop so that the controlling device can control the controlled device. However, in related technologies, during remote control, the resources occupied (for example, CPU resources, network resources, etc.) are relatively high. Summary of the invention

[0004] The embodiments of the present application provide a remote control method, apparatus, device and medium, which can solve the problem of high resource occupation by remote control.

[0005] In a first aspect, an embodiment of the present application provides a remote control method, which is applied to a controlled device, and the method includes:

[0006] After establishing a remote control connection with the first control device, creating a first screen capture task and a second screen capture task, wherein the first screen capture task is a screen capture task based on a Graphics Device Interface (GDI) mode, and the second screen capture task is a screen capture task based on a Direct eXtension (DirectX) mode;

[0007] At a target screen capture frequency, calling the first screen capture task and the second screen capture task to simultaneously perform screen capture;

[0008] Determine whether the first image is the same as the second image, wherein the second image is a previous frame image of the first image obtained by calling the second screen capture task to perform screen capture;

[0009] In the case where the first image is different from the second image, target processing is performed on the image obtained by calling the first screen capture task to perform screen capture, and the step of calling the first screen capture task and the second screen capture task to perform screen capture at the target screen capture frequency is continued, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device;

[0010] When the first image is the same as the second image, stop target processing of the image obtained by calling the first screen capture task to perform screen capture, reduce the target screen capture frequency, use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency, and continue to execute the steps of calling the first screen capture task and the second screen capture task to perform screen capture simultaneously with the target screen capture frequency.

[0011] In a second aspect, an embodiment of the present application provides a remote control device, which is applied to a controlled device, and the device includes:

[0012] A creation module, used to create a first screen capture task and a second screen capture task after establishing a remote control connection with the first control device, wherein the first screen capture task is a screen capture task based on the GDI mode, and the second screen capture task is a screen capture task based on the DirectX mode;

[0013] A screen capture module, used to call the first screen capture task and the second screen capture task to perform screen capture simultaneously at a target screen capture frequency;

[0014] a first judging module, configured to judge whether the first image is identical to the second image, and trigger the first processing module if the first image is different from the second image, and trigger the first stopping module if the first image is identical to the second image, wherein the second image is a previous frame image of the first image obtained by calling the second screen capturing task to perform screen capturing;

[0015] A first processing module is used to perform target processing on the image obtained by calling the first screen capture task to perform screen capture, and trigger the screen capture module, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device;

[0016] A first stopping module is used to stop target processing of the image obtained by screen capture by calling the first screen capture task;

[0017] The reducing module is used to reduce the target screen capture frequency, and uses the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the remote control method provided in the first aspect of the embodiment of the present application is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the remote control method provided in the first aspect of the embodiment of the present application is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the remote control method provided in the first aspect of the embodiment of the present application.

[0021] In an embodiment of the present application, after the controlled device establishes a remote control connection with the first control device, it creates a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode; at the target screen capture frequency, the first screen capture task and the second screen capture task are called to perform screen capture at the same time; it is determined whether the first image is the same as the second image, wherein the second image is the previous frame image of the first image obtained by calling the second screen capture task to perform screen capture; when the first image is different from the second image, the image obtained by calling the first screen capture task to perform screen capture is subjected to target processing, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device; when the first image is the same as the second image, the target processing of the image obtained by calling the first screen capture task to perform screen capture is stopped, and the target screen capture frequency is reduced. In this way, when the first image and the second image are the same, by reducing the screen capture frequency, the resources occupied during screen capture can be reduced, and thus the resources occupied during remote control can be reduced; and, since the screen capture frequency is reduced, the captured images are reduced, and thus the images subjected to target processing are reduced, and thus the resources occupied during remote control can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 is a flowchart of a remote control method provided in an embodiment of the present application;

[0024] Figure 2 This is a schematic diagram of the first process of remote control provided by an embodiment of the present application;

[0025] Figure 3 This is a schematic diagram of a second process of remote control provided by an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the structure of a remote control device provided in an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0029] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0030] The remote control method, apparatus, device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0031] It should be noted that the remote control method and device provided in the embodiments of the present application can be applied to a controlled device. The controlled device can be a terminal or other device other than a terminal. Exemplarily, the controlled device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.

[0032] The controlled device in the embodiment of the present application may be an electronic device having an operating system. The operating system may be a Microsoft (Windows) operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0033] Figure 1 Schematic diagram of the remote control method provided by the embodiment of the present application. Figure 1 As shown, the remote control method may include:

[0034] Step 101: after establishing a remote control connection with a first control device, creating a first screen capture task and a second screen capture task;

[0035] In some possible implementations of the embodiments of the present application, in step 101, the first screen capture task is a screen capture task based on the GDI mode, and the second screen capture task is a screen capture task based on the DirectX mode.

[0036] In some possible implementations of the embodiments of the present application, in step 101, a screen capture (ScreenCapturer) task may be created and set to GDI mode, and then another ScreenCapturer task may be created and set to DirectX mode.

[0037] Step 102: calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at a target screen capture frequency;

[0038] In some possible implementations of the embodiments of the present application, in step 102, the first screen capture task and the second screen capture task simultaneously perform screen capture using the same screen capture frequency.

[0039] Step 103: Determine whether the first image is identical to the second image, if not, execute step 104, if yes, execute step 105;

[0040] In some possible implementations of the embodiments of the present application, the second image is a previous frame image of the first image obtained by calling the second screen capture task to perform screen capture.

[0041] The embodiment of the present application does not limit the method used to determine whether the first image and the second image are the same, and any available method can be applied to the embodiment of the present application. For example, by calculating the similarity between the first image and the second image, it is determined whether the first image and the second image are the same. If the similarity between the first image and the second image is greater than a preset similarity threshold, it is determined that the first image and the second image are the same. If the similarity between the first image and the second image is not greater than the preset similarity threshold, it is determined that the first image and the second image are not the same.

[0042] In some possible implementations of the embodiments of the present application, step 103 may include: determining through the desktop frame whether the update area corresponding to the second image is empty; if the update area is empty, the second image is the same as the first image; if the update area is not empty, the second image is different from the first image, wherein the desktop frame is obtained based on the callback of the second screen capture task.

[0043] Specifically, the desktop frame (DesktopFrame) can be obtained from the callback of the screen capture task based on the DirectX mode, and the updated region (updated_region) method of the DesktopFrame is used to determine whether the updated region of the currently captured video frame (hereinafter referred to as the current video frame) is empty to determine whether the first image and the second image are the same. When the updated region of the current video frame is empty, it means that the current video frame is the same as the previous video frame, that is, the first image and the second image are the same; when the updated region of the current video frame is not empty, it means that the current video frame is different from the previous video frame, that is, the first image and the second image are different.

[0044] Step 104: performing target processing on the image obtained by calling the first screen capture task to perform screen capture, and continuing to execute step 102;

[0045] In some possible implementations of the embodiments of the present application, the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device.

[0046] The process of image conversion, image encoding and sending the converted and encoded image to the control device can be specifically referred to the process of image conversion, image encoding and sending the converted and encoded image to the control device in the relevant design, and the embodiments of the present application will not be described in detail here.

[0047] It should be noted that when the target processing is performed on the image obtained by calling the first screen capture task to perform screen capture, the target processing is not performed on the image obtained by calling the second screen capture task to perform screen capture. In other words, the image obtained by calling the second screen capture task to perform screen capture in the embodiment of the present application is only used to determine whether the images are the same, and the target processing is not performed on the image obtained by calling the second screen capture task to perform screen capture in the embodiment of the present application.

[0048] Step 105: stop performing target processing on the image obtained by calling the first screen capture task to perform screen capture;

[0049] Step 106: lowering the target screen capture frequency, taking the screen capture frequency obtained by lowering the target screen capture frequency as the target screen capture frequency, and continuing to execute step 102.

[0050] Exemplarily, device A and device B establish a remote control connection, device A is the controlled device, and device B is the controlling device. A first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode are created on device A. Then the two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 25 frames per second.

[0051] Assuming that the images captured by the second screen capture task determine that the first 30 frames of images captured at a frequency of 25 frames per second are all different, image conversion and image encoding are performed on the first 30 frames of images obtained by calling the first screen capture task for screen capture, and the images after image conversion and image encoding are sent to device B for display, so that device B can remotely control device A.

[0052] When the 31st frame image is captured, at this time, the image captured by the second screen capture task determines that the 31st frame image is the same as the 30th frame image, then the image conversion and image encoding processing of the 31st frame image captured by calling the first screen capture task is stopped, that is, the image conversion and image encoding processing of the 31st frame image captured by calling the first screen capture task is not performed, and the image is not sent to device B. At this time, the window screen used to control device A in device B stays at the last video frame (that is, the image obtained after the image conversion and image encoding processing of the 30th frame image obtained by calling the first screen capture task for screen capture). Since the image itself has not changed, the window screen used to control device A in device B stays at the last video frame without affecting the use. Then, reduce the screen capture frequency, for example, reduce it to 5 frames per second. Then, call two screen capture tasks to capture the screen of device A at the same time at a frequency of 5 frames per second.

[0053] When the content of the desktop screen changes, the screen capture method based on the GDI mode consumes fewer resources than the screen capture method based on the DirectX mode. However, since the screen capture method based on the GDI mode does not have a video frame update area method, frame conversion, encoding and transmission will also be performed when the content of the desktop screen changes or does not change. However, the screen capture method based on the DirectX mode has a video frame update area method. When the content of the desktop screen changes or does not change, frame conversion, encoding and transmission may not be performed subsequently. At this time, the screen capture method based on the DirectX mode consumes fewer resources than the screen capture method based on the GDI mode. Based on this, the embodiment of the present application combines the screen capture method based on the GDI mode with the screen capture method based on the DirectX mode, and determines whether the content of the desktop screen changes by using the video frame update area method of the screen capture method in the DirectX mode. When the content of the desktop screen does not change, the screen capture frequency is reduced, and the frame conversion, encoding and transmission of the video frames captured by the screen capture method based on the DirectX mode are stopped, so as to achieve the purpose of consuming lower resources when the screen picture is static or changes.

[0054] In an embodiment of the present application, after the controlled device establishes a remote control connection with the first control device, it creates a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode; at the target screen capture frequency, the first screen capture task and the second screen capture task are called to perform screen capture at the same time; it is determined whether the first image is the same as the second image, wherein the second image is the previous frame image of the first image obtained by calling the second screen capture task to perform screen capture; if the first image is different from the second image, the image obtained by calling the first screen capture task to perform screen capture is subjected to target processing, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device; if the first image is the same as the second image, the target screen capture frequency is reduced. In this way, when the first image is the same as the second image, by reducing the screen capture frequency, the resources occupied during screen capture can be reduced, and thus the resources occupied during remote control can be reduced; and, since the screen capture frequency is reduced, the captured images are reduced, and thus the images subjected to target processing are reduced, and thus the resources occupied during remote control can be reduced.

[0055] In some possible implementations of the embodiments of the present application, after step 106, the remote control method provided by the embodiments of the present application may further include: determining whether the target screen capture frequency is greater than or equal to a first screen capture frequency threshold; if the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, continue to execute step 102.

[0056] In some possible implementations of the embodiments of the present application, the first screen capture frequency threshold in the embodiments of the present application may be the minimum frequency value that the screen capture frequency can reach, and the first screen capture frequency threshold in the embodiments of the present application may be set according to actual needs.

[0057] For example, assuming that the first screen capture frequency threshold is 5 frames per second; device A and device B establish a remote control connection, device A is the controlled device, and device B is the controlling device. A first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode are created on device A. Then, the two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 25 frames per second.

[0058] Assuming that the images captured by the second screen capture task determine that the first 30 frames of images captured at a frequency of 25 frames per second are all different, image conversion and image encoding are performed on the first 30 frames of images obtained by calling the first screen capture task for screen capture, and the images after image conversion and image encoding are sent to device B for display, so that device B can remotely control device A.

[0059] When the 31st frame image is captured, at this time, the image captured by the second screen capture task determines that the 31st frame image is the same as the 30th frame image, then stop stopping the image conversion and image encoding processing of the 31st frame image captured by calling the first screen capture task, and reduce the screen capture frequency, for example, to 15 frames per second. At this time, the target screen capture frequency "15 frames per second" is greater than the first screen capture frequency threshold "5 frames per second", and call two screen capture tasks to capture the screen of device A at the same time at a frequency of 15 frames per second.

[0060] In an embodiment of the present application, the screen is captured only when the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, which can avoid the screen capture frequency being too low, the screen capture interval being too long, and the screen changing during the screen capture interval without timely feedback to the control device, resulting in inaccurate control.

[0061] In some possible implementations of the embodiments of the present application, the remote control method provided by the embodiments of the present application may further include: when the target screen capture frequency is less than the first screen capture frequency threshold, assigning the target screen capture frequency to the first screen capture frequency threshold, and continuing to execute step 102.

[0062] For example, assuming that the first screen capture frequency threshold is 5 frames per second; device A and device B establish a remote control connection, device A is the controlled device, and device B is the controlling device. A first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode are created on device A. Then, the two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 14 frames per second.

[0063] Assuming that the images captured by the second screen capture task determine that the first 30 frames of images captured at a frequency of 14 frames per second are all different, image conversion and image encoding are performed on the first 30 frames of images obtained by calling the first screen capture task for screen capture, and the images after image conversion and image encoding are sent to device B for display, so that device B can remotely control device A.

[0064] When the 31st frame image is captured, at this time, the image captured by the second screen capture task determines that the 31st frame image is the same as the 30th frame image, then stop the image conversion and image encoding processing of the 31st frame image captured by calling the first screen capture task, and reduce the screen capture frequency, for example, to 4 frames per second. At this time, the target screen capture frequency "4 frames per second" is less than the first screen capture frequency threshold "5 frames per second", then the target screen capture frequency is assigned to "5 frames per second", and the two screen capture tasks are called to capture the screen of device A at the same time at a frequency of 5 frames per second.

[0065] In an embodiment of the present application, when the target screen capture frequency is not greater than the first screen capture frequency threshold, by assigning the target screen capture frequency to the first screen capture frequency threshold, it is possible to avoid the screen capture frequency being too low and the screen capture interval being too long. During the screen capture interval, the screen changes without timely feedback to the control device, resulting in inaccurate control.

[0066] In some possible implementations of the embodiments of the present application, after step 104, the remote control method provided by the embodiments of the present application may further include: increasing the target screen capture frequency, using the screen capture frequency obtained by increasing the target screen capture frequency as the target screen capture frequency, and continuing to execute step 102.

[0067] Exemplarily, device A and device B establish a remote control connection, device A is the controlled device, and device B is the controlling device. A first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode are created on device A. Then the two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 5 frames per second.

[0068] Assume that the image captured by the second screen capture task determines that the second frame of image captured at a frequency of 5 frames per second is different from the first frame of image, then the image obtained by calling the first screen capture task to perform screen capture is converted and encoded, and the image after the image conversion and encoding is sent to device B for display, so that device B can remotely control device A. Increase the screen capture frequency, for example, to 15 frames per second. Then, call two screen capture tasks to simultaneously capture the screen of device A at a frequency of 15 frames per second.

[0069] Assume that the image captured by the second screen capture task determines that the second frame of image captured at a frequency of 15 frames per second is different from the first frame of image, then the image obtained by calling the first screen capture task to perform screen capture is converted and encoded, and the image after the image conversion and encoding is sent to device B for display, so that device B can remotely control device A. Increase the screen capture frequency, for example, to 25 frames per second. Then, call two screen capture tasks to simultaneously capture the screen of device A at a frequency of 25 frames per second.

[0070] In some possible implementations of the embodiments of the present application, after the screen capture frequency obtained by increasing the target screen capture frequency is used as the target screen capture frequency, the remote control method provided by the embodiments of the present application may also include: determining whether the target screen capture frequency is less than or equal to a second screen capture frequency threshold; if the target screen capture frequency is less than or equal to the second screen capture frequency threshold, continue to execute step 102.

[0071] In some possible implementations of the embodiments of the present application, the second screen capture frequency threshold in the embodiments of the present application may be the maximum frequency value that the screen capture frequency can reach, and the second screen capture frequency threshold in the embodiments of the present application may be set according to actual needs.

[0072] For example, the second screen capture frequency threshold is 25 frames per second; device A and device B establish a remote control connection, device A is the controlled device, and device B is the controlling device. A first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode are created on device A. Then, the two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 10 frames per second.

[0073] Assuming that the image captured by the second screen capture task determines that the second frame of image captured at a frequency of 10 frames per second is different from the first frame of image, the image obtained by calling the first screen capture task to perform screen capture is converted and encoded, and the image after image conversion and encoding is sent to device B for display, so that device B can remotely control device A. And the screen capture frequency is increased, for example, to 20 frames per second. At this time, the target screen capture frequency "20 frames per second" is less than the second screen capture frequency threshold "25 frames per second", and then two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 20 frames per second.

[0074] In the embodiment of the present application, the screen is captured only when the target screen capture frequency is less than or equal to the second screen capture frequency threshold, which can avoid the occurrence of excessive screen capture frequency, frequent screen capture, and resource occupation.

[0075] In some possible implementations of the embodiments of the present application, the remote control method provided by the embodiments of the present application may further include: when the target screen capture frequency is greater than the second screen capture frequency threshold, assigning the target screen capture frequency to the second screen capture frequency threshold, and continuing to execute step 102.

[0076] For example, the second screen capture frequency threshold is 25 frames per second; device A and device B establish a remote control connection, device A is the controlled device, and device B is the controlling device. A first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode are created on device A. Then, the two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 10 frames per second.

[0077] Assuming that the image captured by the second screen capture task determines that the second frame of image captured at a frequency of 10 frames per second is different from the first frame of image, the image obtained by calling the first screen capture task to perform screen capture is converted and encoded, and the image after image conversion and encoding is sent to device B for display, so that device B can remotely control device A. And the screen capture frequency is increased, for example, to 20 frames per second. At this time, the target screen capture frequency "20 frames per second" is less than the second screen capture frequency threshold "25 frames per second", and then two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 20 frames per second.

[0078] Assuming that the image captured by the second screen capture task determines that the second frame of image captured at a frequency of 20 frames per second is different from the first frame of image, the image obtained by calling the first screen capture task to perform screen capture is converted and encoded, and the image after image conversion and encoding is sent to device B for display, so that device B can remotely control device A. The screen capture frequency is increased, for example, to 30 frames per second. At this time, the target screen capture frequency "30 frames per second" is greater than the second screen capture frequency threshold "25 frames per second", and the target screen capture frequency is assigned to 25 frames per second. Then, two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 25 frames per second.

[0079] In an embodiment of the present application, when the target screen capture frequency is greater than the second screen capture frequency threshold, by assigning the target screen capture frequency to the second screen capture frequency threshold, it is possible to avoid the screen capture frequency being too high, frequent screen capture, and occupation of resources.

[0080] It should be noted that the second screen capture frequency threshold in the embodiment of the present application is greater than the first screen capture frequency threshold.

[0081] In some possible implementations of the embodiments of the present application, after lowering the target screen capture frequency, the target screen capture frequency can be compared with a first screen capture frequency threshold. When the target screen capture frequency is less than the first screen capture frequency threshold, the target screen capture frequency is reassigned to the first screen capture frequency threshold. After increasing the target screen capture frequency, the target screen capture frequency can be compared with a second screen capture frequency threshold. When the target screen capture frequency is greater than the first screen capture frequency threshold, the target screen capture frequency is reassigned to the second screen capture frequency threshold.

[0082] That is to say, after lowering the target screen capture frequency, the target screen capture frequency takes the maximum value between the lowered frequency and the first screen capture frequency threshold; after increasing the target screen capture frequency, the target screen capture frequency takes the minimum value between the increased frequency and the second screen capture frequency threshold.

[0083] In the embodiment of the present application, it can be ensured that the target screen capture frequency does not exceed the value range of the first screen capture frequency threshold and the second screen capture frequency threshold.

[0084] In some possible implementations of the embodiments of the present application, there is a situation where a new control device establishes a remote control connection with a controlled device when the first image is the same as the second image. At this time, since the controlled device does not transmit an image, the new control device cannot control the controlled device. Based on this, the remote control method provided by the embodiments of the present application may also include: when the first image is the same as the second image, after the second control device establishes a remote control connection with the controlled device, target processing is performed on the image obtained by screen capture by calling the first screen capture task.

[0085] In the embodiment of the present application, even if the first image is the same as the second image, when a new control device establishes a remote control connection with a controlled device, the image will be sent to the control device, enabling the new control device to control the controlled device.

[0086] In some possible implementations of the embodiments of the present application, after the second control device establishes a remote control connection with the controlled device and before performing target processing on the image obtained by screen capture by calling the first screen capture task, the remote control method provided by the embodiments of the present application may further include: stopping execution of step 103.

[0087] In some possible implementations of the embodiments of the present application, the frame image comparison step 103 can be stopped for a period of time (e.g., 5 seconds), while the first screen capture task and the second screen capture task are performed normally. During this period of time, the control device can receive the image sent by the controlled device, and then control the controlled device based on the received image. After a period of time, step 103 is continued.

[0088] Figure 2 This is a schematic diagram of the first process of remote control provided by the embodiment of the present application. The remote control process includes the following steps:

[0089] Step 201: Establishing a remote control connection with a control device;

[0090] Step 202: creating a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode, and setting a minimum frequency value and a maximum frequency value that the screen capture frequency can reach;

[0091] Step 203: calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency;

[0092] Step 204: determine whether the current frame image obtained by calling the second screen capture task to perform screen capture is the same as the previous frame image of the current frame image. If they are not the same, execute step 205; if they are the same, execute step 208;

[0093] Step 205: performing target processing on the current frame image obtained by calling the first screen capture task to perform screen capture;

[0094] Step 206: Increase the target screen capture frequency;

[0095] Step 207: the target screen capture frequency is assigned a value that is the minimum value between the increased target screen capture frequency and the maximum frequency value, and the process continues with step 203;

[0096] Step 208: reducing the target screen capture frequency;

[0097] Step 209 : the target screen capture frequency is assigned a value equal to the maximum value between the reduced target screen capture frequency and the minimum frequency value, and step 203 is continued.

[0098] Figure 3 This is a schematic diagram of the second process of remote control provided in an embodiment of the present application.

[0099] exist Figure 3 In the embodiment, the controlled device first creates a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode.

[0100] For the first screen capture task based on the GDI mode, the DesktopFrame is obtained from its callback; for the second screen capture task based on the DirectX mode, the DesktopFrame is obtained from its callback.

[0101] Based on the DesktopFrame obtained from the callback of the second screen capture task based on DirectX mode, determine whether updated_region is empty. If updated_region is empty, set the screen capture frequency to 5 frames per second, and the first screen capture task based on GDI mode and the second screen capture task based on DirectX mode perform screen capture at a screen capture frequency of 5 frames per second. Then, perform frame transformation, encoding and transmission on the DesktopFrame obtained from the callback of the first screen capture task based on GDI mode. If updated_region is not empty, set the screen capture frequency to 25 frames per second, and the first screen capture task based on GDI mode and the second screen capture task based on DirectX mode perform screen capture at a screen capture frequency of 25 frames per second. Then, perform frame transformation, encoding and transmission on the DesktopFrame obtained from the callback of the first screen capture task based on GDI mode.

[0102] The first control device receives and displays the frame data sent by the controlled device.

[0103] When a new control device is connected (i.e., the second control device establishes a remote control connection with the controlled device), it is not determined whether updated_region is empty for 5 seconds. During these 5 seconds, the DesktopFrame obtained from the callback of the first screen capture task based on the GDI mode is frame transformed, encoded and transmitted. The first control device and the second control device receive the frame data sent by the controlled device and display them respectively.

[0104] The present application also provides a remote control device, such as Figure 4 shown. Figure 4 4 is a schematic diagram of the structure of a remote control device provided in an embodiment of the present application. The remote control device 400 may include:

[0105] A creation module 401 is used to create a first screen capture task and a second screen capture task after establishing a remote control connection with a first control device, wherein the first screen capture task is a screen capture task based on a GDI mode, and the second screen capture task is a screen capture task based on a DirectX mode;

[0106] The screen capture module 402 is used to call the first screen capture task and the second screen capture task to perform screen capture at the target screen capture frequency;

[0107] A first judging module 403 is used to judge whether the first image is the same as the second image, and trigger the first processing module 404 if the first image is different from the second image, and trigger the first stopping module 405 if the first image is the same as the second image, wherein the second image is the previous frame image of the first image obtained by calling the second screen capturing task to perform screen capturing;

[0108] The first processing module 404 is used to perform target processing on the image obtained by calling the first screen capture task to perform screen capture, and trigger the screen capture module 402, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device;

[0109] The first stop module 405 is used to stop the target processing of the image obtained by calling the first screen capture task to capture the screen

[0110] The reducing module 406 is used to reduce the target screen capture frequency, and use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module 402.

[0111] In an embodiment of the present application, after the controlled device establishes a remote control connection with the first control device, it creates a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode; at the target screen capture frequency, the first screen capture task and the second screen capture task are called to perform screen capture at the same time; it is determined whether the first image is the same as the second image, wherein the second image is the previous frame image of the first image obtained by calling the second screen capture task to perform screen capture; if the first image is different from the second image, the image obtained by calling the first screen capture task to perform screen capture is subjected to target processing, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to the control device; if the first image is the same as the second image, the target screen capture frequency is reduced. In this way, when the first image is the same as the second image, by reducing the screen capture frequency, the resources occupied during screen capture can be reduced, and thus the resources occupied during remote control can be reduced; and, since the screen capture frequency is reduced, the captured images are reduced, and thus the images subjected to target processing are reduced, and thus the resources occupied during remote control can be reduced.

[0112] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0113] The second judgment module is used to judge whether the target screen capture frequency is greater than or equal to the first screen capture frequency threshold; when the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, the screen capture module 402 is triggered.

[0114] In an embodiment of the present application, the screen is captured only when the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, which can avoid the screen capture frequency being too low, the screen capture interval being too long, and the screen changing during the screen capture interval without timely feedback to the control device, resulting in inaccurate control.

[0115] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0116] The first assignment module is configured to assign the target screen capture frequency to the first screen capture frequency threshold when the target screen capture frequency is less than the first screen capture frequency threshold, thereby triggering the screen capture module 402 .

[0117] In an embodiment of the present application, when the target screen capture frequency is less than the first screen capture frequency threshold, by assigning the target screen capture frequency to the first screen capture frequency threshold, it is possible to avoid the screen capture frequency being too low and the screen capture interval being too long. During the screen capture interval, the screen changes without timely feedback to the control device, resulting in inaccurate control.

[0118] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0119] The increasing module is used to increase the target screen capture frequency, and uses the screen capture frequency obtained by increasing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module 402.

[0120] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0121] The third judgment module is used to judge whether the target screen capture frequency is less than or equal to the second screen capture frequency threshold; when the target screen capture frequency is less than or equal to the second screen capture frequency threshold, the screen capture module 402 is triggered.

[0122] In the embodiment of the present application, the screen is captured only when the target screen capture frequency is less than or equal to the second screen capture frequency threshold, which can avoid the occurrence of excessive screen capture frequency, frequent screen capture, and resource occupation.

[0123] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0124] The second assignment module is configured to assign the target screen capture frequency to the second screen capture frequency threshold when the target screen capture frequency is greater than the second screen capture frequency threshold, thereby triggering the screen capture module 402 .

[0125] In an embodiment of the present application, when the target screen capture frequency is greater than the second screen capture frequency threshold, by assigning the target screen capture frequency to the second screen capture frequency threshold, it is possible to avoid the screen capture frequency being too high, frequent screen capture, and occupation of resources.

[0126] In the embodiment of the present application, it can be ensured that the target screen capture frequency does not exceed the value range of the first screen capture frequency threshold and the second screen capture frequency threshold.

[0127] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0128] The second processing module is used for performing target processing on the image obtained by screen capturing by calling the first screen capture task after the second control device establishes a remote control connection with the controlled device when the first image is identical to the second image.

[0129] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0130] The second stopping module is used to stop determining whether the first image is identical to the second image after the second controlling device establishes a remote control connection with the controlled device.

[0131] In some possible implementations of the embodiment of the present application, the first determination module 403 is specifically configured to:

[0132] The desktop frame is used to determine whether the update area corresponding to the second image is empty. If the update area is empty, the second image is the same as the first image. If the update area is not empty, the second image is different from the first image. The desktop frame is obtained based on the callback of the second screen capture task.

[0133] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0134] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0135] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0136] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the electronic device. In some specific embodiments, the memory 502 is a non-volatile solid-state memory.

[0137] In some specific embodiments, the memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the remote control method according to the present application.

[0138] The processor 501 implements the remote control method provided in the embodiment of the present application by reading and executing the computer program instructions stored in the memory 502.

[0139] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0140] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0141] The bus 510 includes hardware, software or both, coupling the components of the electronic device to each other. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro channel architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus or other suitable buses or a combination of two or more of these. Where appropriate, the bus 510 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0142] The electronic device can execute the remote control method provided in the embodiment of the present application, thereby achieving the corresponding technical effects of the remote control method provided in the embodiment of the present application.

[0143] In addition, in combination with the remote control method in the above embodiment, the embodiment of the present application also provides a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the remote control method provided in the embodiment of the present application is implemented. Examples of computer-readable storage media include non-transitory computer-readable media, such as ROM, RAM, magnetic disk or optical disk, etc.

[0144] An embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the remote control method provided in the embodiment of the present application and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0145] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0146] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (Erasable Read Only Memory, EROM), floppy disks, compact discs (Compact Disc Read-Only Memory, CD-ROM), optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0147] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0148] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0149] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A remote control method, characterized in that: Applied to a controlled device, the method comprises: After establishing a remote control connection with the first control device, creating a first screen capture task and a second screen capture task, wherein the first screen capture task is a screen capture task based on a graphics device interface GDI mode, and the second screen capture task is a screen capture task based on a direct extension DirectX mode; At a target screen capture frequency, calling the first screen capture task and the second screen capture task to simultaneously perform screen capture; Determine whether the first image is the same as the second image, wherein the second image is a previous frame image of the first image obtained by calling the second screen capture task to perform screen capture; In the case that the first image is different from the second image, target processing is performed on the image obtained by calling the first screen capture task to perform screen capture, and the step of calling the first screen capture task and the second screen capture task to perform screen capture at the target screen capture frequency is continued, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device; When the first image is the same as the second image, stop performing the target processing on the image obtained by calling the first screen capture task to perform screen capture, reduce the target screen capture frequency, use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency, and continue to execute the step of calling the first screen capture task and the second screen capture task to perform screen capture simultaneously with the target screen capture frequency.

2. The method according to claim 1, characterized in that After the screen capture frequency obtained by reducing the target screen capture frequency is used as the target screen capture frequency, the method further includes: Determining whether the target screen capture frequency is greater than or equal to a first screen capture frequency threshold; When the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, continue to execute the step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency.

3. The method according to claim 2, characterized in that The method further comprises: When the target screen capture frequency is less than the first screen capture frequency threshold, the target screen capture frequency is assigned to the first screen capture frequency threshold, and the step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency is continued.

4. The method according to claim 1, characterized in that: After performing target processing on the image obtained by screen capture by calling the first screen capture task, the method further includes: The target screen capture frequency is increased, and the screen capture frequency obtained by increasing the target screen capture frequency is used as the target screen capture frequency. The step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency is continued.

5. The method according to claim 4, characterized in that After the screen capture frequency obtained by increasing the target screen capture frequency is used as the target screen capture frequency, the method further includes: Determining whether the target screen capture frequency is less than or equal to a second screen capture frequency threshold; When the target screen capture frequency is less than or equal to the second screen capture frequency threshold, continue to execute the step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency.

6. The method according to claim 5, characterized in that The method further comprises: When the target screen capture frequency is greater than the second screen capture frequency threshold, the target screen capture frequency is assigned to the second screen capture frequency threshold, and the step of calling the first screen capture task and the second screen capture task to perform screen capture simultaneously at the target screen capture frequency is continued.

7. The method according to claim 1, characterized in that The method further comprises: In the case that the first image is identical to the second image, after the second controlling device establishes a remote control connection with the controlled device, the target processing is performed on the image obtained by calling the first screen capturing task to perform screen capturing.

8. The method according to claim 7, characterized in that Before performing the target processing on the image obtained by screen capturing by calling the first screen capture task, the method further includes: Stop executing the step of determining whether the first image is identical to the second image.

9. The method according to claim 1, characterized in that: The determining whether the first image is identical to the second image includes: The desktop frame is used to determine whether the update area corresponding to the second image is empty. If the update area is empty, the second image is the same as the first image. If the update area is not empty, the second image is different from the first image. The desktop frame is obtained based on the callback of the second screen capture task.

10. A remote control device, characterized in that: Applied to a controlled device, the device comprises: A creation module, used for creating a first screen capture task and a second screen capture task after establishing a remote control connection with a first control device, wherein the first screen capture task is a screen capture task based on a graphics device interface GDI mode, and the second screen capture task is a screen capture task based on a direct extension DirectX mode; A screen capture module, used to call the first screen capture task and the second screen capture task to perform screen capture simultaneously at a target screen capture frequency; a first judging module, configured to judge whether the first image is identical to the second image, and trigger the first processing module if the first image is different from the second image, and trigger the first stopping module if the first image is identical to the second image, wherein the second image is a previous frame image of the first image obtained by calling the second screen capturing task to perform screen capturing; A first processing module, configured to perform target processing on the image obtained by calling the first screen capture task to perform screen capture, and trigger the screen capture module, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to a control device; A first stopping module, used for stopping the target processing of the image obtained by screen capturing by calling the first screen capture task; A reducing module is used to reduce the target screen capture frequency, and use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module.

11. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the remote control method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the remote control method according to any one of claims 1 to 9 is implemented.

Images