Remote control method and apparatus, and device and medium

Abstract

The present application relates to the technical field of remote control. Disclosed are a remote control method and apparatus, and a device and a medium. The remote control method, which is applied to a controlled device, comprises: after establishing a remote control connection with a first control device, creating a first screen capturer task based on a graphics device interface (GDI) mode and a second screen capturer task based on a direct extension (DirectX) mode; calling the first screen capturer task and the second screen capturer task at a target screen capturer frequency to simultaneously perform screen capture; determining whether a first image is the same as a second image, wherein the second image is the previous image frame of the first image obtained by means of calling the second screen capturer task to perform screen capture; if the first image is different from the second image, performing target processing on an image obtained by means of calling the first screen capturer task to perform screen capture; and if the first image is the same as the second image, reducing the target screen capturer frequency. According to the embodiments of the present application, resources occupied by remote control can be reduced.

Description

Remote control method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311664930.2 filed in China on December 5, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of remote control technology, and in particular relates to a remote control method, apparatus, device and medium. Background Art

[0004] Remote control refers to connecting a control device to a controlled device, displaying the controlled device's desktop environment on the control device, and controlling the controlled device (e.g., configuring, installing software, modifying, etc.) through the control device.

[0005] During remote control, the controlled device needs to perform screen capture, image conversion, image encoding and data transmission on its desktop so that the controlling device can control the controlled device. However, in related technologies, the resources occupied (for example, CPU resources, network resources, etc.) during remote control are relatively high.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a remote control method, apparatus, device, and medium, which can solve the problem of high resource usage of remote control.

[0008] In a first aspect, an embodiment of the present application provides a remote control method, applied to a controlled device, the method comprising:

[0009] After establishing a remote control connection with the first control device, creating a first screen capture task and a second screen capture task, wherein the first screen capture task is a screen capture task based on a Graphics Device Interface (GDI) mode, and the second screen capture task is a screen capture task based on a Direct eXtension (DirectX) mode;

[0010] At the target screen capture frequency, calling the first screen capture task and the second screen capture task to simultaneously perform screen capture;

[0011] Determining whether the first image is identical to the second image, wherein the second image is a previous frame of the first image obtained by calling the second screen capture task to perform screen capture;

[0012] When the first image is different from the second image, performing target processing on the image obtained by calling the first screen capture task to perform screen capture, and continuing to perform the step of calling the first screen capture task and the second screen capture task to perform screen capture simultaneously at the target screen capture frequency, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device;

[0013] When the first image is identical to the second image, stop target processing of the image obtained by calling the first screen capture task for screen capture, reduce the target screen capture frequency, use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency, and continue to execute the steps of calling the first screen capture task and the second screen capture task to perform screen capture at the same time with the target screen capture frequency.

[0014] In a second aspect, an embodiment of the present application provides a remote control device, which is applied to a controlled device, and includes:

[0015] A creation module, configured to create a first screen capture task and a second screen capture task after establishing a remote control connection with the first control device, wherein the first screen capture task is a screen capture task based on the GDI mode, and the second screen capture task is a screen capture task based on the DirectX mode;

[0016] A screen capture module, configured to call a first screen capture task and a second screen capture task to simultaneously perform screen capture at a target screen capture frequency;

[0017] a first determining module, configured to determine whether the first image and the second image are identical, and triggering the first processing module if the first image and the second image are different, and triggering the first stopping module if the first image and the second image are identical, wherein the second image is an image frame preceding the first image obtained by screen capturing by calling the second screen capture task;

[0018] a first processing module, configured to perform target processing on the image obtained by calling the first screen capture task to perform screen capture, and trigger the screen capture module, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device;

[0019] A first stopping module is used to stop target processing of the image obtained by screen capture by calling the first screen capture task;

[0020] The reducing module is used to reduce the target screen capture frequency, and uses the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the remote control method provided in the first aspect of the embodiment of the present application is implemented.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the remote control method provided in the first aspect of the embodiment of the present application is implemented.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the remote control method provided in the first aspect of the embodiment of the present application.

[0024] In an embodiment of the present application, after establishing a remote control connection with a first control device, a controlled device creates a first screen capture task based on GDI mode and a second screen capture task based on DirectX mode; calls the first screen capture task and the second screen capture task simultaneously to perform screen capture at a target screen capture frequency; determines whether a first image and a second image are identical, wherein the second image is a frame preceding the first image captured by calling the second screen capture task; if the first image and the second image are different, performs target processing on the image captured by calling the first screen capture task, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device; if the first image and the second image are identical, stops performing target processing on the image captured by calling the first screen capture task, and reduces the target screen capture frequency. In this way, when the first image and the second image are identical, by reducing the screen capture frequency, the resources occupied by screen capture can be reduced, thereby reducing the resources occupied by remote control; and because the screen capture frequency is reduced, fewer images are captured, and thus fewer images undergo target processing, which can also reduce the resources occupied by remote control. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] FIG1 is a schematic diagram of a flow chart of a remote control method provided in an embodiment of the present application;

[0027] FIG2 is a schematic diagram of a first process of remote control provided by an embodiment of the present application;

[0028] FIG3 is a schematic diagram of a second process of remote control provided by an embodiment of the present application;

[0029] FIG4 is a schematic structural diagram of a remote control device provided in an embodiment of the present application;

[0030] FIG5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0032] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0033] The remote control method, apparatus, device and medium provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0034] It should be noted that the remote control method and apparatus provided in the embodiments of the present application can be applied to a controlled device. The controlled device can be a terminal or other device other than a terminal. For example, the controlled device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), an ATM or a self-service machine, etc., and the embodiments of the present application do not specifically limit it.

[0035] The controlled device in the embodiment of the present application may be an electronic device having an operating system. The operating system may be a Microsoft (Windows) operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0036] FIG1 is a flow chart of a remote control method provided by an embodiment of the present application. As shown in FIG1 , the remote control method may include:

[0037] Step 101: After establishing a remote control connection with a first control device, create a first screen capture task and a second screen capture task;

[0038] In some possible implementations of the embodiments of the present application, in step 101, the first screen capture task is a screen capture task based on the GDI mode, and the second screen capture task is a screen capture task based on the DirectX mode.

[0039] In some possible implementations of the embodiments of the present application, in step 101, a screen capture (ScreenCapturer) task may be created and set to GDI mode, and then another ScreenCapturer task may be created and set to DirectX mode.

[0040] Step 102: calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency;

[0041] In some possible implementations of the embodiments of the present application, in step 102, the first screen capture task and the second screen capture task perform screen capture simultaneously using the same screen capture frequency.

[0042] Step 103: Determine whether the first image and the second image are the same, if not, execute step 104, if yes, execute step 105;

[0043] In some possible implementations of the embodiments of the present application, the second image is a previous frame image of the first image obtained by calling the second screen capture task to perform screen capture.

[0044] The embodiments of the present application do not limit the method used to determine whether the first image and the second image are identical; any applicable method may be applied to the embodiments of the present application. For example, the determination of whether the first image and the second image are identical may be made by calculating the similarity between the first image and the second image. If the similarity between the first image and the second image is greater than a preset similarity threshold, the determination is made that the first image and the second image are identical; if the similarity between the first image and the second image is not greater than the preset similarity threshold, the determination is made that the first image and the second image are not identical.

[0045] In some possible implementations of the embodiments of the present application, step 103 may include: determining through the desktop frame whether the update area corresponding to the second image is empty; if the update area is empty, the second image is the same as the first image; if the update area is not empty, the second image is different from the first image, wherein the desktop frame is obtained based on the callback of the second screen capture task.

[0046] Specifically, a desktop frame (DesktopFrame) can be obtained from the callback of a screen capture task based on DirectX mode. The updated region (updated_region) method of DesktopFrame is used to determine whether the updated region of the currently captured video frame (hereinafter referred to as the current video frame) is empty to determine whether the first image and the second image are the same. When the updated region of the current video frame is empty, it indicates that the current video frame is the same as the previous video frame, that is, the first image and the second image are the same; when the updated region of the current video frame is not empty, it indicates that the current video frame is different from the previous video frame, that is, the first image and the second image are different.

[0047] Step 104: performing target processing on the image obtained by calling the first screen capture task to capture the screen, and continuing to step 102;

[0048] In some possible implementations of the embodiments of the present application, the target processing includes image conversion, image encoding, and sending the image after image conversion and image encoding to the control device.

[0049] The process of image conversion, image encoding and sending the converted and encoded image to the control device can be specifically referred to in the relevant design, and the embodiment of the present application will not go into details here.

[0050] It should be noted that when target processing is performed on the image obtained by calling the first screen capture task to perform screen capture, target processing is not performed on the image obtained by calling the second screen capture task to perform screen capture. In other words, in this embodiment of the application, the image obtained by calling the second screen capture task to perform screen capture is only used to determine whether the images are identical, and in this embodiment of the application, target processing is not performed on the image obtained by calling the second screen capture task to perform screen capture.

[0051] Step 105: Stop performing target processing on the image captured by calling the first screen capture task;

[0052] Step 106 : lowering the target screen capture frequency, taking the screen capture frequency obtained by lowering the target screen capture frequency as the target screen capture frequency, and continuing to execute step 102 .

[0053] For example, device A and device B establish a remote control connection, with device A serving as the controlled device and device B as the controlling device. A first screen capture task based on GDI mode and a second screen capture task based on DirectX mode are created on device A. Both screen capture tasks are then called to simultaneously capture device A's screen at a frequency of 25 frames per second.

[0054] Assume that the images captured by the second screen capture task determine that the first 30 frames of images captured at a frequency of 25 frames per second are all different, then the first 30 frames of images obtained by calling the first screen capture task for screen capture are converted and encoded, and the images after image conversion and image encoding are sent to device B for display, so that device B can remotely control device A.

[0055] When the 31st frame is captured, the image captured by the second screen capture task determines that the 31st frame is identical to the 30th frame. Therefore, the image conversion and encoding process for the 31st frame captured by the first screen capture task is stopped. That is, the image conversion and encoding process for the 31st frame captured by the first screen capture task is not performed, and the image is not sent to device B. At this point, the window screen on device B used to control device A remains at the last video frame (i.e., the image obtained by performing the image conversion and encoding process on the 30th frame captured by the first screen capture task). Since the image itself has not changed, the window screen on device B used to control device A remains at the last video frame without affecting its use. The screen capture frequency is then reduced, for example, to 5 frames per second. Then, two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 5 frames per second.

[0056] When the desktop screen content changes, the screen capture method based on the GDI mode consumes fewer resources than the screen capture method based on the DirectX mode. However, because the screen capture method based on the GDI mode does not have a video frame update region method, frame conversion, encoding, and transmission will be performed even when the desktop screen content has not changed. The screen capture method based on the DirectX mode, on the other hand, has a video frame update region method. When the desktop screen content has not changed, subsequent frame conversion, encoding, and transmission may not be performed. In this case, the screen capture method based on the DirectX mode consumes fewer resources than the screen capture method based on the GDI mode. Based on this, the embodiments of the present application combine the screen capture method based on the GDI mode with the screen capture method based on the DirectX mode. The video frame update region method of the screen capture method in the DirectX mode is used to determine whether the desktop screen content has changed. When the desktop screen content has not changed, the screen capture frequency is reduced, and frame conversion, encoding, and transmission of the video frames captured by the screen capture method in the DirectX mode are stopped. This achieves the goal of lowering resource consumption both when the screen image is static and when it changes.

[0057] In an embodiment of the present application, after establishing a remote control connection with a first control device, a controlled device creates a first screen capture task based on GDI mode and a second screen capture task based on DirectX mode; calls the first screen capture task and the second screen capture task simultaneously to perform screen capture at a target screen capture frequency; determines whether a first image and a second image are identical, wherein the second image is a frame preceding the first image captured by calling the second screen capture task; if the first image and the second image are different, performs target processing on the image captured by calling the first screen capture task, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device; and if the first image and the second image are identical, reduces the target screen capture frequency. Thus, if the first image and the second image are identical, reducing the screen capture frequency can reduce the resources occupied by screen capture, thereby reducing the resources occupied by remote control. Furthermore, since the screen capture frequency is reduced, fewer images are captured, and thus fewer images undergo target processing, thereby reducing the resources occupied by remote control.

[0058] In some possible implementations of the embodiments of the present application, after step 106, the remote control method provided by the embodiments of the present application may further include: determining whether the target screen capture frequency is greater than or equal to a first screen capture frequency threshold; if the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, continuing to execute step 102.

[0059] In some possible implementations of the embodiments of the present application, the first screen capture frequency threshold in the embodiments of the present application may be the minimum frequency value that the screen capture frequency can reach. The first screen capture frequency threshold in the embodiments of the present application may be set according to actual needs.

[0060] For example, assume the first screen capture frequency threshold is 5 frames per second. Device A and device B establish a remote control connection, with device A acting as the controlled device and device B acting as the controlling device. A first screen capture task based on GDI mode and a second screen capture task based on DirectX mode are created on device A. Both screen capture tasks are then called simultaneously to capture device A's screen at a frequency of 25 frames per second.

[0061] Assume that the images captured by the second screen capture task determine that the first 30 frames of images captured at a frequency of 25 frames per second are all different, then the first 30 frames of images obtained by calling the first screen capture task for screen capture are converted and encoded, and the images after image conversion and image encoding are sent to device B for display, so that device B can remotely control device A.

[0062] When the 31st frame image is captured, at this time, the image captured by the second screen capture task determines that the 31st frame image is the same as the 30th frame image, then stop the image conversion and image encoding processing of the 31st frame image captured by calling the first screen capture task, and reduce the screen capture frequency, for example, to 15 frames per second. At this time, the target screen capture frequency "15 frames per second" is greater than the first screen capture frequency threshold "5 frames per second", and call two screen capture tasks to capture the screen of device A at the same time at a frequency of 15 frames per second.

[0063] In an embodiment of the present application, screen capture is continued only when the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, which can avoid the situation where the screen capture frequency is too low, the screen capture interval is too long, and the screen changes during the screen capture interval without timely feedback to the control device, resulting in inaccurate control.

[0064] In some possible implementations of the embodiments of the present application, the remote control method provided by the embodiments of the present application may further include: when the target screen capture frequency is less than the first screen capture frequency threshold, assigning the target screen capture frequency to the first screen capture frequency threshold, and continuing to execute step 102.

[0065] For example, assume the first screen capture frequency threshold is 5 frames per second. Device A and device B establish a remote control connection, with device A acting as the controlled device and device B acting as the controlling device. A first screen capture task based on GDI mode and a second screen capture task based on DirectX mode are created on device A. Both screen capture tasks are then called to simultaneously capture device A's screen at a frequency of 14 frames per second.

[0066] Assume that the images captured by the second screen capture task determine that the first 30 frames of images captured at a frequency of 14 frames per second are all different, then the first 30 frames of images obtained by calling the first screen capture task for screen capture are subjected to image conversion and image encoding processing, and the images after image conversion and image encoding processing are sent to device B for display, so that device B can remotely control device A.

[0067] When the 31st frame image is captured, at this time, the image captured by the second screen capture task determines that the 31st frame image is the same as the 30th frame image, then stop the image conversion and image encoding processing of the 31st frame image captured by calling the first screen capture task, and reduce the screen capture frequency, for example, to 4 frames per second. At this time, the target screen capture frequency "4 frames per second" is less than the first screen capture frequency threshold "5 frames per second", then the target screen capture frequency is assigned to "5 frames per second", and the two screen capture tasks are called to capture the screen of device A at the same time at a frequency of 5 frames per second.

[0068] In an embodiment of the present application, when the target screen capture frequency is not greater than the first screen capture frequency threshold, by assigning the target screen capture frequency to the first screen capture frequency threshold, it is possible to avoid the screen capture frequency being too low and the screen capture interval being too long. During the screen capture interval, the screen changes without timely feedback to the control device, resulting in inaccurate control.

[0069] In some possible implementations of the embodiments of the present application, after step 104, the remote control method provided by the embodiments of the present application may further include: increasing the target screen capture frequency, using the screen capture frequency obtained by increasing the target screen capture frequency as the target screen capture frequency, and continuing to execute step 102.

[0070] For example, device A and device B establish a remote control connection, with device A serving as the controlled device and device B as the controlling device. A first screen capture task based on GDI mode and a second screen capture task based on DirectX mode are created on device A. Both screen capture tasks are then called to simultaneously capture device A's screen at a rate of 5 frames per second.

[0071] Assume that the image captured by the second screen capture task determines that the second image frame, captured at a frequency of 5 frames per second, is different from the first image frame. The image captured by the first screen capture task is converted and encoded, and the converted and encoded image is sent to device B for display, allowing device B to remotely control device A. The screen capture frequency is increased, for example, to 15 frames per second. Then, two screen capture tasks are called simultaneously to capture device A's screen at a frequency of 15 frames per second.

[0072] Assume that the image captured by the second screen capture task determines that the second image frame, captured at a frequency of 15 frames per second, is different from the first image frame. The image captured by the first screen capture task is converted and encoded, and the converted and encoded image is sent to device B for display, allowing device B to remotely control device A. The screen capture frequency is increased, for example, to 25 frames per second. Then, two screen capture tasks are called to simultaneously capture the screen of device A at a frequency of 25 frames per second.

[0073] In some possible implementations of the embodiments of the present application, after the screen capture frequency obtained by increasing the target screen capture frequency is used as the target screen capture frequency, the remote control method provided by the embodiments of the present application may further include: determining whether the target screen capture frequency is less than or equal to a second screen capture frequency threshold; if the target screen capture frequency is less than or equal to the second screen capture frequency threshold, continuing to execute step 102.

[0074] In some possible implementations of the embodiments of the present application, the second screen capture frequency threshold in the embodiments of the present application may be the maximum frequency value that the screen capture frequency can reach. The second screen capture frequency threshold in the embodiments of the present application may be set according to actual needs.

[0075] For example, the second screen capture frequency threshold is 25 frames per second. Device A and device B establish a remote control connection, with device A serving as the controlled device and device B as the controlling device. A first screen capture task based on GDI mode and a second screen capture task based on DirectX mode are created on device A. Both screen capture tasks are then called to simultaneously capture device A's screen at a frequency of 10 frames per second.

[0076] Assume that the image captured by the second screen capture task determines that the second image frame, captured at a frequency of 10 frames per second, is different from the first image frame. The image captured by the first screen capture task is converted and encoded, and the converted and encoded image is sent to device B for display, allowing device B to remotely control device A. The screen capture frequency is then increased, for example to 20 frames per second. At this point, the target screen capture frequency of "20 frames per second" is less than the second screen capture frequency threshold of "25 frames per second." Therefore, two screen capture tasks are invoked to simultaneously capture device A's screen at a frequency of 20 frames per second.

[0077] In the embodiment of the present application, the screen is captured only when the target screen capture frequency is less than or equal to the second screen capture frequency threshold, which can avoid the occurrence of excessive screen capture frequency, frequent screen capture, and resource occupation.

[0078] In some possible implementations of the embodiments of the present application, the remote control method provided by the embodiments of the present application may further include: when the target screen capture frequency is greater than the second screen capture frequency threshold, assigning the target screen capture frequency to the second screen capture frequency threshold, and continuing to execute step 102.

[0079] For example, the second screen capture frequency threshold is 25 frames per second. Device A and device B establish a remote control connection, with device A serving as the controlled device and device B as the controlling device. A first screen capture task based on GDI mode and a second screen capture task based on DirectX mode are created on device A. Both screen capture tasks are then called to simultaneously capture device A's screen at a frequency of 10 frames per second.

[0080] Assume that the image captured by the second screen capture task determines that the second image frame, captured at a frequency of 10 frames per second, is different from the first image frame. The image captured by the first screen capture task is converted and encoded, and the converted and encoded image is sent to device B for display, allowing device B to remotely control device A. The screen capture frequency is then increased, for example to 20 frames per second. At this point, the target screen capture frequency of "20 frames per second" is less than the second screen capture frequency threshold of "25 frames per second." Therefore, two screen capture tasks are invoked to simultaneously capture device A's screen at a frequency of 20 frames per second.

[0081] Assume that the second screen capture task determines that the second image frame, captured at a frequency of 20 frames per second, is different from the first image frame. The image captured by the first screen capture task is then converted and encoded, and the converted and encoded image is sent to device B for display, allowing device B to remotely control device A. The screen capture frequency is then increased, for example, to 30 frames per second. At this point, the target screen capture frequency of "30 frames per second" is greater than the second screen capture frequency threshold of "25 frames per second." Therefore, the target screen capture frequency is set to 25 frames per second. Then, both screen capture tasks are called simultaneously to capture device A's screen at a frequency of 25 frames per second.

[0082] In an embodiment of the present application, when the target screen capture frequency is greater than the second screen capture frequency threshold, by assigning the target screen capture frequency to the second screen capture frequency threshold, it is possible to avoid the screen capture frequency being too high, frequent screen capture, and resource occupation.

[0083] It should be noted that, in the embodiment of the present application, the second screen capture frequency threshold is greater than the first screen capture frequency threshold.

[0084] In some possible implementations of the embodiments of the present application, after lowering the target screen capture frequency, the target screen capture frequency can be compared with a first screen capture frequency threshold. When the target screen capture frequency is less than the first screen capture frequency threshold, the target screen capture frequency is reassigned to the first screen capture frequency threshold; after increasing the target screen capture frequency, the target screen capture frequency can be compared with a second screen capture frequency threshold. When the target screen capture frequency is greater than the first screen capture frequency threshold, the target screen capture frequency is reassigned to the second screen capture frequency threshold.

[0085] That is, after lowering the target screen capture frequency, the target screen capture frequency takes the maximum value between the lowered frequency and the first screen capture frequency threshold; after increasing the target screen capture frequency, the target screen capture frequency takes the minimum value between the increased frequency and the second screen capture frequency threshold.

[0086] In the embodiment of the present application, it is possible to ensure that the target screen capture frequency does not exceed the value range of the first screen capture frequency threshold and the second screen capture frequency threshold.

[0087] In some possible implementations of the embodiments of the present application, there is a situation where a new control device establishes a remote control connection with a controlled device when the first image and the second image are identical. In this case, because the controlled device does not transmit an image, the new control device cannot control the controlled device. Based on this, the remote control method provided in the embodiments of the present application may also include: when the first image and the second image are identical, after the second control device establishes a remote control connection with the controlled device, performing target processing on the image obtained by screen capture by calling the first screen capture task.

[0088] In the embodiment of the present application, even if the first image is the same as the second image, when a new control device establishes a remote control connection with the controlled device, the image will be sent to the control device, enabling the new control device to control the controlled device.

[0089] In some possible implementations of the embodiments of the present application, after the second control device establishes a remote control connection with the controlled device, and before performing target processing on the image obtained by calling the first screen capture task to perform screen capture, the remote control method provided by the embodiments of the present application may further include: stopping execution of step 103.

[0090] In some possible implementations of the present invention, step 103 of the frame image comparison can be paused for a period of time (e.g., 5 seconds), while the first and second screen capture tasks proceed normally. During this period, the controlling device can receive the images sent by the controlled device and then control the controlled device based on the received images. After the period of time, step 103 is resumed.

[0091] FIG2 is a schematic diagram of the first process of remote control provided by an embodiment of the present application. The remote control process includes the following steps:

[0092] Step 201: Establishing a remote control connection with a control device;

[0093] Step 202: creating a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode, and setting the minimum frequency value and the maximum frequency value that the screen capture frequency can reach;

[0094] Step 203: calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency;

[0095] Step 204: Determine whether the current frame image obtained by calling the second screen capture task to perform screen capture is the same as the previous frame image of the current frame image. If they are not the same, execute step 205; if they are the same, execute step 208;

[0096] Step 205: performing target processing on the current frame image obtained by calling the first screen capture task to perform screen capture;

[0097] Step 206: Increase the target screen capture frequency;

[0098] Step 207: The target screen capture frequency is assigned a minimum value between the increased target screen capture frequency and the maximum frequency value, and the process continues with step 203;

[0099] Step 208: Reduce the target screen capture frequency;

[0100] Step 209 : The target screen capture frequency is assigned a value equal to the maximum value between the lowered target screen capture frequency and the minimum frequency value, and the process continues with step 203 .

[0101] FIG3 is a schematic diagram of a second process of remote control provided in an embodiment of the present application.

[0102] In FIG3 , the controlled device first creates a first screen capture task based on the GDI mode and a second screen capture task based on the DirectX mode.

[0103] For the first screen capture task based on GDI mode, DesktopFrame is obtained from its callback; for the second screen capture task based on DirectX mode, DesktopFrame is obtained from its callback.

[0104] Based on the DesktopFrame obtained from the callback of the second screen capture task based on DirectX mode, determine whether updated_region is empty. If updated_region is empty, set the screen capture frequency to 5 frames per second, and the first screen capture task based on GDI mode and the second screen capture task based on DirectX mode perform screen capture at a screen capture frequency of 5 frames per second. Then, perform frame conversion, encoding and transmission on the DesktopFrame obtained from the callback of the first screen capture task based on GDI mode. If updated_region is not empty, set the screen capture frequency to 25 frames per second, and the first screen capture task based on GDI mode and the second screen capture task based on DirectX mode perform screen capture at a screen capture frequency of 25 frames per second. Then, perform frame conversion, encoding and transmission on the DesktopFrame obtained from the callback of the first screen capture task based on GDI mode.

[0105] The first control device receives and displays the frame data sent by the controlled device.

[0106] When a new control device is connected (i.e., a second control device establishes a remote control connection with the controlled device), the updated_region is not checked for null for 5 seconds. During this 5-second period, the DesktopFrame obtained from the callback of the first screen capture task in GDI mode is transformed, encoded, and transmitted. The first and second control devices each receive and display the frame data sent by the controlled device.

[0107] The present application also provides a remote control device, as shown in Figure 4. Figure 4 is a schematic diagram of the structure of the remote control device provided in the present application. The remote control device 400 may include:

[0108] A creation module 401 is configured to create a first screen capture task and a second screen capture task after establishing a remote control connection with a first control device, wherein the first screen capture task is a screen capture task based on a GDI mode, and the second screen capture task is a screen capture task based on a DirectX mode;

[0109] The screen capture module 402 is configured to call the first screen capture task and the second screen capture task to simultaneously perform screen capture at a target screen capture frequency;

[0110] A first determination module 403 is configured to determine whether the first image and the second image are identical. If the first image and the second image are different, the first processing module 404 is triggered. If the first image and the second image are identical, the first stopping module 405 is triggered. The second image is a previous frame of the first image captured by calling the second screen capture task.

[0111] A first processing module 404 is configured to perform target processing on the image captured by calling the first screen capture task, thereby triggering the screen capture module 402, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device;

[0112] A first stopping module 405 is used to stop target processing of the image captured by calling the first screen capture task;

[0113] The reducing module 406 is configured to reduce the target screen capture frequency, and use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module 402 .

[0114] In an embodiment of the present application, after establishing a remote control connection with a first control device, a controlled device creates a first screen capture task based on GDI mode and a second screen capture task based on DirectX mode; calls the first screen capture task and the second screen capture task simultaneously to perform screen capture at a target screen capture frequency; determines whether a first image and a second image are identical, wherein the second image is a frame preceding the first image captured by calling the second screen capture task; if the first image and the second image are different, performs target processing on the image captured by calling the first screen capture task, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device; and if the first image and the second image are identical, reduces the target screen capture frequency. Thus, if the first image and the second image are identical, reducing the screen capture frequency can reduce the resources occupied by screen capture, thereby reducing the resources occupied by remote control. Furthermore, since the screen capture frequency is reduced, fewer images are captured, and thus fewer images undergo target processing, thereby reducing the resources occupied by remote control.

[0115] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0116] The second judgment module is used to judge whether the target screen capture frequency is greater than or equal to the first screen capture frequency threshold; if the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, the screen capture module 402 is triggered.

[0117] In an embodiment of the present application, screen capture is continued only when the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, which can avoid the situation where the screen capture frequency is too low, the screen capture interval is too long, and the screen changes during the screen capture interval without timely feedback to the control device, resulting in inaccurate control.

[0118] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0119] The first assignment module is configured to assign the target screen capture frequency to the first screen capture frequency threshold when the target screen capture frequency is less than the first screen capture frequency threshold, thereby triggering the screen capture module 402 .

[0120] In an embodiment of the present application, when the target screen capture frequency is less than the first screen capture frequency threshold, by assigning the target screen capture frequency to the first screen capture frequency threshold, it is possible to avoid the screen capture frequency being too low and the screen capture interval being too long. During the screen capture interval, the screen changes without timely feedback to the control device, resulting in inaccurate control.

[0121] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0122] The increasing module is configured to increase the target screen capture frequency, and use the screen capture frequency obtained by increasing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module 402 .

[0123] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0124] The third judgment module is used to judge whether the target screen capture frequency is less than or equal to the second screen capture frequency threshold; if the target screen capture frequency is less than or equal to the second screen capture frequency threshold, the screen capture module 402 is triggered.

[0125] In the embodiment of the present application, the screen is captured only when the target screen capture frequency is less than or equal to the second screen capture frequency threshold, which can avoid the occurrence of excessive screen capture frequency, frequent screen capture, and resource occupation.

[0126] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0127] The second assignment module is configured to assign the target screen capture frequency to the second screen capture frequency threshold when the target screen capture frequency is greater than the second screen capture frequency threshold, thereby triggering the screen capture module 402 .

[0128] In an embodiment of the present application, when the target screen capture frequency is greater than the second screen capture frequency threshold, by assigning the target screen capture frequency to the second screen capture frequency threshold, it is possible to avoid the screen capture frequency being too high, frequent screen capture, and resource occupation.

[0129] In the embodiment of the present application, it is possible to ensure that the target screen capture frequency does not exceed the value range of the first screen capture frequency threshold and the second screen capture frequency threshold.

[0130] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0131] The second processing module is configured to perform target processing on an image captured by calling the first screen capture task after the second control device establishes a remote control connection with the controlled device, when the first image and the second image are identical.

[0132] In some possible implementations of the embodiments of the present application, the remote control device 400 provided in the embodiments of the present application further includes:

[0133] The second stopping module is configured to stop determining whether the first image and the second image are identical after the second controlling device establishes a remote control connection with the controlled device.

[0134] In some possible implementations of the embodiment of the present application, the first determination module 403 is specifically configured to:

[0135] The desktop frame is used to determine whether the update area corresponding to the second image is empty. If the update area is empty, the second image is the same as the first image. If the update area is not empty, the second image is different from the first image. The desktop frame is obtained based on the callback of the second screen capture task.

[0136] FIG5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0137] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0138] Specifically, the processor 501 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0139] Memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be inside or outside the electronic device. In some specific embodiments, memory 502 is a non-volatile solid-state memory.

[0140] In some specific embodiments, the memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the remote control method according to the present application.

[0141] The processor 501 implements the remote control method provided in the embodiment of the present application by reading and executing the computer program instructions stored in the memory 502.

[0142] In one example, the electronic device may further include a communication interface 503 and a bus 510. As shown in FIG5, the processor 501, the memory 502, and the communication interface 503 are connected via the bus 510 and communicate with each other.

[0143] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0144] The bus 510 includes hardware, software, or both that couples components of the electronic device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 510 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0145] The electronic device can execute the remote control method provided in the embodiment of the present application, thereby achieving the corresponding technical effects of the remote control method provided in the embodiment of the present application.

[0146] In addition, in conjunction with the remote control method in the above embodiment, the present application also provides a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, the remote control method provided in the embodiment of the present application is implemented. Examples of computer-readable storage media include non-transitory computer-readable media, such as ROM, RAM, magnetic disks, or optical disks.

[0147] An embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device executes the remote control method provided in the embodiment of the present application and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0148] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0149] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (ERM), floppy disks, compact disc read-only memory (CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0150] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0151] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A remote control method, applied to a controlled device, comprising: After establishing a remote control connection with the first control device, creating a first screen capture task and a second screen capture task, wherein the first screen capture task is a screen capture task based on a graphics device interface GDI mode, and the second screen capture task is a screen capture task based on a direct extension DirectX mode; At a target screen capture frequency, calling the first screen capture task and the second screen capture task to simultaneously perform screen capture; Determine whether the first image is the same as the second image, wherein the second image is a previous frame image of the first image obtained by calling the second screen capture task to perform screen capture; In the case that the first image is different from the second image, target processing is performed on the image obtained by calling the first screen capture task to perform screen capture, and the step of calling the first screen capture task and the second screen capture task to perform screen capture at the target screen capture frequency is continued, wherein the target processing includes image conversion, image encoding, and sending the converted and encoded image to the control device; When the first image is the same as the second image, stop performing the target processing on the image obtained by calling the first screen capture task to perform screen capture, reduce the target screen capture frequency, use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency, and continue to execute the step of calling the first screen capture task and the second screen capture task to perform screen capture simultaneously with the target screen capture frequency.

2. The method according to claim 1, after using the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency, the method further comprises: Determining whether the target screen capture frequency is greater than or equal to a first screen capture frequency threshold; When the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, continue to execute the step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency.

3. The method according to claim 2, further comprising: When the target screen capture frequency is less than the first screen capture frequency threshold, the target screen capture frequency is assigned to the first screen capture frequency threshold, and the step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency is continued.

4. The method according to claim 1, after performing target processing on the image obtained by calling the first screen capture task to perform screen capture, the method further comprises: The target screen capture frequency is increased, and the screen capture frequency obtained by increasing the target screen capture frequency is used as the target screen capture frequency. The step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency is continued.

5. The method according to claim 4, after using the screen capture frequency obtained by increasing the target screen capture frequency as the target screen capture frequency, the method further comprises: Determining whether the target screen capture frequency is less than or equal to a second screen capture frequency threshold; When the target screen capture frequency is less than or equal to the second screen capture frequency threshold, continue to execute the step of calling the first screen capture task and the second screen capture task to simultaneously perform screen capture at the target screen capture frequency.

6. The method according to claim 5, further comprising: When the target screen capture frequency is greater than the second screen capture frequency threshold, the target screen capture frequency is assigned to the second screen capture frequency threshold, and the step of calling the first screen capture task and the second screen capture task to perform screen capture simultaneously at the target screen capture frequency is continued.

7. The method according to claim 1, further comprising: In the case that the first image is identical to the second image, after the second controlling device establishes a remote control connection with the controlled device, the target processing is performed on the image obtained by calling the first screen capturing task to perform screen capturing.

8. The method according to claim 7, before performing the target processing on the image obtained by calling the first screen capture task to perform screen capture, the method further comprises: Stop executing the step of determining whether the first image is identical to the second image.

9. The method according to claim 1, wherein: The determining whether the first image is identical to the second image comprises: The desktop frame is used to determine whether the update area corresponding to the second image is empty. If the update area is empty, the second image is the same as the first image. If the update area is not empty, the second image is different from the first image. The desktop frame is obtained based on the callback of the second screen capture task.

10. A remote control device, applied to a controlled device, comprising: A creation module, used for creating a first screen capture task and a second screen capture task after establishing a remote control connection with a first control device, wherein the first screen capture task is a screen capture task based on a graphics device interface GDI mode, and the second screen capture task is a screen capture task based on a direct extension DirectX mode; A screen capture module, used to call the first screen capture task and the second screen capture task to perform screen capture simultaneously at a target screen capture frequency; a first judging module, configured to judge whether the first image is identical to the second image, and trigger the first processing module if the first image is different from the second image, and trigger the first stopping module if the first image is identical to the second image, wherein the second image is a previous frame image of the first image obtained by calling the second screen capturing task to perform screen capturing; A first processing module, configured to perform target processing on the image obtained by calling the first screen capture task to perform screen capture, and trigger the screen capture module, wherein the target processing includes image conversion, image encoding, and sending the image converted and encoded image to a control device; A first stopping module, used for stopping the target processing of the image obtained by screen capturing by calling the first screen capture task; A reducing module is used to reduce the target screen capture frequency, and use the screen capture frequency obtained by reducing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module.

11. The device according to claim 10, further comprising: A second judging module is used to judge whether the target screen capturing frequency is greater than or equal to a first screen capturing frequency threshold; When the target screen capture frequency is greater than or equal to the first screen capture frequency threshold, the screen capture module is triggered.

12. The device according to claim 11, further comprising: The first assignment module is used to assign the target screen capture frequency to the first screen capture frequency threshold when the target screen capture frequency is less than the first screen capture frequency threshold, so as to trigger the screen capture module.

13. The device according to claim 10, further comprising: The increasing module is used to increase the target screen capture frequency, and use the screen capture frequency obtained by increasing the target screen capture frequency as the target screen capture frequency to trigger the screen capture module.

14. The apparatus according to claim 13, further comprising: A third judging module is used to judge whether the target screen capturing frequency is less than or equal to a second screen capturing frequency threshold; When the target screen capture frequency is less than or equal to the second screen capture frequency threshold, the screen capture module is triggered.

15. The device according to claim 14, further comprising: A second assignment module is configured to assign the target screen capture frequency to the second screen capture frequency threshold to trigger the screen capture module when the target screen capture frequency is greater than the second screen capture frequency threshold.

16. The apparatus according to claim 10, further comprising: The second processing module is used to perform the target processing on the image obtained by screen capturing by calling the first screen capture task after the second control device establishes a remote control connection with the controlled device when the first image is the same as the second image.

17. The apparatus according to claim 16, further comprising: The second stopping module is used to stop executing the step of determining whether the first image is identical to the second image.

18. The device according to claim 10, wherein: The first judgment module is specifically used for: The desktop frame is used to determine whether the update area corresponding to the second image is empty. If the update area is empty, the second image is the same as the first image. If the update area is not empty, the second image is different from the first image. The desktop frame is obtained based on the callback of the second screen capture task.

19. An electronic device, comprising: a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the remote control method according to any one of claims 1 to 9.

20. A computer-readable storage medium, wherein computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, the remote control method according to any one of claims 1 to 9 is implemented.

Images