Remote control method and device

Abstract

The embodiment of the present application provides a remote control method and device, which is applied to the controlled end, and specifically includes: when it is determined that the control end and the controlled end have established a remote connection, obtaining the first screen data of the controlled end, the first screen data of the controlled end includes the screen data obtained by encoding the second screen data according to a preset encoding method, and the preset encoding method includes a Huffman encoding method; sending the first screen data to the intermediate end; receiving remote control information sent by the intermediate end, the remote control information is determined by the intermediate end based on the user operation data sent by the control end, and the user operation data is obtained by the control end when the control end plays the second screen data; based on the remote control information, executing the control operation corresponding to the remote control information. The embodiment of the present application improves the effect of remote control.

Description

Technical Field

[0001] The present application belongs to the field of remote control technology, and in particular, relates to a remote control method and device. Background Art

[0002] In real life, electronic device manufacturers release a large number of new electronic devices such as mobile phones and tablets every year, and these new electronic devices will add some new functions and improvements, so there may be software adaptation problems. To address the above problems, software developers can use remote control technology to debug software, however, the remote control effect of existing remote control technology is poor. Summary of the invention

[0003] The embodiments of the present application provide a remote control method and device, which can improve the effect of remote control.

[0004] In a first aspect, an embodiment of the present application provides a remote control method, which is applied to a controlled end, and the method includes:

[0005] When it is determined that the control end and the controlled end have established a remote connection, first screen data of the controlled end is obtained, wherein the first screen data of the controlled end includes screen data obtained by encoding the second screen data according to a preset encoding method, wherein the preset encoding method includes a Huffman encoding method;

[0006] Sending the first screen data to the middle end;

[0007] Receiving remote control information sent by the intermediate end, the remote control information is determined by the intermediate end based on user operation data sent by the control end, and the user operation data is obtained by the control end when the control end plays the second screen data;

[0008] Based on the remote control information, a control operation corresponding to the remote control information is executed.

[0009] In a second aspect, a remote control method is provided, which is applied to a control terminal, and the method includes:

[0010] In the case of determining that the control end and the controlled end have established a remote connection, receiving first screen data forwarded by the intermediate end, the first screen data including screen data obtained by compressing the second screen data according to a preset coding method, the preset coding method including a Huffman coding method;

[0011] When the control terminal plays the second screen data, obtaining the user operation data;

[0012] The user operation data is sent to the intermediate end so that the intermediate end generates remote control information based on the user operation data, and the remote control information is sent to the controlled end so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

[0013] In a third aspect, a remote control method is provided, which is applied to an intermediate end, and the method includes:

[0014] When a remote connection is established between the control end and the controlled end, first screen data sent by the controlled end is received, the first screen data comprising screen data obtained by encoding the second screen data in a preset encoding method, the preset encoding method comprising a Huffman encoding method;

[0015] forwarding the first screen data to the control end;

[0016] receiving user operation data sent by the control terminal, where the user operation data is obtained by the control terminal when the control terminal plays the second screen data;

[0017] Generate remote control information based on user operation data;

[0018] The remote control information is sent to the controlled end, so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

[0019] In a fourth aspect, an embodiment of the present application provides a remote control device, which is applied to a controlled end, and the device includes:

[0020] an acquisition module, for acquiring first screen data of the controlled end when it is determined that the control end and the controlled end have established a remote connection, wherein the first screen data of the controlled end includes screen data obtained by encoding the second screen data in a preset encoding method, wherein the preset encoding method includes a Huffman encoding method;

[0021] A sending module, used for sending the first screen data to the middle end;

[0022] A receiving module, used for receiving remote control information sent by the intermediate end, the remote control information is determined by the intermediate end based on user operation data sent by the control end, and the user operation data is obtained by the control end when the control end plays the second screen data;

[0023] The execution module is used to execute the control operation corresponding to the remote control information based on the remote control information.

[0024] In a fifth aspect, a remote control device is provided, which is applied to a control terminal, and the device includes:

[0025] A receiving module, for receiving first screen data forwarded by the intermediate end when it is determined that the control end and the controlled end have established a remote connection, the first screen data comprising screen data obtained by compressing the second screen data according to a preset coding method, the preset coding method comprising a Huffman coding method;

[0026] An acquisition module, used for acquiring user operation data when the control end plays the second screen data;

[0027] The sending module is used to send user operation data to the intermediate end so that the intermediate end generates remote control information based on the user operation data, and to send the remote control information to the controlled end so that the controlled end performs the control operation corresponding to the remote control information based on the remote control information.

[0028] In a sixth aspect, a remote control device is provided, which is applied to an intermediate end, and the device includes:

[0029] A receiving module, used for receiving first screen data sent by the controlled end when the control end and the controlled end establish a remote connection, the first screen data including screen data obtained by compressing the second screen data according to a preset coding method, and the preset coding method includes a Huffman coding method;

[0030] A forwarding module, used for forwarding the first screen data to the control terminal;

[0031] The receiving module is further used to receive user operation data sent by the control terminal, where the user operation data is obtained by the control terminal when the control terminal plays the second screen data;

[0032] A generation module, used for generating remote control information based on user operation data;

[0033] The sending module is used to send remote control information to the controlled end, so that the controlled end can perform a control operation corresponding to the remote control information based on the remote control information.

[0034] In the seventh aspect, an electronic device is provided, comprising: a memory for storing computer program instructions; and a processor for reading and running the computer program instructions stored in the memory to execute the remote control method provided by any optional implementation of the first aspect, the second aspect, and the third aspect.

[0035] In an eighth aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the remote control method provided by any optional implementation of the first aspect, the second aspect and the third aspect is implemented.

[0036] In a ninth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the remote control method provided by any optional implementation of the first aspect, the second aspect and the third aspect is implemented.

[0037] In an embodiment of the present application, the controlled end can obtain the first screen data and send the first screen data to the intermediate end when it is determined that the control end and the controlled end have established a remote connection, and then can receive the remote control information sent by the intermediate end, the remote control information is determined by the intermediate end based on the user operation data sent by the control end, the user operation data is obtained by the control end when the control end plays the second screen data, and then the control end can perform the control operation corresponding to the remote control information based on the remote control information sent by the intermediate end. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data can include the screen data encoded and processed by the second screen data according to the Huffman coding method, so that the screen data can be efficiently and smoothly transmitted in the remote control, thereby improving the effect of the remote control. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is an architecture diagram between a controlled end, an intermediate end, and a control end provided in an embodiment of the present application;

[0040] Figure 2 This is another architecture diagram between a controlled terminal, an intermediate terminal, and a control terminal provided in an embodiment of the present application;

[0041] Figure 3 This is one of the flowcharts of a remote control method provided in an embodiment of the present application;

[0042] Figure 4 This is a second flow chart of a remote control method provided in an embodiment of the present application;

[0043] Figure 5 is a schematic diagram of a storage structure of video data provided by an embodiment of the present application;

[0044] Figure 6 is a schematic diagram of a storage structure of audio data provided by an embodiment of the present application;

[0045] Figure 7This is a flowchart of a remote control method provided in an embodiment of the present application;

[0046] Figure 8 This is a fourth flow chart of a remote control method provided in an embodiment of the present application;

[0047] Fig. 9 This is a schematic diagram of interaction between a controlled terminal, an intermediate terminal, and a control terminal provided in an embodiment of the present application;

[0048] Fig.10 This is one of the structural schematic diagrams of a remote control device provided in an embodiment of the present application;

[0049] Fig.11 This is a second structural diagram of a remote control device provided in an embodiment of the present application;

[0050] Fig.12 This is a third structural diagram of a remote control device provided in an embodiment of the present application;

[0051] Fig.13 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0054] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0055] As mentioned in the background technology, in real life, when faced with software adaptation issues caused by the release of new electronic devices, software developers may debug the software through remote control technology, that is, debug the software by remotely controlling another electronic device through one electronic device.

[0056] However, the existing remote control technology has the following disadvantages: first, only the control gesture information can be transmitted during the remote control process; second, it is unclear how to smoothly transmit the real-time image of the mobile phone; third, it is also unclear how to transmit and play the audio of the mobile phone to the control end. As a result, the remote control effect of the existing remote control technology is poor.

[0057] In order to solve the above problems, the embodiment of the present application provides a remote control method and device. By using the remote control method provided by the embodiment of the present application, when it is determined that the control end and the controlled end have established a remote connection, the first screen data of the controlled end can be obtained, and the first screen data can be sent to the intermediate end. The intermediate end can forward the first screen data to the control end. Since the first screen data includes screen data obtained by encoding the second screen data according to a preset encoding method, the control end can obtain user operation data when the control end plays the second screen data, and send the user operation data to the intermediate end. Then the intermediate end can generate remote control information based on the user operation data, and send the remote control information to the controlled end, so that the controlled end can perform the control operation corresponding to the remote control information based on the remote control information. In this way, since the above remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above first screen data can include screen data encoded by the second screen data according to the Huffman encoding method, so that the screen data is efficiently and smoothly transmitted in the remote control, thereby improving the effect of remote control.

[0058] Since the remote control method provided in the embodiment of the present application involves interaction between the controlled end, the intermediate end and the control end, in order to facilitate the subsequent understanding of the remote control method provided in the embodiment of the present application, before introducing the remote control method provided in the embodiment of the present application, the architecture between the controlled end, the intermediate end and the control end involved in the embodiment of the present application is first described in detail with reference to the accompanying drawings.

[0059] Figure 1 It is an architecture diagram between a controlled end, an intermediate end and a control end provided in an embodiment of the present application.

[0060] like Figure 1 As shown, the architecture diagram may include a controlled terminal 10, an intermediate terminal 20, and a control terminal 30. The controlled terminal 10 is connected to the intermediate terminal 20, and the intermediate terminal 20 is connected to the control terminal 30. It should be noted that the intermediate terminal 20 may be a related device that is independent of the above-mentioned controlled terminal 10 and the control terminal 30, or may be composed of some devices in the controlled terminal 10 and some devices in the control terminal 30, which is not specifically limited here.

[0061] Based on the architecture shown in Figure 1 above, the remote control method provided in the embodiment of the present application can obtain the first screen data of the controlled terminal 10 when it is determined that the control terminal 30 establishes a remote connection with the controlled terminal 10. The controlled terminal 10 can send the first screen data to the intermediate terminal 20, and the intermediate terminal 20 can forward the first screen data to the control terminal 30. Since the above-mentioned first screen data includes screen data obtained by encoding the second screen data according to a preset encoding method, and the preset encoding method includes a Huffman encoding method, the control terminal 30 can obtain user operation data and send the user operation data to the intermediate terminal 20 when the control terminal 30 plays the second screen data. The intermediate terminal 20 can generate remote control information based on the user operation data and send the remote control information to the controlled terminal 10, so that the controlled terminal 10 can perform a control operation corresponding to the remote control information based on the remote control information.

[0062] In order to facilitate the subsequent more accurate and detailed description of the remote control method provided by the embodiment of the present application, the embodiment of the present application also provides another architecture diagram between the controlled end, the intermediate end and the control end, such as Figure 2 As shown, the intermediate end 20 may include a forwarder 21 and an operation controller 22. It should be noted that the forwarder 21 and the operation controller 22 may be independent devices from each other. In addition, the forwarder 21 may also be a device in the control end 30, and the operation controller 22 may also be a device in the controlled end 10, which is not specifically limited here. It should also be noted that the forwarder 21 may be an independent program or a microservice deployed in the cloud.

[0063] In addition, continue as Figure 2As shown, the architecture diagram may also include a video processor 40 and an audio processor 50, and the above-mentioned controlled terminal 10 and the control terminal 30 are connected to the video processor 40 and the audio processor 50 respectively. Among them, the video processor 40 may include a video collector 41 and a video encoder 42, and the audio processor 50 may include an audio collector 51 and an audio encoder 52. It should also be noted that the above-mentioned video collector 41, video encoder 42, audio collector 51 and audio encoder 52 can be devices independent of the controlled terminal, or can be related devices existing in the controlled terminal, and no specific limitation is made here.

[0064] based on Figure 2 As shown in the architecture diagram, the remote control method provided in the embodiment of the present application can, when it is determined that the control terminal 30 has established a remote connection with the controlled terminal 10, the controlled terminal 10 can obtain the first screen data through the video processor and / or the audio processor, and send the first screen data to the repeater 21, and the repeater 21 can forward the first screen data to the control terminal 30. In this way, the control terminal 30 can obtain the user operation data while playing the second screen data, and send the user operation data to the repeater 21, the repeater 21 can forward the user operation data to the operation controller 22, the operation controller 22 can generate remote control information based on the user operation data, and send the remote control information to the controlled terminal 10, so that the controlled terminal 10 can perform the control operation corresponding to the remote control information based on the remote control information.

[0065] Based on the above Figure 1 The architecture shown below is combined with Figure 3 and Figure 4 The method for remote control applied to a controlled terminal provided in an embodiment of the present application is described in detail.

[0066] Figure 3 It is a flowchart of a remote control method provided in an embodiment of the present application.

[0067] like Figure 3 As shown, the execution subject of the method may be a controlled terminal, which may be an electronic device with a display screen such as a mobile phone or a tablet computer, and is not specifically limited here. Based on this, the method may specifically include the following steps:

[0068] S310, when it is determined that a remote connection is established between the control terminal and the controlled terminal, first screen data of the controlled terminal is obtained.

[0069] The first screen data of the controlled terminal may include screen data obtained by encoding the second screen data according to a preset encoding method. The preset encoding method may be a pre-set encoding method based on actual experience or circumstances. In some embodiments, the preset encoding method may be a Huffman encoding method, which is not specifically limited here.

[0070] In addition, the second screen data may include at least one of video data and audio data, and correspondingly, the first screen data may include at least one of encoded video data and encoded audio data. The video data refers to the relevant video data displayed on the display screen of the controlled terminal, and the audio data may refer to the relevant audio being played by the controlled terminal, which is not specifically limited here.

[0071] Specifically, when it is determined that the control end and the controlled end have established a remote connection, the controlled end can obtain the first screen data of the controlled end, that is, the controlled end can obtain screen data such as the video data displayed on its own display screen and / or the audio data being played.

[0072] S320, sending the first screen data to the intermediate end.

[0073] Specifically, after acquiring the first screen data, the controlled end can send the first screen data to the intermediate end, so that the intermediate end can forward the first screen data to the control end, so as to facilitate the subsequent remote control of the controlled end by the control end.

[0074] S330, receiving remote control information sent by the intermediate end.

[0075] In some embodiments, the remote control information may be determined by the intermediate end based on user operation data sent by the control end, wherein the user operation data may be obtained by the control end when the control end plays the second screen data. It should be noted that the user operation data may be data related to user operations on virtual objects such as display interfaces and virtual spaces, or data related to user operations on physical objects such as physical buttons.

[0076] Specifically, after the controlled end sends the first screen data to the intermediate end, the intermediate end can forward the first screen data to the control end, so that the control end can decode the first screen data to obtain the second screen data and play the second screen data. Then, when the control end plays the second screen data, the control end can obtain the user operation data and send the user operation data to the intermediate end, and the intermediate end can generate remote control information based on the user operation data and send it to the controlled end. In this way, the controlled end can receive the remote control information sent by the intermediate end.

[0077] S340: Based on the remote control information, execute a control operation corresponding to the remote control information.

[0078] Specifically, after receiving the remote control information sent by the intermediate end, the controlled end can perform the control operation corresponding to the remote control information based on the remote control information.

[0079] In an embodiment of the present application, the controlled end can obtain the first screen data and send the first screen data to the intermediate end when it is determined that the control end and the controlled end have established a remote connection, and then can receive the remote control information sent by the intermediate end, the remote control information is determined by the intermediate end based on the user operation data sent by the control end, the user operation data is obtained by the control end when the control end plays the second screen data, and then the control end can perform the control operation corresponding to the remote control information based on the remote control information sent by the intermediate end. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data can include the screen data encoded and processed by the second screen data according to the Huffman coding method, so that the screen data can be efficiently and smoothly transmitted in the remote control, thereby improving the effect of the remote control.

[0080] Since in the embodiment of the present application, the second screen data obtained by the controlled end needs to be encoded and transmitted, and in order to accurately describe the remote control method provided by the embodiment of the present application, it is necessary to explain here that the above-mentioned second screen data involved in the embodiment of the present application may include at least one of video data and audio data. Taking video data as an example, video data is generally composed of multiple image frames. In this way, for an image frame, if the resolution of an image frame is 800x600, it means that there are 480,000 pixels in an image frame. If a three-channel image identification (RGB three channels, R: red G: green B: blue) is used, a pixel is identified by three values, and 24 bits of binary are required for identification. In this way, 480,000 pixels in an image frame need to occupy: 480,000*24 / 8 / 1024=1406.25KB, and a 30fps video has a storage of up to 1406.25*30 / 1024=41.20MB per second. Therefore, in order to realize efficient transmission of screen data in remote control, the embodiment of the present application needs to encode the screen data to reduce the related resources occupied by the screen data, so that efficient transmission of the screen can be realized in remote control later.

[0081] Thus, in one embodiment, the step of obtaining the first screen data of the controlled terminal may specifically include the following steps:

[0082] Get the second screen data of the controlled terminal;

[0083] The second screen data of the controlled terminal is encoded according to a preset encoding method to obtain the first screen data.

[0084] Specifically, the controlled terminal can first obtain the second screen data, which can be understood as the initial screen data obtained by the controlled terminal, and then the second screen data of the controlled terminal can be encoded according to a preset encoding method to obtain the first screen data.

[0085] Based on this, since the second screen data may include at least one of video data and audio data, the step of acquiring the second screen data of the controlled terminal may include the following steps:

[0086] Collect video data of the controlled end through a video collector;

[0087] and / or,

[0088] The audio data of the controlled end is collected through the audio collector.

[0089] Specifically, since the second screen data may include at least one of video data and audio data, the controlled end may collect video data of the controlled end through a video collector, and / or the controlled end may collect audio data of the controlled end through an audio collector.

[0090] It should be noted that, in the process of the video collector collecting video data, if the above-mentioned video collector is a device that exists independently of the controlled end, such as a video capture card, then the controlled end can directly collect video data through the video collector. If the above-mentioned video collector is related software built into the controlled end, then the controlled end needs to use the relevant interface of the operating system to obtain video data. If the operating system is an Android system, the screen recording operation can be implemented through the relevant interface under the Surface package. If the operating system is other systems, it can be implemented using the relevant interface under ReplayKit. In addition, the controlled end can also directly obtain the video data in the video memory through the relevant interface of the driver layer of the graphics card device for copying. If the controlled end is implemented based on Linux or Unix system, the device information in / dev / graphics / can be read for recording to obtain video data.

[0091] It should also be noted that the above-mentioned audio collector can collect audio data in the following ways: first, directly connect the audio cable through the audio output port (headphone jack) of the controlled end to collect audio data; second, connect a Bluetooth device to play the audio of the controlled end through the Bluetooth device, and then read the information of the Bluetooth device for collection; third, develop and implant recording software in the controlled end to collect audio data. For example, the controlled end with an Android operating system can be implemented through the relevant interfaces under the MediaRecorder package, and the controlled end with an operating system of other systems can be implemented through the relevant interface functions of AudioToolbox, or it can be implemented by collecting the driver information of the sound card of the controlled end.

[0092] Accordingly, the step of encoding the second screen data of the controlled terminal according to the preset encoding method to obtain the first screen data may specifically include the following steps:

[0093] The video encoder encodes the video data of the controlled terminal according to a preset encoding method to obtain encoded video data;

[0094] and / or,

[0095] The audio data of the controlled terminal is encoded by the audio encoder according to a preset encoding method to obtain encoded video data.

[0096] In addition, it should be noted that, since the second screen data may be at least one of video data and audio data, correspondingly, the first screen data may include at least one of encoded video data and encoded audio data.

[0097] Specifically, after acquiring the second screen data, since the second screen data may include at least one of video data and audio data, the controlled end can encode the video data of the controlled end through a video encoder in accordance with a preset encoding method to obtain encoded video data, and / or the controlled end can encode the video data of the controlled end through an audio encoder in accordance with a preset encoding method to obtain encoded audio data.

[0098] In this embodiment, the first screen data can be obtained by acquiring the second screen data of the controlled terminal, that is, the initial screen data, and encoding the second screen data according to a preset encoding method. In this way, the transmission pressure of the screen data can be effectively reduced by encoding the screen data, thereby realizing efficient and smooth transmission of the screen data in remote control.

[0099] Based on this, when the second screen data includes video data, in order to accurately encode the video data, in some embodiments, the above-mentioned video data may include multiple image frames, each image frame may include multiple pixels and the pixel value of each pixel. It should be noted that the pixel value may be an RGB value, which is not specifically limited here.

[0100] So, like Figure 4 As shown, the step of encoding the second screen data of the controlled terminal according to the preset encoding method to obtain the first screen data may specifically include the following steps:

[0101] S410 , for each image frame in a plurality of image frames, determining a coding reference value for each pixel value in the image frame.

[0102] The above-mentioned coding reference value refers to the reference value for encoding each pixel point, which is not specifically limited here.

[0103] S420: construct a Huffman tree based on the coding reference value of each pixel value in the image frame.

[0104] S430, encoding the image frame based on the position of the pixel value of each pixel point in the image frame in the Huffman tree to obtain encoded data of the image frame.

[0105] The first screen data includes encoding data of each image frame in a plurality of image frames.

[0106] Specifically, in the case where the second screen data includes video data, since the video data may include multiple image frames, each image frame may include multiple pixels and the pixel value of each pixel, based on this, the controlled end can determine, for each image frame in the multiple image frames, the encoding reference value corresponding to each pixel value in the image frame, and then construct a Huffman tree based on the encoding reference value of each pixel value in the image frame, and can encode the image frame based on the position of the pixel value of each pixel point in the image frame in the Huffman tree to obtain the encoded data of the image frame, and then obtain the first screen data.

[0107] In this way, after the second coded data is encoded using the Huffman coding method to obtain the first screen data, the first screen data can be sent to the control end through the intermediate end, and the control end can restore and splice the first screen data to obtain the second screen data. In the above transmission process, the above first screen data can be as follows Figure 5 The structure shown is transmitted.

[0108] It should be noted that if Figure 5 As shown, Figure 5The timestamp in the can refer to the time of the current image frame. The time can be represented by a 64-bit binary number, which is accurate to seconds (up to 15:30:08, December 4, 292,277,026,596 Greenwich Mean Time). The sequence number refers to the number of pictures in the corresponding timestamp, which is represented by 16-bit binary, with a maximum value of 255, so 256 frames of pictures can be identified, which is equivalent to identifying a 256FPS video. The Huffman structure character is because the RGB value range is 0-255, and a 16-bit binary identifier can be used to correspond to 0-255 respectively. The frequency corresponding to the Huffman structure character refers to the number of bits stored, which needs to be judged according to the resolution of the image frame. For example, a picture with a resolution of 800x600 has a total of 480,000 pixels. In the extreme case, all RGB values ​​are the same, such as (255, 255, 255), so the frequency of 255 is 480,000*3=1,440,000. Therefore, using 24-bit binary (decimal maximum value: 16777215) is sufficient. A total of 256 frequencies corresponding to each character are required. It is calculated that a total of 256*24 / 8=168 bytes of storage are required. The encoded pixel refers to the encoding value of the corresponding pixel according to the Huffman structure tree. For example, if the RGB of the pixel is (31, 24, 129), according to the Huffman structure tree, the encoding value of 31 is 01, the encoding value of 24 is 101, and the encoding value of 129 is 11, then the encoding value of the pixel is: 0110111.

[0109] In this embodiment, the video data in the second screen data can be encoded using the Huffman encoding method, so that the video data can be effectively encoded, which facilitates the subsequent efficient transmission of the video data.

[0110] In order to accurately and in detail describe the remote control method provided in the embodiment of the present application, in one embodiment, the above-mentioned S410 may specifically include the following steps:

[0111] For each image frame in the plurality of image frames, based on pixel values ​​of a plurality of pixel points in the image frame, determining an occurrence frequency of each pixel value in the image frame;

[0112] or,

[0113] For each image frame in the plurality of image frames, based on pixel values ​​of a plurality of pixel points in the image frame, determining an occurrence frequency of each pixel value in the image frame;

[0114] Based on a preset first corresponding relationship between the occurrence frequency and the weight, the occurrence frequency of each pixel value in the image frame is matched to determine the weight of each pixel value.

[0115] The above-mentioned preset correspondence between the occurrence frequency and the weight may be a preset correspondence based on actual experience or circumstances, and is not specifically limited here.

[0116] In addition, the above-mentioned encoding reference value includes the occurrence frequency of each pixel value or the weight of each pixel value.

[0117] Specifically, when the above-mentioned encoding reference value can include the occurrence frequency of each pixel value or the weight of each pixel value, the controlled end can determine the occurrence frequency of each pixel value for each image frame in multiple image frames based on the pixel values ​​of multiple pixels in the image frame.

[0118] Alternatively, the controlled end can also determine, for each image frame in multiple image frames, the frequency of occurrence of each pixel value based on the pixel values ​​of multiple pixel points in the image frame, and then match the frequency of occurrence of each pixel value in the image frame based on the correspondence between a preset frequency of occurrence and a weight, to determine the weight of each pixel value.

[0119] In one example, if the above-mentioned coding reference value includes the frequency of occurrence of each pixel value, for each image frame, the controlled end can determine the frequency of occurrence of each pixel value in the image frame based on the image frame, and construct a corresponding Huffman tree, so that the node with a higher frequency of occurrence has a shorter path, thereby obtaining a complete Huffman tree.

[0120] In another example, if the above-mentioned coding reference value includes the weight of each pixel value, for each image frame, the controlled end can determine the frequency of occurrence of each pixel value in the image frame based on the image frame, and then obtain the weight of each pixel value, and construct the corresponding Huffman tree based on the weight of each pixel value, so that the node with a larger weight has a shorter path, thereby obtaining a complete Huffman tree.

[0121] In this embodiment, since the above-mentioned coding reference value may include the occurrence frequency of each pixel value or the weight of each pixel value, the controlled end can determine the occurrence frequency of each pixel value for each image frame based on the pixel values ​​of each pixel point in the image frame, or, on this basis, determine the weight of each pixel value based on the correspondence between the preset occurrence frequency and the weight. In this way, the coding reference value of each pixel value can be accurately determined, so that effective coding can be performed based on the coding reference value of each pixel value later.

[0122] It should also be noted that after the video encoder receives the first frame of video information transmitted by the video collector, it first saves a copy, then encodes it, and sends it to the forwarder; when receiving an image other than the first frame, it first compares it with the saved copy. If the image content is consistent, it is directly discarded. If the content is inconsistent, the saved copy is replaced, and then it is encoded and sent to the forwarder.

[0123] In another embodiment, when the second screen data includes audio data, in order to encode the audio data, in some embodiments, the audio data may include multiple sub-audio data and audio information corresponding to each sub-audio data, and the audio information may include sampling rate, channel and other information, which is not specifically limited here. Based on this, the step of encoding the second screen data of the controlled terminal according to the preset encoding method to obtain the first screen data may specifically include the following steps:

[0124] For each sub-audio data in the plurality of sub-audio data, determining a coding reference value for each audio information in the sub-audio data;

[0125] The encoding reference value here refers to the reference value for encoding each sub-audio data, and is not specifically limited here.

[0126] Constructing a Huffman tree based on the encoding reference value of each audio information in the sub-audio data;

[0127] Based on the position of the pixel value of each audio information in the sub-audio data in the Huffman tree, the sub-audio data is encoded to obtain the encoded data of the sub-audio data.

[0128] The first screen data includes the encoded data of each sub-audio data in the plurality of sub-audio data. It should also be noted that the encoded data of each sub-audio data may be audio data in PCM format, which is not specifically limited here.

[0129] Specifically, when the second screen data includes audio data, the controlled end can determine the encoding reference value of each audio information in the sub-audio data for each sub-audio data among multiple sub-audio data in the audio data, and construct a Huffman tree based on the encoding reference value of each audio information in the sub-audio data, and then encode the sub-audio data based on the position of the pixel value of each audio information in the sub-audio data in the Huffman tree to obtain the encoded data of the sub-audio data.

[0130] In one example, analog audio signals are converted into digital audio signals by sampling the audio. The audio point corresponding to each timestamp is obtained according to the sampling frequency, and the sampled data is stored and identified using 16-bit binary (16bit=2 bytes) and recorded using dual channels (i.e., stereo).

[0131] Because the sampling frequency affects the accuracy of audio data, according to the Nyquist-Shannon sampling theorem, the sampling frequency must be twice the highest frequency in the analog signal to accurately restore the digital model to the analog signal. When the sampling frequency is set to 44100Hz, it is equivalent to the quality level of CD, which is more than twice the human hearing limit of 20000Hz, and can meet the requirements of high restoration.

[0132] According to the sampling frequency of 44100 Hz per second, the time interval of each sampling point is: 1 / 44100 is about 22.68 microseconds. There are 44100 sampling points in one second, so 16-bit binary can be used to represent the sampling point of each second to which the current sampling point belongs. The specific storage structure is as follows Figure 6 shown.

[0133] like Figure 6 As shown, Figure 6 The information in the can be the sampling rate, bit depth, number of channels and other key information of the audio transmission, and the corresponding binary number needs to be defined in advance. The timestamp can be the time corresponding to the current audio, which can be accurate to seconds, one cycle per second. The system time is represented by a 64-bit binary number, accurate to seconds (up to 15:30:08, December 4, 292,277,026,596 Greenwich time). Sampling point: uses 16-bit binary storage, the maximum value of the sampling point is 65535, representing the sampling point of each second, such as the 300th sampling point, the recorded data is: 0x012c; Channel 0 (left channel): uses 16-bit binary storage, representing the sample value of channel 0 at the current sampling point, such as the sample value 2000, it is recorded as: 0x07d0. Channel 1 (right channel): uses 16-bit binary storage, representing the sample value of channel 1 at the current sampling point, such as the sample value 3000, it is recorded as: 0x0bb8.

[0134] It should also be noted that if the compression rate requirement is high and a lower audio restoration degree is acceptable, the sampling rate and bit depth can be lowered for compressed audio transmission, which mainly depends on the transmission quality requirements.

[0135] In this embodiment, the audio data in the second screen data can be encoded using the Huffman coding method, so that the audio data can be effectively encoded, which facilitates the subsequent efficient transmission of the audio data.

[0136] In order to accurately and in detail describe the remote control method provided by the embodiment of the present application, in one embodiment, the above-mentioned step of determining the encoding reference value of each audio information in the sub-audio data may specifically include the following steps:

[0137] Determine the frequency of occurrence of each audio message;

[0138] or,

[0139] Determine the frequency of occurrence of each audio message;

[0140] Based on the preset second corresponding relationship between the occurrence frequency and the weight, the occurrence frequency of each audio information is matched to determine the weight of each audio information.

[0141] The coding reference value of each audio information includes the occurrence frequency of each audio information or the weight of each audio information.

[0142] In this embodiment, the controlled end can determine the frequency of occurrence of each audio information, or first determine the frequency of occurrence of each audio information, and then match the frequency of occurrence of each audio information according to the second corresponding relationship between the preset frequency of occurrence and the weight to determine the weight of each audio information. In this way, the encoding reference value of each audio information can be accurately determined, so that effective encoding can be performed based on the encoding reference value of each audio information later.

[0143] Combine the following Figure 7 The remote control method applied to the control terminal provided in the embodiment of the present application is described in detail.

[0144] like Figure 7 As shown, the execution subject of the method may be a control terminal, and the control terminal may be any interface with a display area, such as a personal computer (PC) terminal with a display area, a mobile phone or a pad-type product, etc. Based on this, the vehicle control method provided in the embodiment of the present application may specifically include the following steps:

[0145] S710, when it is determined that the control end and the controlled end have established a remote connection, receiving first screen data forwarded by the intermediate end.

[0146] In some embodiments, the first screen data includes screen data obtained by compressing the second screen data according to a preset encoding method. The preset encoding method can be preset based on actual experience or circumstances. In some embodiments, the preset encoding method includes a Huffman encoding method.

[0147] In addition, the second screen data mentioned above can be understood as the initial screen data collected by the controlled terminal, and the second screen data can include at least one of video data and audio data.

[0148] Specifically, when it is determined that the control end and the controlled end have established a remote connection, the controlled end can obtain the first screen data and send the first screen data to the intermediate end, so that the control end can receive the first screen data forwarded by the intermediate end.

[0149] S720, obtaining user operation data when the control terminal plays the second screen data.

[0150] The user operation data may be data of related operations performed by the user on the control terminal, which is not specifically limited here.

[0151] Specifically, after the control end receives the first screen data, since the first screen data is related data obtained by encoding the second screen data, the control end can decode the first screen data to obtain the second screen data and play the second screen data. In this way, when the control end plays the second screen data, if the user performs corresponding operations on the control end, the control end can obtain the corresponding user operation data.

[0152] S730, sending the user operation data to the intermediate end so that the intermediate end generates remote control information based on the user operation data, and sending the remote control information to the controlled end so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

[0153] Specifically, after acquiring the user operation data, the control end can send the user operation data to the intermediate end, so that the intermediate end generates remote control information based on the user operation data, and send the remote control information to the controlled end, so that the controlled end performs the control operation corresponding to the remote control information based on the remote control information. In this way, the control end remotely controls the controlled end.

[0154] In an embodiment of the present application, the control end can receive the first screen data forwarded by the intermediate end when determining that the control end and the controlled end have established a remote connection. The first screen data may include screen data obtained by compressing the second screen data according to a preset encoding method, and the preset encoding method may include a Huffman encoding method. The control end can obtain user operation data when the control end plays the second screen data, and send the user operation data to the intermediate end, so that the intermediate end can generate remote control information based on the user operation data, and send the remote control information to the controlled end, so that the controlled end can perform a control operation corresponding to the remote control information based on the remote control information. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data may include screen data encoded by the second screen data according to the Huffman encoding method, thereby realizing efficient and smooth transmission of screen data in remote control, thereby improving the effect of remote control.

[0155] Based on this, in one embodiment, the above step of obtaining user operation data may specifically include the following steps:

[0156] Receive user input for the target object;

[0157] In response to the input, user operation data is acquired.

[0158] The control end may include a target object.

[0159] In addition, the above input may be a related input performed by the user on the target object based on the second screen data, and the input may be a click input, a touch input, etc., which is not specifically limited here.

[0160] Specifically, the control end can receive a user's input to a target object, and acquire user operation data in response to the user's input to the target object.

[0161] In this embodiment, the corresponding user operation data can be obtained by receiving and responding to the user's input to the target object in the control end. In this way, the user operation data can be obtained in real time and accurately, so that the subsequent intermediate end can generate accurate remote control information based on the user operation data, thereby realizing effective and accurate remote control of the controlled end.

[0162] It should be noted that the above-mentioned target object may include at least one of a virtual object and a physical object, wherein the virtual object may be a related interface, control, and other objects displayed on the display screen of the control terminal, and the physical object may be a physical button existing in the control terminal, such as a volume button, a power button, etc., which is not specifically limited here.

[0163] Based on this, in one embodiment, when the target object includes the virtual object, the step of receiving the user's input on the target object may specifically include the following steps:

[0164] A first sub-input of a user on a virtual object is received.

[0165] The first sub-input may be a click input, a touch input, etc. for a virtual object, which is not specifically limited here.

[0166] Based on this, the above step of obtaining user operation data in response to input may specifically include the following steps:

[0167] In response to the first sub-input, user operation data is acquired.

[0168] In some embodiments, the user operation data may include an input type of the first sub-input and an input position of the first sub-input in a display area of ​​the control terminal.

[0169] Specifically, when the target object includes a virtual object, the control end is able to receive a first sub-input from a user to the virtual object, and in response to the first sub-input, obtain an input type of the first sub-input and an input position of the first sub-input in the display area of ​​the control end, so that the control end can obtain user operation data.

[0170] It should be noted that, considering that the existing remote control technology will record single-click, double-click and other operations, there are actually no so-called single-click and double-click operations at the system level, only press and lift operations. Therefore, in the embodiment of the present application, the input type of the first sub-input can include a press type, a lift type and a move type, and the existing single-click, double-click and other operations are discarded.

[0171] In addition, the input position of the first sub-input in the display area of ​​the control terminal can be not only the coordinate position of the first sub-input in the display area, but also the relative position of the first sub-input in the display area. The relative position has a value range of [0, 1] of floating-point type data, and the value is determined by the ratio of the input coordinate position to the length or width of the display screen.

[0172] In this embodiment, the control end can receive the first sub-input of the user to the virtual object when the target object includes a virtual object, and obtain the user operation data in response to the first sub-input. In this way, the user operation data can be accurately obtained, so that the subsequent intermediate end can generate accurate remote control information based on the user operation data, thereby realizing effective and accurate remote control of the controlled end.

[0173] In another embodiment, when the target object includes the physical object, the step of receiving the user's input on the target object may specifically include the following steps:

[0174] A second sub-input of the user to the entity object is received.

[0175] The second sub-input may indicate an object identifier of a physical object. In addition, the second sub-input may be a click input, a long press input, etc., which are not specifically limited here.

[0176] Based on this, the above step of obtaining user operation data in response to input may specifically include the following steps:

[0177] In response to the second sub-input, the object identifier of the physical object is matched based on a preset correspondence relationship between the object identifier and the operation instruction, and the operation instruction corresponding to the physical object is determined.

[0178] The correspondence between the preset object identifier and the operation instruction may be preset based on actual experience or circumstances, and is not specifically limited here.

[0179] In addition, the above user operation data may include operation instructions corresponding to the entity object.

[0180] Specifically, when the target object includes a physical object, the control terminal can receive the second sub-input of the user to the physical object, and in response to the second sub-input, match the object identifier of the physical object based on the correspondence between the preset object identifier and the operation instruction, and determine the operation instruction corresponding to the physical object. In this way, the control terminal can obtain the user operation data.

[0181] In this embodiment, when the target object is a physical object, the user operation data can be received and responded to by the user's second sub-input on the physical object. In this way, the user operation data can be accurately obtained, so that the subsequent intermediate end can generate accurate remote control information based on the user operation data, thereby realizing effective and accurate remote control of the controlled end.

[0182] Based on the above content, it should be noted that the above-mentioned first sub-input may include at least one sub-input, and accordingly, the user operation data obtained in response to the first sub-input may include the input type corresponding to each sub-input and the input position of each sub-input in the display area of ​​the control end.

[0183] For example, the first sub-input may be [input 1, input 2]. In this way, it is possible to use the aggregate data type to store multiple user operation data. In addition, if the control terminal is implemented in a Web form, the original data of all multi-point operations can be obtained through the touchs attribute in the event data of the user operation.

[0184] Similarly, the second sub-input may also include at least one sub-input, and the user operation data obtained in response to the second sub-input may include an operation instruction corresponding to each sub-input in the second sub-input.

[0185] In addition, in order to accurately and comprehensively describe the remote control method provided in the embodiment of the present application, in one embodiment, when the second screen data includes audio data and video data, before the user operation data is obtained, the remote control method involved may specifically include the following steps:

[0186] Decoding the first screen data to obtain second screen data;

[0187] Synchronously processing the audio data and the video data to obtain synchronized audio data and video data;

[0188] Play the synchronized audio and video data.

[0189] Specifically, after receiving the first screen data, the control end can decode the first screen data to obtain the second screen data, which includes audio data and video data. In this way, the audio data and video data included in the second screen data can be synchronously processed to obtain the synchronized audio data and video data, and then the synchronized audio data and video data can be played.

[0190] It should be noted that the transmission of video streams is ultimately converted into the form of transmission data packets. Network transmission uses the same transmission protocol RTP (Real-time Transport Protocol) as audio to transmit video data. It is divided into multiple data packets for transmission in the same way as audio processing. The control end receives each complete data packet information, and obtains the video position of the current image frame based on the timestamp and sequence number of the data packet. The complete Huffman tree is obtained from the data stored by the Huffman structure characters and frequencies. The Huffman coding of each pixel is used to restore it to a three-channel pixel, thereby restoring the image frame and finally obtaining the initial video data.

[0191] In this embodiment, the control end can decode the first screen data to obtain the second screen data. Since the second screen data includes video data and audio data, the audio data and video data in the second screen data can be synchronously processed to obtain the synchronized audio data and video data, and then the synchronized audio data and video data can be played, avoiding the situation where the audio and video are out of sync.

[0192] After the control end receives the first screen data, the audio data therein is relatively small and has a fast transmission speed, so it arrives at the control end first. The control end decodes and decompresses the audio data and then plays it. Therefore, in order to accurately synchronize the audio data and video data in the screen data and avoid the situation where the audio and video are out of sync, in one embodiment, the audio data involved above includes N sub-audio data and the timestamp of each sub-audio data.

[0193] Based on this, the above-mentioned step of synchronously processing the audio data and the video data to obtain the audio data and the video data after the synchronization processing may be specifically performed as follows:

[0194] For the i-th sub-audio data in the N sub-audio data, based on the timestamp of the i-th sub-audio data, determine the sub-video data corresponding to the timestamp in the video data, where N and i are both positive integers;

[0195] Merge the i-th sub-audio data with the sub-video data corresponding to the i-th sub-audio data.

[0196] The sub-video data may include at least one image frame.

[0197] Specifically, since the audio data includes N sub-audio data and the timestamp of each sub-audio data, for the i-th sub-audio data in the N sub-audio data, based on the timestamp of the i-th sub-audio data, the sub-video data corresponding to the timestamp in the video data is determined, and then the i-th sub-audio data and the sub-video data corresponding to the i-th sub-audio data can be merged. In this way, the audio data and video data after synchronization processing can be obtained.

[0198] In this embodiment, since the audio data includes N sub-audio data and the timestamp of each sub-audio data, for each sub-audio data, the sub-video data corresponding to the timestamp of the sub-audio data can be screened from the video data, and then the sub-audio data and the sub-video data corresponding to the sub-audio data are merged. In this way, the audio data and video data after synchronization processing can be accurately obtained, and then the audio data and video data after synchronization processing can be played, avoiding the situation where the audio and video are out of sync.

[0199] The following is combined with Figure 8 The method for remote control applied to the intermediate end provided in an embodiment of the present application is described in detail.

[0200] Figure 8 It is a flowchart of a remote control method provided in an embodiment of the present application.

[0201] like Figure 8 As shown, the execution subject of the method may be an intermediate end. Based on this, the remote control method provided in the embodiment of the present application may specifically include the following steps:

[0202] S810, when a remote connection is established between the control terminal and the controlled terminal, receiving first screen data sent by the controlled terminal.

[0203] The first screen data includes screen data obtained by encoding the second screen data according to a preset encoding method, and the preset encoding method includes a Huffman encoding method;

[0204] S820, forwarding the first screen data to the control end.

[0205] S830, receiving user operation data sent by the control terminal.

[0206] The user operation data is obtained by the control terminal when the control terminal plays the second screen data.

[0207] S840, generating remote control information based on the user operation data.

[0208] S850, sending remote control information to the controlled end, so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

[0209] The above remote control information can be related information that can be controlled by software or hardware methods. The software method can call the automatic test interface of the electronic device to perform the control operation of the electronic device; or call the mouse or keyboard event interface of the operating system driver layer to send mouse or keyboard events to operate the electronic device; the hardware method can use a mechanical arm to control the mobile phone, execute the click or key operation of the mobile phone, and complete the control of the mobile phone.

[0210] It should be noted that the specific processes involved in the above S810 to S850 can be found in the above examples and will not be elaborated herein.

[0211] In an embodiment of the present application, the intermediate end can receive the first screen data sent by the controlled end when the control end and the controlled end establish a remote connection, the first screen data includes screen data obtained by encoding the second screen data according to a preset encoding method, the preset encoding method includes a Huffman encoding method, and forward the first screen data to the control end, and then can receive the user operation data sent by the control end, the user operation data is obtained when the control end plays the second screen data, and then can generate remote control information based on the user operation data, and send the remote control information to the controlled end, so that the controlled end can perform the control operation corresponding to the remote control information based on the remote control information. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data can include screen data encoded and processed by the second screen data according to the Huffman encoding method, so that the screen data can be efficiently and smoothly transmitted in the remote control, thereby improving the effect of the remote control.

[0212] In order to accurately and completely describe the remote control method applied to the controlled end, the remote control method applied to the control end, and the remote control method applied to the intermediate end provided in the embodiments of the present application, in one embodiment, in combination with the attached Fig. 9 The interaction process between the controlled end, the control end and the intermediate end involved in the embodiment of the present application is described in detail.

[0213] like Fig. 9 As shown, the interaction process between the controlled end, the control end and the intermediate end in the embodiment of the present application may include the following steps:

[0214] S901, when a remote connection is established between a control terminal and a controlled terminal, the controlled terminal obtains first screen data of the controlled terminal.

[0215] S902, the controlled end sends the first screen data to the intermediate end.

[0216] S903, the intermediate end forwards the first screen data to the control end.

[0217] S904, the control end obtains user operation data when the control end plays the second screen data.

[0218] S905, the control end sends the user operation data to the middle end.

[0219] S906: The middle end generates remote control information based on the user operation data.

[0220] S907, the intermediate end sends remote control information to the controlled end.

[0221] S908, the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

[0222] It should be noted that the specific processes involved in the above S901 to S908 can be found in the above embodiments, and will not be elaborated herein.

[0223] In an embodiment of the present application, when a remote connection is established between a control terminal and a controlled terminal, the controlled terminal can obtain the first screen data of the controlled terminal and send the first screen data to the intermediate terminal. In this way, the intermediate terminal can forward the first screen data to the control terminal. Since the first screen data includes screen data obtained by encoding the second screen data according to a preset encoding method, the control terminal can obtain user operation data when the control terminal plays the second screen data, and send the user operation data to the intermediate terminal. In this way, the intermediate terminal can generate remote control information based on the user operation data, and send the remote control information to the controlled terminal, so that the controlled terminal can perform the control operation corresponding to the remote control information based on the remote control information. Since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data can include screen data obtained by encoding the second screen data according to the Huffman encoding method, which realizes the efficient and smooth transmission of screen data in the remote control, thereby improving the effect of remote control.

[0224] Based on the same inventive concept, the embodiment of the present application provides a remote control device, which can be applied to the controlled end. Fig.10 The remote control device provided in the embodiment of the present application is described in detail.

[0225] like Fig.10 As shown, the remote control device 1000 may include:

[0226] The acquisition module 1010 is used to acquire first screen data of the controlled end when it is determined that the control end and the controlled end have established a remote connection, wherein the first screen data of the controlled end includes screen data obtained by encoding the second screen data according to a preset encoding method, wherein the preset encoding method includes a Huffman encoding method;

[0227] A sending module 1020, configured to send the first screen data to the intermediate end;

[0228] The receiving module 1030 is used to receive remote control information sent by the intermediate terminal, where the remote control information is determined by the intermediate terminal based on user operation data sent by the control terminal, where the user operation data is obtained by the control terminal when the control terminal plays the second screen data;

[0229] The execution module 1040 is used to execute the control operation corresponding to the remote control information based on the remote control information.

[0230] In one embodiment, the remote control device provided in the embodiment of the present application further includes:

[0231] The acquisition module is also used to acquire the second screen data of the controlled terminal;

[0232] The encoding module is used to encode the second screen data of the controlled terminal according to a preset encoding method to obtain the first screen data.

[0233] In some embodiments, the second screen data includes video data; the video data includes multiple image frames, each image frame includes multiple pixels and a pixel value of each pixel; based on this, the remote control device provided in the embodiment of the present application also includes:

[0234] A determination module, configured to determine, for each image frame in a plurality of image frames, a coding reference value for each pixel value in the image frame;

[0235] A construction module, for constructing a Huffman tree based on a coding reference value of each pixel value in the image frame;

[0236] The encoding module is specifically used to encode the image frame based on the position of the pixel value of each pixel point in the image frame in the Huffman tree to obtain the encoded data of the image frame. The first screen data includes the encoded data of each image frame in the multiple image frames.

[0237] In one embodiment, the remote control device provided in the embodiment of the present application further includes:

[0238] The above-mentioned determination module is specifically used to determine the occurrence frequency of each pixel value in the image frame based on the pixel values ​​of multiple pixels in the image frame for each image frame in the multiple image frames;

[0239] or,

[0240] The above-mentioned determination module is specifically used to determine the occurrence frequency of each pixel value in the image frame based on the pixel values ​​of multiple pixels in the image frame for each image frame in the multiple image frames;

[0241] A matching module, configured to match the occurrence frequency of each pixel value in the image frame based on a preset first correspondence between the occurrence frequency and the weight, and determine the weight of each pixel value;

[0242] The coding reference value includes the occurrence frequency of each pixel value or the weight of each pixel value.

[0243] In an embodiment of the present application, the controlled end can obtain the first screen data and send the first screen data to the intermediate end when it is determined that the control end and the controlled end have established a remote connection, and then can receive the remote control information sent by the intermediate end, the remote control information is determined by the intermediate end based on the user operation data sent by the control end, the user operation data is obtained by the control end when the control end plays the second screen data, and then the control end can perform the control operation corresponding to the remote control information based on the remote control information sent by the intermediate end. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data can include the screen data encoded and processed by the second screen data according to the Huffman coding method, so that the screen data can be efficiently and smoothly transmitted in the remote control, thereby improving the effect of the remote control.

[0244] Each module in the remote control device provided in the embodiment of the present application can be implemented as follows: Figure 3 or Figure 4 The method steps of the illustrated embodiment can achieve the corresponding technical effects, and for the sake of brevity, they will not be repeated here.

[0245] Based on the same inventive concept, the embodiment of the present application further provides a remote control device, which can be applied to a control terminal. Fig.11 The remote control device provided in the embodiment of the present application is described in detail.

[0246] Fig.11 It is a structural schematic diagram of a remote control device provided in an embodiment of the present application.

[0247] like Fig.11 As shown, the remote control device 1100 may include:

[0248] The receiving module 1110 is used to receive the first screen data forwarded by the intermediate end when it is determined that the control end and the controlled end have established a remote connection, the first screen data including screen data obtained by compressing the second screen data according to a preset coding method, and the preset coding method includes a Huffman coding method;

[0249] The acquisition module 1120 is used to acquire user operation data when the control terminal plays the second screen data;

[0250] The sending module 1130 is used to send user operation data to the intermediate end so that the intermediate end generates remote control information based on the user operation data, and to send the remote control information to the controlled end so that the controlled end performs the control operation corresponding to the remote control information based on the remote control information.

[0251] In one embodiment, the receiving module is further used to receive a user input on a target object, the control end includes the target object, and the input is an input made by the user on the target object based on the second screen data;

[0252] The acquisition module is specifically used to acquire user operation data in response to input.

[0253] In one embodiment, the target object includes a virtual object; based on this,

[0254] The receiving module is specifically used to receive a first sub-input of a user to a virtual object;

[0255] The acquisition module is specifically used to acquire user operation data in response to the first sub-input, and the user operation data includes the input type of the first sub-input and the input position of the first sub-input in the display area of ​​the control terminal.

[0256] In one embodiment, the target object includes a physical object; based on this, the remote control device provided in the embodiment of the present application also includes:

[0257] The receiving module is specifically configured to receive a second sub-input of the user on the entity object, where the second sub-input indicates an object identifier of the entity object;

[0258] The matching module is used to match the object identifier of the physical object in response to the second sub-input based on the correspondence between the preset object identifier and the operation instruction, and determine the operation instruction corresponding to the physical object, and the user operation data includes the operation instruction corresponding to the physical object.

[0259] In one embodiment, the second screen data includes audio data and video data; based on this, the remote control device provided in the embodiment of the present application further includes:

[0260] A decoding module, used for decoding the first screen data to obtain second screen data;

[0261] A synchronization processing module, used for performing synchronization processing on the audio data and the video data to obtain the audio data and the video data after synchronization processing;

[0262] The playback module is used to play the audio data and video data after synchronization processing.

[0263] In one embodiment, the audio data includes N sub-audio data and a timestamp of each sub-audio data; based on this, the synchronization processing module is specifically used to:

[0264] For the i-th sub-audio data in the N sub-audio data, based on the timestamp of the i-th sub-audio data, determine the sub-video data corresponding to the timestamp in the video data, where N and i are both positive integers;

[0265] Merge the i-th sub-audio data with the sub-video data corresponding to the i-th sub-audio data.

[0266] In an embodiment of the present application, the control end can receive the first screen data forwarded by the intermediate end when determining that the control end and the controlled end have established a remote connection. The first screen data may include screen data obtained by compressing the second screen data according to a preset encoding method, and the preset encoding method may include a Huffman encoding method. The control end can obtain user operation data when the control end plays the second screen data, and send the user operation data to the intermediate end, so that the intermediate end can generate remote control information based on the user operation data, and send the remote control information to the controlled end, so that the controlled end can perform a control operation corresponding to the remote control information based on the remote control information. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data may include screen data encoded by the second screen data according to the Huffman encoding method, thereby realizing efficient and smooth transmission of screen data in remote control, thereby improving the effect of remote control.

[0267] Each module in the remote control device provided in the embodiment of the present application can realize Figure 7 The method steps of the illustrated embodiment can achieve the corresponding technical effects, and for the sake of brevity, they will not be repeated here.

[0268] Based on the same inventive concept, the embodiment of the present application further provides a remote control device, which can be applied to the intermediate end. Fig.12 The remote control device provided in the embodiment of the present application is described in detail.

[0269] Fig.12 It is a structural schematic diagram of a remote control device provided in an embodiment of the present application.

[0270] like Fig.12 As shown, the remote control device 1200 may specifically include:

[0271] The receiving module 1210 is used to receive first screen data sent by the controlled end when the control end and the controlled end establish a remote connection, wherein the first screen data includes screen data obtained by compressing the second screen data according to a preset coding method, and the preset coding method includes a Huffman coding method;

[0272] The forwarding module 1220 is used to forward the first screen data to the control terminal;

[0273] The receiving module 1210 is further used to receive user operation data sent by the control terminal, where the user operation data is obtained by the control terminal when the control terminal plays the second screen data;

[0274] A generating module 1230, for generating remote control information based on user operation data;

[0275] The sending module 1240 is used to send remote control information to the controlled end, so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

[0276] In an embodiment of the present application, the intermediate end can receive the first screen data sent by the controlled end when the control end and the controlled end establish a remote connection, the first screen data includes screen data obtained by encoding the second screen data according to a preset encoding method, the preset encoding method includes a Huffman encoding method, and forward the first screen data to the control end, and then can receive the user operation data sent by the control end, the user operation data is obtained when the control end plays the second screen data, and then can generate remote control information based on the user operation data, and send the remote control information to the controlled end, so that the controlled end can perform the control operation corresponding to the remote control information based on the remote control information. In this way, since the above-mentioned remote control information is generated based on the user operation data, the situation in the prior art that only the control gesture information can be transmitted in the remote control is avoided, and the above-mentioned first screen data can include screen data encoded and processed by the second screen data according to the Huffman encoding method, so that the screen data can be efficiently and smoothly transmitted in the remote control, thereby improving the effect of the remote control.

[0277] Each module in the remote control device provided in the embodiment of the present application can realize Figure 8 The method steps of the illustrated embodiment can achieve the corresponding technical effects, and for the sake of brevity, they will not be repeated here.

[0278] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0279] The electronic device may include a processor 1301 and a memory 1302 storing computer program instructions.

[0280] Specifically, the processor 1301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0281] Memory 1302 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 1302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1302 may include a removable or non-removable (or fixed) medium. Where appropriate, memory 1302 may be inside or outside of an electronic device. In a particular embodiment, memory 1302 is a non-volatile solid-state memory.

[0282] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0283] The processor 1301 implements any one of the remote control methods in the above embodiments by reading and executing computer program instructions stored in the memory 1302 .

[0284] In one example, the electronic device may further include a communication interface 1303 and a bus 1310. Fig.13 As shown, the processor 1301, the memory 1302, and the communication interface 1303 are connected via a bus 1310 and communicate with each other.

[0285] The communication interface 1303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0286] Bus 1310 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 1310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0287] In addition, in combination with the remote control method in the above embodiment, the embodiment of the present application can provide a computer storage medium to implement. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the remote control method provided in the embodiment of the present application is implemented.

[0288] An embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the remote control method provided in the embodiment of the present application.

[0289] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0290] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0291] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0292] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable remote-controlled device to produce a machine so that these instructions executed by the processor of the computer or other programmable remote-controlled device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0293] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A remote control method, characterized in that: Applied to the controlled end, the method includes: When it is determined that the control end and the controlled end have established a remote connection, first screen data of the controlled end is obtained, wherein the first screen data of the controlled end includes screen data obtained by encoding the second screen data according to a preset encoding method, and the preset encoding method includes a Huffman encoding method; Sending the first screen data to the intermediate end; receiving remote control information sent by the intermediate end, wherein the remote control information is determined by the intermediate end based on user operation data sent by the control end, and the user operation data is obtained by the control end when the control end plays the second screen data; Based on the remote control information, a control operation corresponding to the remote control information is executed.

2. The method according to claim 1, characterized in that The obtaining of the first screen data of the controlled terminal includes: Acquiring second screen data of the controlled terminal; The second screen data of the controlled terminal is encoded according to a preset encoding method to obtain the first screen data.

3. The method according to claim 1, characterized in that The second screen data includes video data; the video data includes a plurality of image frames, each image frame includes a plurality of pixels and a pixel value of each of the pixels; The step of encoding the second screen data of the controlled terminal according to a preset encoding method to obtain the first screen data includes: For each image frame of the plurality of image frames, determining a coding reference value for each pixel value in the image frame; Constructing a Huffman tree based on the encoding reference value of each pixel value in the image frame; Based on the position of the pixel value of each pixel point in the image frame in the Huffman tree, the image frame is encoded to obtain the encoded data of the image frame, and the first screen data includes the encoded data of each image frame in the multiple image frames.

4. The method according to claim 3, characterized in that The step of determining, for each image frame in the plurality of image frames, a coding reference value for each pixel value in the image frame comprises: For each image frame in the plurality of image frames, determining an occurrence frequency of each pixel value in the image frame based on pixel values ​​of a plurality of pixel points in the image frame; or, For each image frame in the plurality of image frames, determining an occurrence frequency of each pixel value in the image frame based on pixel values ​​of a plurality of pixel points in the image frame; Based on a preset first correspondence between the occurrence frequency and the weight, matching the occurrence frequency of each pixel value in the image frame to determine the weight of each pixel value; The coding reference value includes the occurrence frequency of each pixel value or the weight of each pixel value.

5. A remote control method, characterized in that: Applied to the control end, the method includes: In the case of determining that the control end and the controlled end have established a remote connection, receiving first screen data forwarded by the intermediate end, the first screen data comprising screen data obtained by compressing the second screen data according to a preset coding method, the preset coding method comprising a Huffman coding method; When the control terminal plays the second screen data, acquiring user operation data; The user operation data is sent to the intermediate end so that the intermediate end generates remote control information based on the user operation data, and the remote control information is sent to the controlled end so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

6. The method according to claim 5, characterized in that The obtaining of user operation data includes: receiving a user input on a target object, the control end including the target object, the input being an input made by the user on the target object based on the second screen data; In response to the input, user operation data is acquired.

7. The method according to claim 6, characterized in that The target object includes a virtual object; The receiving user input on the target object includes: receiving a first sub-input of the user on the virtual object; The step of acquiring user operation data in response to the input includes: In response to the first sub-input, user operation data is acquired, the user operation data including an input type of the first sub-input and an input position of the first sub-input in a display area of ​​the control terminal.

8. The method according to claim 6, characterized in that The target object includes a physical object; The receiving user input on the target object includes: receiving a second sub-input of the user regarding the entity object, wherein the second sub-input indicates an object identifier of the entity object; The step of obtaining user operation data in response to the input includes: In response to the second sub-input, based on the correspondence between the preset object identifier and the operation instruction, the object identifier of the physical object is matched to determine the operation instruction corresponding to the physical object, and the user operation data includes the operation instruction corresponding to the physical object.

9. The method according to any one of claims 5 to 8, characterized in that The second screen data includes audio data and video data; before obtaining the user operation data, the method further includes: Decoding the first screen data to obtain second screen data; Synchronously processing the audio data and the video data to obtain synchronized audio data and video data; The synchronized audio and video data are played.

10. The method according to claim 9, characterized in that The audio data includes N sub-audio data and a timestamp of each sub-audio data; The step of synchronously processing the audio data and the video data to obtain the synchronously processed audio data and the video data includes: For an i-th sub-audio data in the N sub-audio data, based on a timestamp of the i-th sub-audio data, determine the sub-video data in the video data corresponding to the timestamp, where N and i are both positive integers; The i-th sub-audio data and the sub-video data corresponding to the i-th sub-audio data are merged.

11. A remote control method, characterized in that: Applied to the middle end, the method includes: In the case where a remote connection is established between a control terminal and a controlled terminal, receiving first screen data sent by the controlled terminal, wherein the first screen data includes screen data obtained by encoding second screen data in a preset encoding method, wherein the preset encoding method includes a Huffman encoding method; forwarding the first screen data to the control terminal; receiving user operation data sent by the control terminal, wherein the user operation data is obtained by the control terminal when the control terminal plays the second screen data; generating remote control information based on the user operation data; The remote control information is sent to the controlled end, so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

12. A remote control device, characterized in that: Applied to the controlled end, the device comprises: an acquisition module, configured to acquire first screen data of the controlled end when it is determined that the control end and the controlled end have established a remote connection, wherein the first screen data of the controlled end includes screen data obtained by encoding the second screen data in a preset encoding method, wherein the preset encoding method includes a Huffman encoding method; A sending module, used for sending the first screen data to the intermediate end; a receiving module, configured to receive remote control information sent by the intermediate terminal, wherein the remote control information is determined by the intermediate terminal based on user operation data sent by the control terminal, wherein the user operation data is obtained by the control terminal when the control terminal plays the second screen data; An execution module is used to execute the control operation corresponding to the remote control information based on the remote control information.

13. A remote control device, characterized in that: Applied to a control terminal, the device comprises: A receiving module, configured to receive first screen data forwarded by the intermediate end when it is determined that the control end and the controlled end have established a remote connection, wherein the first screen data includes screen data obtained by compressing the second screen data according to a preset coding method, and the preset coding method includes a Huffman coding method; An acquisition module, used for acquiring user operation data when the control end plays the second screen data; A sending module is used to send the user operation data to the intermediate end so that the intermediate end generates remote control information based on the user operation data, and to send the remote control information to the controlled end so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

14. A remote control device, characterized in that: Applied to the middle end, the device comprises: A receiving module, used for receiving first screen data sent by the controlled end when the control end and the controlled end establish a remote connection, wherein the first screen data includes screen data obtained by compressing the second screen data according to a preset coding method, and the preset coding method includes a Huffman coding method; A forwarding module, used for forwarding the first screen data to the control terminal; The receiving module is further used to receive user operation data sent by the control terminal, where the user operation data is obtained by the control terminal when the control terminal plays the second screen data; A generating module, used for generating remote control information based on the user operation data; The sending module is used to send the remote control information to the controlled end, so that the controlled end performs a control operation corresponding to the remote control information based on the remote control information.

Images