Method and electronic device for remote data sharing

Abstract

The application provides a method and an electronic device for remote data sharing. The method comprises the following steps: during remote cooperation between a master device and a slave device, the master device selects a suitable remote data transmission channel according to the data type (service type) of the data to be shared from the slave device, and receives the data to be shared from the slave device by means of the selected target transmission channel. According to the method, the suitable target transmission channel is selected according to different service types, so that different types of service data can be transmitted according to the transmission strategy suitable for the service data, and the quality of remote data sharing is improved.

Description

Technical Field

[0001] This application relates to the field of data transmission technology in device collaboration, and more particularly to a method and electronic device for remote data sharing. Background Technology

[0002] The need for multiple devices to work together is increasing. For example, users can project their mobile phone screen onto a computer for operation or remote work. For short-distance device collaboration, near-field communication technologies such as Wi-Fi and Bluetooth can be used, offering relatively high stability. However, for long-distance device collaboration, the collaboration effect is poor due to factors such as distance and network connectivity. For instance, if a user wants to remotely control an office computer (the controlled device) from their home computer (the controlling device), they might want to remotely view or copy files on the controlled device, or log into websites saved on the controlled device. This requires the user to see the controlled device's live feed, remotely control it, and open different applications on it. These increasingly diverse remote collaboration needs also place higher demands on data sharing in remote collaboration. Summary of the Invention

[0003] This application provides a method and electronic device for remote data sharing, which can better meet the needs of different business types, thereby improving the quality of remote data sharing.

[0004] In a first aspect, a method for remote data sharing is provided, comprising: a master control device determining a first data type of data to be shared, wherein the first data type is audio / video data or file data, the audio / video data including at least one of screen display data of the controlled device, real-time audio data or control commands, the file data being a file already stored in the controlled device, and the data to be shared representing shared data to be transmitted from the controlled device to the master control device during remote collaborative work between the master control device and the controlled device; the master control device remotely controlling the controlled device; the master control device selecting a target transmission channel from multiple remote data transmission channels according to the first data type; and the master control device receiving the data to be shared from the controlled device through the target transmission channel.

[0005] In the technical solution of this application, the appropriate target transmission channel is selected according to different service types, so that different types of service data can be transmitted according to the corresponding transmission strategy, thereby improving the quality of remote data sharing.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, multiple remote data transmission channels include real-time communication (RTC) channels, relay channels, and peer-to-peer (P2P) channels. The RTC channel is a general-purpose remote transmission channel, the relay channel is a dedicated remote transmission channel for the main control device that requires forwarding via a relay server corresponding to the main control device, and the P2P channel is a dedicated remote transmission channel for the main control device that requires creation using P2P hole punching technology. When selecting a target transmission channel from multiple remote data transmission channels based on the first data type, it may include: selecting the target transmission channel from the RTC channel and the P2P channel when the first data type is audio / video data; or selecting the target transmission channel from the relay channel and the P2P channel when the first data type is file data. This implementation method achieves progressive selection. First, it locks out multiple candidate transmission channels that can be selected from multiple remote data transmission channels for the current data type. Then, it makes a further selection from these multiple candidate transmission channels. This progressive selection method can facilitate the quick selection of the target transmission channel when there are a large number of multiple remote data transmission channels. Moreover, since the progressive selection process is equivalent to two screenings, it can also avoid selecting the wrong target transmission channel.

[0007] In one example, when the primary data type is audio / video data, the target transmission channel is selected from the RTC channel and the P2P channel. This includes: checking whether the P2P channel has been successfully established; if the P2P channel is successfully established, selecting the P2P channel as the target transmission channel; or, if the P2P channel is not successfully established, selecting the RTC channel as the target transmission channel. In this example, for audio / video data, the P2P channel is prioritized to fully utilize its advantages of low latency and low cost. If the P2P channel fails to be established, the RTC channel is selected as a fallback to prioritize meeting the high latency requirements of audio / video data. This progressive selection method fully utilizes the advantages of the P2P channel while using the RTC channel as a backup in case the P2P channel fails to be established, making it a solution that balances low transmission latency and cost.

[0008] In another example, when the primary data type is file data, the target transmission channel is selected from the relay channel and the P2P channel. This includes: checking if the P2P channel was successfully established; if the P2P channel was successfully established, selecting the P2P channel as the target transmission channel; or, if the P2P channel was not successfully established, selecting the relay channel as the target transmission channel. In this example, for file data, the P2P channel is prioritized to fully utilize its advantages of low latency and low cost. If the P2P channel creation fails, the relay channel is chosen as a fallback to prioritize meeting the transmission cost of file data. This is because the file data volume is too large, and using the RTC channel would result in excessively high transmission costs. This progressive selection method fully utilizes the advantages of the P2P channel while using the relay channel as a backup in case the P2P channel creation fails. It is a solution that achieves the lowest possible transmission latency without increasing costs.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, when the master device receives data to be shared from the controlled device through the target transmission channel, the process may include: the master device sending transmission parameters to the controlled device, the transmission parameters including at least one of video frame rate, resolution, codec type, audio sampling rate, number of channels, and file block size; the controlled device sending the data to be shared to the master device based on the transmission parameters; and the master device receiving the data to be shared through the target transmission channel. In this implementation, the master device informs the controlled device of the transmission parameters to guide the controlled device to send the data to be shared to the master device according to the transmission parameters, thereby enabling the master device to receive and subsequently parse and store the data to be shared according to the transmission parameters when receiving the data to be shared through the target transmission channel.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the master device sending congestion control parameters to the controlled device, the congestion control parameters including at least one of the initial size of the data packet, the initial size of the window value, the maximum window value, and a congestion control strategy, the congestion control strategy including at least one of a slow start and congestion avoidance strategy and a fast recovery strategy; while the master device receives data to be shared from the controlled device through the target transmission channel, the master device and the controlled device perform congestion control based on the congestion control parameters. In this implementation, the master device agrees on the congestion control parameters with the controlled device before transmitting data, so that the two devices can actively perform congestion control based on the pre-agreed congestion control parameters when transmitting data.

[0011] In one example, when the data to be shared is audio / video data, the window value is set in units of frames; when the data to be shared is file data, the window value is set in units of file blocks. A file block represents any file block among multiple file blocks into which each file to be transmitted is divided. Each file block includes multiple data packets, and the data packet is the smallest unit of data transmission. In this example, using different units to set the window value for different data types makes congestion control more suitable for the remote data sharing scenario of this application. For audio / video data, using data frames is because the master control device renders and presents the received audio / video data frame by frame. Using frames avoids the same frame being transmitted across transmission rounds, making congestion control during data transmission more reasonable. For file data, using file blocks is because the master control device needs to store all file blocks of the same file on disk. Using file blocks avoids the same file being transmitted across transmission rounds, making congestion control during data transmission more reasonable. Using frames means that the window size is an integer multiple of the data frame. Similarly, "in units of file blocks" means that the window size is an integer multiple of each file block.

[0012] In another example, during the process of the master device receiving data to be shared from the controlled device through the target transmission channel, the master device and the controlled device perform congestion control based on congestion control parameters. This may include: selecting a target congestion control strategy and a target adjustment strategy from congestion control strategies and / or congestion window size adjustment strategies according to a first data type, the target transmission channel, and the size of the buffer free window; and performing congestion control based on the target congestion control strategy and the target adjustment strategy. In this example, selecting a more suitable congestion control strategy and window adjustment strategy by comprehensively considering the data type, channel type, and buffer free window size can further improve the effectiveness of congestion control and enhance the quality of remote data transmission.

[0013] In one example, when selecting a target congestion control strategy and a target adjustment strategy from congestion control strategies and / or congestion window size adjustment strategies based on a first data type, the target transmission channel, and the size of the buffer free window, the following can be included: If the first data type is audio / video data and the target transmission channel is an RTC channel or a P2P channel, then a fast recovery strategy is determined as the target congestion control strategy; if the first data type is file data and the target transmission channel is a relay channel, then a slow start plus congestion avoidance strategy is determined as the target congestion control strategy; if the first data type is file data and the target transmission channel is a P2P channel, then a fast recovery strategy is determined as the target congestion control strategy. In this example, the target congestion control strategy is determined by combining the data type and the type of the target transmission channel, making the adopted congestion control strategy more suitable for actual needs, thereby improving the effectiveness of congestion control.

[0014] In another example, when selecting a target congestion control strategy and a target adjustment strategy from congestion control strategies and / or congestion window size adjustment strategies based on a first data type, the target transmission channel, and the size of the buffer free window, the strategy may include: when the first data type is audio / video data and the target transmission channel is an RTC channel, determining exponential increase as the target adjustment strategy; when the first data type is audio / video data and the target transmission channel is a P2P channel, determining linear increase as the target adjustment strategy; when the first data type is file data and the target transmission channel is a relay channel, determining exponential increase as the target adjustment strategy; and when the first data type is file data and the target transmission channel is a P2P channel, determining exponential increase as the target adjustment strategy. In this example, the target congestion control strategy is determined by combining the data type and the type of the target transmission channel, making the adopted congestion control strategy more suitable for actual needs, thereby improving the effectiveness of congestion control.

[0015] In another example, the above method further includes: when the target adjustment strategy is linearly increasing, determining whether to change the target adjustment strategy based on the size of the cache free window; when the size of the cache free window is greater than or equal to a preset threshold, changing the target adjustment strategy to exponentially increasing; or, when the size of the cache free window is less than the preset threshold, continuing to use linearly increasing as the target adjustment strategy.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: running a first application on both the master control device and the controlled device, the first application being a remote control application; remotely controlling a second application on the controlled device via the first application on the master control device, the second application being any application on the controlled device other than the first application; and the controlled device transmitting data generated in real time by the second application during remote control to the master control device as shared data. This implementation provides an example of an interaction scenario for the solution of this application, demonstrating how remote control applications can control any application on the controlled device.

[0017] In a second aspect, a remote data sharing apparatus is provided, comprising a unit consisting of software and / or hardware for performing any of the methods of the first aspect.

[0018] Thirdly, an electronic device is provided, comprising: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to enable the electronic device to implement any of the methods of the first aspect.

[0019] When the electronic device is used to perform the steps executed by the master device in any method of the first aspect, the electronic device is the master device; when the electronic device is used to perform the steps executed by the controlled device in any method of the first aspect, the electronic device is the controlled device.

[0020] Fourthly, a remote data sharing system is provided, which includes a master control device and a controlled device. The master control device is used to remotely control the controlled device. The master control device is capable of executing the steps performed by the master control device in any of the methods of the first aspect, and the controlled device is capable of executing the steps performed by the controlled device in any of the methods of the first aspect.

[0021] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed on an electronic device, enable the electronic device to implement any of the methods of the first aspect.

[0022] In a sixth aspect, a computer program product is provided, comprising a computer program that, when executed by an electronic device, can implement any of the methods of the first aspect. Attached Figure Description

[0023] Figure 1 This is an interactive schematic diagram of a remote data sharing scenario applicable to the embodiments of this application.

[0024] Figure 2 This is a schematic flowchart illustrating a remote data sharing method according to an embodiment of this application.

[0025] Figure 3 This is a schematic flowchart illustrating another method for remote data sharing according to an embodiment of this application.

[0026] Figure 4 This is a software architecture diagram of a remote data transmission according to an embodiment of this application. Both the master control device and the controlled device can adopt... Figure 4 The software architecture is shown.

[0027] Figure 5 This is a schematic flowchart illustrating a notification method for remote data sharing front-end device online detection and channel connection establishment according to an embodiment of this application.

[0028] Figure 6 This is a schematic flowchart illustrating another method for remote data sharing according to an embodiment of this application.

[0029] Figure 7 This is a schematic flowchart illustrating a negotiation parameter and data transmission based on the negotiation parameter, according to an embodiment of this application.

[0030] Figure 8 This is a schematic diagram of the window value change strategy for two congestion control strategies in embodiments of this application.

[0031] Figure 9 This is a schematic flowchart of a congestion control process according to an embodiment of this application.

[0032] Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0033] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0034] Figure 1 This is an interactive diagram illustrating a remote data sharing scenario applicable to an embodiment of this application. For example... Figure 1 As shown, electronic device A (an example of a master control device) and electronic device B establish a remote control connection through a remote control application (application C). This allows a user to remotely control electronic device B through application C on electronic device A. For example, the user can copy stored files from electronic device B, display the controlled screen of electronic device B on electronic device A, or play online audio data from electronic device B. For instance, the user can remotely open a browser on electronic device B from electronic device A. Figure 1(An example of application D) When entering a URL, performing a search, or browsing web pages online in the browser, the screen of electronic device B is synchronously transmitted to electronic device A as audio and video data. Operations on electronic device B, such as clicking, cursor movement, and typing, are input via a keyboard and mouse connected to the electronic device and sent to electronic device B. Electronic device B then responds and feeds back the resulting screen changes as audio and video data to electronic device A. If electronic device B is playing a song or audio / video online, the playing video is fed back to electronic device A frame-by-frame based on changes in the screen (captured by application C), rather than the source video stream being sent directly to electronic device A. Similarly, the audio data played online by electronic device B is also fed back to electronic device A frame-by-frame as audio data captured by application C, rather than the source audio stream being sent directly to electronic device A. This is what's known as collaborative work across multiple devices. User input is provided via the input module of the main control device (e.g., keyboard, mouse, touch panel). Application C then sends this input data (which can be considered control commands for remotely controlling the controlled device) to the controlled device. After the controlled device responds to the input data, the response result is displayed directly on the controlled device, and this displayed response result is simultaneously fed back to the main control device. For example, if a user wants to copy a file from the controlled device, they first need to remotely search for the corresponding folder until the file is found. During this process, frame-by-frame remote transmission of audio and video data is involved. Once the user copies the file to the local storage unit of the main control device, the remote transmission of the file data begins. Other scenarios are not listed here.

[0035] This application addresses the instability that can occur when sending response results back to the master control device, due to factors such as large data volume, long transmission distance, the possibility of the two devices being on different local area networks, and network transmission fluctuations. To resolve these issues, this application categorizes the types of business data (data types requiring shared transmission) that may be involved in remote control scenarios into two types: audio / video data and file data. The former refers to the video and audio / video data generated under real-time control, while the latter refers to file data when copying local files (already stored files) from the controlled device. The application also summarizes the advantages and disadvantages of each available remote data transmission channel, thereby selecting the most suitable remote data transmission channel for different data types to improve the quality of remote data sharing. This selection process is dynamic, rather than rigid, allowing the data transmission method in remote control scenarios to adaptively adjust according to real-time changes in data types, thus improving overall data transmission quality.

[0036] Furthermore, this application does not limit the specific file type of the file data. For example, it can be various documents such as .exe files, Word, Excel, and PDF, various compressed files, audio files, and video files, etc., which will not be listed one by one. When it is an audio file or video file, it refers to a file that has been stored on the controlled device, not the audio and video played online as mentioned above, nor the video stream of the shared screen as mentioned above.

[0037] It should be understood that the "master control" in the master control device here can be understood as active control. The terms master control and controlled are mainly used to illustrate the use of one device to control another device.

[0038] The master control device and the controlled device can be any electronic device that can be used remotely in collaboration with multiple devices, such as mobile phones, tablets, laptops, personal computers, servers, vehicle-mounted systems, and smart wearable devices.

[0039] Figure 2 This is a schematic flowchart illustrating a remote data sharing method according to an embodiment of this application. The following is a description of... Figure 2 The steps shown are explained below.

[0040] S201, The main control device determines the first data type of the data to be shared.

[0041] The first data type is audio / video data or file data. Audio / video data includes at least one of the screen display data of the controlled device, real-time audio data, or control commands. File data is a file already stored in the controlled device. The data to be shared is used to represent the shared data that the controlled device is to transmit to the master device during the remote collaborative work between the master device and the controlled device. The master device is used to remotely control the controlled device.

[0042] Figure 1 Electronic device A and electronic device B are examples of the master control device and the controlled device, respectively. For further explanation of audio / video data and file data, please refer to the relevant descriptions above; they will not be repeated here.

[0043] It should be understood that since the user remotely controls the controlled device through the master control device, the control command is received by the input module of the master control device. Therefore, the master control device first knows and determines what type of data it wants to obtain from the controlled device. It can be understood that the execution time of step S201 is not that the controlled device has already prepared the data to be shared and is just waiting to be transmitted, but rather that the master control device determines what type of data it wants to obtain from the controlled device.

[0044] S202. The main control device selects the target transmission channel from multiple remote data transmission channels based on the first data type.

[0045] Once the master control device determines what type of data it wants to obtain from the controlled device, it can select a suitable target transmission channel accordingly.

[0046] In this application embodiment, the data types are divided into audio / video data and file data primarily based on their different requirements for transmission latency and the different sizes of data. Audio / video data has relatively high latency requirements, meaning it needs low transmission latency. File data, on the other hand, is often much larger in size. Given such a large data volume, pursuing low latency would inevitably increase transmission costs. Remote transmission of file data is somewhat similar to downloading files. While users want the download to be as fast as possible, they can also accept a certain amount of waiting time. Therefore, in this application's remote data sharing scenario, file data will be transmitted using a transmission channel with relatively low transmission costs.

[0047] In the embodiments of this application, the multiple remote data transmission channels may include any combination of real-time communication channels, relay channels, and peer-to-peer (P2P) channels, which will be described below. However, it should be understood that with the development of technology, new transmission channels capable of remote data transmission may emerge in the future and can still be added to the multiple remote data transmission channels of this application. It is only necessary to summarize the characteristics of the new data transmission channel, such as transmission latency, transmission cost, and creation success rate, in order to select a suitable remote data transmission channel for data of different types to be shared.

[0048] Relay Channel: This method requires building a self-owned RTC cloud for the controlled device. The controlled and master devices interact with the cloud devices via the traversal using relays around network address translation (TURN NAT) protocol, relying on a cloud relay (relay server) for forwarding. This ultimately enables data exchange between the two devices. Because it utilizes a self-built RTC cloud for relay forwarding, it can also be called a self-developed relay channel, a dedicated remote data transmission channel for the master and controlled devices. RTC is a general term for technologies that allow real-time audio, video, and data transmission over the internet, typically used in remote audio / video, online gaming, and instant messaging scenarios. The TURN protocol is a network protocol primarily used to solve the real-time communication problem between different end-devices in a NAT environment. It allows end-devices behind NAT (or multiple NATs) to bypass NAT and achieve data exchange between clients by interacting with a relay server.

[0049] P2P Channel: A self-built session traversal utilities for NAT (STUNNAT) cloud server allows end-devices (controlled and master devices) to perform NAT detection via the STUN protocol. Data exchange between end-devices then occurs through peer-to-peer hole punching. Because this involves a self-built cloud server for NAT detection and requires P2P hole punching, this remote data transmission channel can be called a self-developed P2P channel. STUN is a network protocol that allows end-devices behind NAT (or multiple NATs) to find their public IP address and port. P2P hole punching is primarily used to establish peer-to-peer (P2P) connections in NAT environments. Due to NAT, direct connections between two devices on different networks are often hindered; P2P hole punching aims to solve this problem. However, hole punching is not guaranteed to succeed, so the aforementioned P2P channel establishment may fail.

[0050] RTC Channel: In this application, it mainly refers to the direct purchase of a complete remote data transmission solution provided by a third-party RTC vendor. This solution offers stable data transmission capabilities and is compatible with any end-device manufacturer. "Third-party" refers to the solution provided by the end-device manufacturer. The two types of remote data transmission channels mentioned above are designed by the end-device manufacturer using self-built relay servers or cloud servers. In contrast, a third-party RTC channel directly uses a complete product already designed by another vendor; therefore, it can be called a third-party RTC channel. It should be understood that because it directly uses a product provided by a third-party RTC vendor, there are additional costs associated with purchase and subsequent usage, resulting in relatively higher transmission costs.

[0051] The characteristics of the three channels mentioned above are summarized in Table 1 below.

[0052] Table 1

[0053]

[0054] As shown in Table 1, the P2P channel has the advantages of low transmission latency and low cost, making it suitable for remote transmission of audio, video, and file data. However, due to its low success rate in hole punching, it also has the disadvantage of a low connection establishment success rate. The RTC channel has the advantages of high connection establishment success rate, low transmission latency, and stable performance, making it suitable for remote transmission of audio, video, and file data. However, since it requires payment to third-party vendors, its cost is higher. Therefore, it is not recommended for file data, which is often large in volume. The relay channel can be seen as a compromise between the other two channels. Its connection establishment success rate is higher than that of the P2P channel, its transmission latency is higher than both the RTC and P2P channels, and its cost is lower than that of the RTC channel. Therefore, it is not suitable for remote transmission of audio and video data with high latency requirements, but it is suitable for the transmission of large file data.

[0055] In one implementation, based on the data type of the data to be shared, the remote data transmission channel corresponding to that data type in a first mapping relationship is determined as the target transmission channel. The first mapping relationship includes the correspondence between each data type and a remote data transmission channel. In this implementation, the transmission channel is determined using a preset method, which only requires a table lookup, making it simple and fast.

[0056] However, the above implementation method also has some problems. As shown in Table 1, a data type can correspond to more than one transmission channel. Therefore, directly and rigidly stipulating that data of this type can only use one transmission channel will prevent the advantages of different transmission channels from being utilized. In particular, P2P channels may fail to be created, which will prevent remote transmission. Therefore, in order to further improve the quality of remote transmission, we can first lock the available transmission channels for this data type based on the advantages and disadvantages of different remote data transmission channels, and then make further selections from them.

[0057] In one implementation, multiple remote data transmission channels include Real-Time Communication (RTC) channels, relay channels, and peer-to-peer (P2P) channels. The RTC channel is a general-purpose remote transmission channel, the relay channel is a dedicated remote transmission channel for the main control device that requires forwarding via a relay server corresponding to the main control device, and the P2P channel is a dedicated remote transmission channel for the main control device that requires P2P hole punching technology. When selecting a target transmission channel from multiple remote data transmission channels based on a first data type, the selection can include: if the first data type is audio / video data, selecting the target transmission channel from both the RTC channel and the P2P channel; or, if the first data type is file data, selecting the target transmission channel from both the relay channel and the P2P channel. This implementation achieves a progressive selection: first, multiple candidate transmission channels for the current data type are locked from the multiple remote data transmission channels, and then further selection is made from these candidate channels. This progressive selection method can facilitate quick and easy selection of the target transmission channel when there are many remote data transmission channels. Furthermore, since the progressive selection process is equivalent to performing two rounds of filtering, it can also help avoid selecting the wrong target transmission channel.

[0058] In one example, when the primary data type is audio / video data, the target transmission channel is selected from the RTC channel and the P2P channel. This includes: checking whether the P2P channel has been successfully established; if the P2P channel is successfully established, selecting the P2P channel as the target transmission channel; or, if the P2P channel is not successfully established, selecting the RTC channel as the target transmission channel. In this example, for audio / video data, the P2P channel is prioritized to fully utilize its advantages of low latency and low cost. If the P2P channel fails to be established, the RTC channel is selected as a fallback to prioritize meeting the high latency requirements of audio / video data. This progressive selection method fully utilizes the advantages of the P2P channel while using the RTC channel as a backup in case the P2P channel fails to be established, making it a solution that balances low transmission latency and cost.

[0059] In another example, when the primary data type is file data, the target transmission channel is selected from the relay channel and the P2P channel. This includes: checking if the P2P channel was successfully established; if the P2P channel was successfully established, selecting the P2P channel as the target transmission channel; or, if the P2P channel was not successfully established, selecting the relay channel as the target transmission channel. In this example, for file data, the P2P channel is prioritized to fully utilize its advantages of low latency and low cost. If the P2P channel creation fails, the relay channel is chosen as a fallback to prioritize meeting the transmission cost of file data. This is because the file data volume is too large, and using the RTC channel would result in excessively high transmission costs. This progressive selection method fully utilizes the advantages of the P2P channel while using the relay channel as a backup in case the P2P channel creation fails. It is a solution that achieves the lowest possible transmission latency without increasing costs.

[0060] S203. The master control device receives the data to be shared from the controlled device through the target transmission channel.

[0061] It should be understood that the controlled device needs to encode or segment the data to be shared (encoding is required for audio and video data, and segmentation is required for file data), and package it before the controlled device sends the packaged data to be shared to the master device through the target transmission channel. Correspondingly, the master device will receive the packaged data to be shared through the target transmission channel, and then can perform decoding or disk writing operations accordingly.

[0062] In order to achieve data transmission, the master control device and the controlled device need to agree on the rules for data transmission. Therefore, before or during the execution of step S203, it is also necessary to inform the controlled device what rules to follow to prepare and send data.

[0063] In one implementation, when the master device receives data to be shared from the controlled device through the target transmission channel, the process may include: the master device sending transmission parameters to the controlled device, the transmission parameters including at least one of video frame rate, resolution, codec type, audio sampling rate, number of channels, and file block size; the controlled device sending the data to be shared to the master device based on the transmission parameters; and the master device receiving the data to be shared through the target transmission channel. In this implementation, the master device informs the controlled device of the transmission parameters to guide the controlled device to send the data to be shared to the master device according to the transmission parameters, thereby enabling the master device to receive and subsequently parse and store the data according to the transmission parameters when receiving the data to be shared through the target transmission channel.

[0064] Video frame rate, resolution, and codec type are transmission parameters for video data; audio sampling rate and number of channels are transmission parameters for video data; and file block size is a transmission parameter for file data. In this embodiment, audio and video data are transmitted frame by frame and presented synchronously on the main control device, while file data is transmitted after being divided into several file blocks according to the file block size. It should be understood that file blocks and data packets are different concepts; the former is the unit of file division, while the latter is the smallest transmission unit for data transmission, and each file block includes several data packets.

[0065] Data transmission is prone to congestion, a common phenomenon in network transmission, which has led to the development of numerous congestion control strategies. Therefore, in the scheme of this application embodiment, different congestion control parameters can be set based on the characteristics of different data types to guide the master control device and the controlled device to perform congestion control according to more reasonable congestion control parameters. However, it should be understood that this application does not limit the control operations during specific congestion control periods.

[0066] In one implementation, the method further includes: the master device sending congestion control parameters to the controlled device. These parameters include at least one of the following: initial packet size, initial window size, maximum window size, and congestion control strategy. The congestion control strategy includes at least one of a slow start and congestion avoidance strategy and a fast recovery strategy. During the period when the master device receives data to be shared from the controlled device through the target transmission channel, the master device and the controlled device perform congestion control based on the congestion control parameters. In this implementation, the master device agrees on the congestion control parameters with the controlled device before transmitting data, enabling both devices to proactively perform congestion control based on the pre-agreed parameters during data transmission.

[0067] It should also be understood that congestion control parameters can be sent to the controlled device along with the aforementioned transmission parameters, thereby reducing the number of data interactions and saving power consumption and network usage to some extent.

[0068] In one example, when the data to be shared is audio / video data, the window value is set in units of frames; when the data to be shared is file data, the window value is set in units of file blocks. A file block represents any file block among multiple file blocks into which each file to be transmitted is divided. Each file block includes multiple data packets, and the data packet is the smallest unit of data transmission. In this example, using different units to set the window value for different data types makes congestion control more suitable for the remote data sharing scenario of this application. For audio / video data, using data frames is because the master control device renders and presents the received audio / video data frame by frame. Using frames avoids the same frame being transmitted across transmission rounds, making congestion control during data transmission more reasonable. For file data, using file blocks is because the master control device needs to store all file blocks of the same file on disk. Using file blocks avoids the same file being transmitted across transmission rounds, making congestion control during data transmission more reasonable. Using frames means that the window size is an integer multiple of the data frame. Similarly, "in units of file blocks" means that the window size is an integer multiple of each file block.

[0069] In another example, during the process of the master device receiving data to be shared from the controlled device through the target transmission channel, the master device and the controlled device perform congestion control based on congestion control parameters. This may include: selecting a target congestion control strategy and a target adjustment strategy from congestion control strategies and / or congestion window size adjustment strategies according to a first data type, the target transmission channel, and the size of the buffer free window; and performing congestion control based on the target congestion control strategy and the target adjustment strategy. In this example, selecting a more suitable congestion control strategy and window adjustment strategy by comprehensively considering the data type, channel type, and buffer free window size can further improve the effectiveness of congestion control and enhance the quality of remote data transmission.

[0070] The cache free window determines whether congestion is about to occur. It can be understood that during data transmission, there is a preset buffer space. When there is too much data to be transmitted, it will be temporarily cached in the buffer space. Therefore, the remaining size of this buffer space (that is, the size of the free window) can intuitively reflect whether congestion has occurred and the degree of congestion. Therefore, the size of the cache free window can be used as a reference parameter for how to perform congestion control.

[0071] In one example, when selecting a target congestion control strategy and a target adjustment strategy from congestion control strategies and / or congestion window size adjustment strategies based on a first data type, the target transmission channel, and the size of the buffer free window, the following can be included: If the first data type is audio / video data and the target transmission channel is an RTC channel or a P2P channel, then a fast recovery strategy is determined as the target congestion control strategy; if the first data type is file data and the target transmission channel is a relay channel, then a slow start plus congestion avoidance strategy is determined as the target congestion control strategy; if the first data type is file data and the target transmission channel is a P2P channel, then a fast recovery strategy is determined as the target congestion control strategy. In this example, the target congestion control strategy is determined by combining the data type and the type of the target transmission channel, making the adopted congestion control strategy more suitable for actual needs, thereby improving the effectiveness of congestion control.

[0072] In another example, when selecting a target congestion control strategy and a target adjustment strategy from congestion control strategies and / or congestion window size adjustment strategies based on a first data type, the target transmission channel, and the size of the buffer free window, the strategy may include: when the first data type is audio / video data and the target transmission channel is an RTC channel, determining exponential increase as the target adjustment strategy; when the first data type is audio / video data and the target transmission channel is a P2P channel, determining linear increase as the target adjustment strategy; when the first data type is file data and the target transmission channel is a relay channel, determining exponential increase as the target adjustment strategy; and when the first data type is file data and the target transmission channel is a P2P channel, determining exponential increase as the target adjustment strategy. In this example, the target congestion control strategy is determined by combining the data type and the type of the target transmission channel, making the adopted congestion control strategy more suitable for actual needs, thereby improving the effectiveness of congestion control.

[0073] In another example, the above method further includes: when the target adjustment strategy is linearly increasing, determining whether to change the target adjustment strategy based on the size of the cache free window; when the size of the cache free window is greater than or equal to a preset threshold, changing the target adjustment strategy to exponentially increasing; or, when the size of the cache free window is less than the preset threshold, continuing to use linearly increasing as the target adjustment strategy.

[0074] The above examples demonstrate how the target congestion control strategy and the target adjustment strategy can be implemented in combination to achieve better congestion control results.

[0075] It should be understood that this application, by summarizing the characteristics of different service types and transmission channels, and combining this with the size of the buffer free window, selects a relatively appropriate congestion control strategy and a congestion window size adjustment strategy for different situations. Table 2 shows an example of the correspondence between the various situations mentioned above.

[0076] Table 2

[0077]

[0078] The specific procedures for congestion control will be explained in detail below, and will not be repeated here.

[0079] In one implementation, the method further includes: running a first application on both the master control device and the controlled device; remotely controlling a second application on the controlled device via the first application on the master control device; the second application being any application on the controlled device other than the first application; and the controlled device transmitting data generated in real-time by the second application during remote control to the master control device as shared data. This implementation provides an example of the interaction scenario of the proposed solution, demonstrating control of any application on the controlled device through a remote control application. This "any application" can be a browser, chat application, shopping application, desktop application, file management application, etc., and will not be listed individually.

[0080] Figure 2 The method shown primarily improves the quality of remote data sharing by selecting appropriate target transmission channels based on different service types, thereby enabling different types of service data to be transmitted according to a suitable transmission strategy.

[0081] Figure 3 This is a schematic flowchart illustrating another method for remote data sharing according to an embodiment of this application. Figure 3 yes Figure 2 An example of the method shown.

[0082] S301, The main control device starts the remote control service.

[0083] For example, an application for remote control can be launched on both the master and slave devices.

[0084] S302. The main control equipment determines the service type.

[0085] The so-called service type can be understood as the service type of remote data transmission, which can include audio and video transmission services and file transmission services. Therefore, step S302 can also be called determining the data type, that is, determining whether the data to be remotely transmitted is audio and video data or file data.

[0086] S302 can be seen as an example of S201.

[0087] S303. When the service type is audio and video transmission service, the main control device simultaneously begins to establish a connection between the self-developed P2P channel (an example of the above P2P channel) and the third-party RTC channel (an example of the above RTC channel).

[0088] S304. The main control device checks whether the self-developed P2P channel has been successfully established. If the test result is "yes", proceed to step S305; otherwise, proceed to step S306.

[0089] S305. The main control device enables the self-developed P2P channel and releases the third-party RTC channel, after which the controlled device begins to execute step S311.

[0090] S306. The main control device enables the third-party RTC channel, and then the controlled device begins to execute step S311.

[0091] S307. When the service type is file transfer, the main control device simultaneously begins to establish a connection between the self-developed P2P channel (an example of the aforementioned P2P channel) and the self-developed relay channel (an example of the aforementioned relay channel).

[0092] S308. The main control device checks whether the self-developed P2P channel has been successfully established. If the test result is "yes", proceed to step S309; ​​otherwise, proceed to step S310.

[0093] S309. The main control device enables the self-developed P2P channel and releases the self-developed relay channel, after which the controlled device begins to execute step S314.

[0094] S310, The main control device enables the self-developed relay channel, and then the controlled device begins to execute step S314.

[0095] S303-S310 can be seen as an example of S202.

[0096] S311 The controlled device performs data acquisition, encoding, and packaging of the audio and video data to be shared.

[0097] S312. The controlled device transmits the packaged audio and video data to the master control device through the selected transmission channel.

[0098] S313 The main control device unpacks, decodes, and renders the received audio and video data.

[0099] S314. The controlled device performs block division, reading, encryption and packaging of the file data to be shared.

[0100] S315. The controlled device transmits the packaged file data to the master control device through the selected transmission channel.

[0101] S316. The main control device unpacks, decrypts, and stores the received file data into the main control device's storage unit.

[0102] S311-S316 can be seen as an example of S203.

[0103] S317. The main control device terminates remote control services.

[0104] exist Figure 3 In the illustrated scheme, given the high latency requirements of audio and video data, RTC or P2P channels are prioritized. For file data, which is often much larger, and considering the higher cost of RTC channels, P2P or relay channels are prioritized. Furthermore, when further selecting between the two channels, a progressive selection method is used based on the success rate of P2P channel establishment. This is primarily due to the low latency and cost of P2P channels, but their low success rate. If a P2P channel is successfully established, it can be prioritized for data transmission. Even if establishment fails, the next best channel will be selected based on the aforementioned progressive selection strategy.

[0105] It should be noted that, Figure 3 The different steps in the scheme shown can be omitted if there is no strong dependency. Figure 3 The steps shown are executed sequentially. For example, data type determination can be skipped initially, and the creation of the three channels can be started simultaneously. After the creation begins and the first preset time has elapsed, it is determined whether the P2P channel has been successfully created. If the P2P channel is successfully created, it is directly used for remote data transmission, and the other two channels are released. Otherwise, if the P2P channel has not been successfully created, the RTC channel or relay channel is further selected for remote data transmission based on the data type. This implementation requires the creation of all three channels each time, but the transmission channel is only further selected based on the service type (the data type of the data to be shared) when the P2P channel creation fails. Therefore, once the P2P channel can be successfully created, no further selection is needed. For example, steps S311 and S314 can be executed after determining the service type, and do not necessarily have to wait until the target transmission channel is selected and created. Other cases will not be listed one by one.

[0106] Figure 4 This is a software architecture diagram of a remote data transmission according to an embodiment of this application. Both the master control device and the controlled device can adopt... Figure 4 The software architecture is shown below. Figure 4As shown, the software architecture diagram includes a remote control app (the app in the application layer), a remote transmission service module (the module in the architecture layer), and a cloud service module.

[0107] The remote control app provides an interactive medium for users to perform remote control. The remote transmission service module (in this application, "remote" can also be referred to as "far-field") includes a service layer interface module, a data acquisition and rendering module, a channel management layer (channel management module), and multiple remote data transmission channel modules.

[0108] The service layer interface module includes various service interfaces involved in remote control scenarios, such as audio call services, video call services, desktop control services, and file transfer services.

[0109] The data acquisition and rendering module is used to acquire or render various types of data in remote control scenarios. The controlled device mainly performs audio and video data acquisition and file reading, while the master device mainly performs audio and video data rendering and file writing (storing to local storage units). The data acquisition and rendering module can include an audio acquisition and rendering module, a video acquisition and rendering module, a desktop acquisition and rendering module, and a file reading and writing module.

[0110] The channel management layer includes a channel scheduling decision module, a congestion control module, and a transmit / receive buffer management module. The channel scheduling decision module is mainly used to select a suitable remote data transmission channel based on the data type of the data to be shared. The congestion control module is mainly used to perform congestion control during data transmission to improve data transmission quality. The transmit / receive buffer management module mainly manages the size of the buffer space for the data to be transmitted during data transmission and / or the buffer location used.

[0111] Multiple remote data transmission channels can include third-party RTC channels, self-developed P2P channels, and self-developed relay channels. The third-party RTC channel can include a third-party software development kit (SDK) and audio / video transmission channels. The self-developed P2P channel can include a NAT detection and hole-punching module and a data sending / receiving module. The self-developed relay channel can include a relay address allocation module and a data sending / receiving module. A third-party SDK can also be referred to as a third-party SDK (so / dll).

[0112] It should be understood that the terms "third-party" or "self-developed" refer to communication methods developed by the main control device manufacturer, in relation to the main control device itself. Third-party RTC is a universal communication method for remote data sharing across all electronic equipment manufacturers, while self-developed P2P and relay channels are dedicated communication methods developed by the main control device manufacturer specifically for its product characteristics and applicable to remote data sharing. If other new remote data transmission channels emerge in the future, they can still be applied to the scheme of this application, simply adding another option to the multiple remote data transmission channels available. However, when selecting a target transmission channel, it is necessary to consider the advantages and disadvantages of the new remote data transmission channel to determine which data type it is more suitable for.

[0113] Cloud services may include cloud storage modules for each channel, such as third-party RTC cloud, self-developed P2P cloud (also known as self-developed RTC cloud or STUN cloud), and self-developed relay cloud (also known as self-developed TURN cloud).

[0114] Figure 5 This is a schematic flowchart illustrating a notification method for remote data sharing front-end device online detection and channel connection establishment according to an embodiment of this application. Figure 5 It can be seen as for Figure 2 or Figure 3 The preparatory work for related steps includes, for example, notifying the controlled device to create the corresponding transmission channel after determining what kind of transmission channel to create, and informing the master device of the identifiers (IDs) of the master and controlled devices being remotely controlled. Both the master and controlled devices include a remote app, a remote transmission service module, a device management module, and a Message Queuing Telemetry Transport (MQTT) remote connection module. The device cloud acts as a "data relay station," storing data sent from the master or controlled device and forwarding it to the controlled or master device. The MQTT remote connection module is mainly used to transmit the data required for remote connection establishment. The transmission parameters and congestion control parameters mentioned above can both be transmitted using the MQTT remote connection module.

[0115] S501, The main control device's device management module starts the sensing function to determine whether the remote data transmission channel connection function of the sensing device is online.

[0116] In other words, it senses whether the master control device and the controlled device can create a remote data transmission channel.

[0117] The device management module can be Figure 4 The module in the channel management layer.

[0118] S502, The device management module of the main control device generates and caches the ID of this device.

[0119] In other words, the device management module of the main control device first generates and saves the ID of the main control device.

[0120] S503, the device management module of the main control device sends a request to the MQTT remote connection module of the main control device to start remote connection establishment.

[0121] S504. The MQTT remote connection module of the main control device establishes an end-to-cloud connection with the device cloud. This end-to-cloud connection is for establishing an MQTT remote connection channel.

[0122] It should be understood that after the connection between the MQTT remote connection module and the device cloud is established, the ID of the master control device can be reported to the device cloud, or the ID of the controlled device can be obtained from the device cloud.

[0123] S505. The device management module of the controlled device starts the sensing function to determine whether the remote data transmission channel connection function of the sensing device is online.

[0124] In other words, it is about sensing whether the controlled device and the controlled device can create a remote data transmission channel.

[0125] The device management module can be Figure 4 The module in the channel management layer.

[0126] S506, The device management module of the controlled device generates and caches the ID of this device.

[0127] In other words, the device management module of the controlled device first generates and saves the ID of the controlled device.

[0128] S507. The device management module of the controlled device sends a request to the MQTT remote connection module of the controlled device to start remote connection establishment.

[0129] S508, The MQTT remote connection module of the controlled device establishes an end-to-cloud connection with the device cloud. This end-to-cloud connection is for establishing an MQTT remote connection channel.

[0130] It should be understood that once the connection between the MQTT remote connection module and the device cloud is established, the ID of the controlled device can be reported to the device cloud.

[0131] It should also be understood that there is no sequential requirement for the execution of steps S501-S504 and steps S505-S508. The above numbering is only to illustrate that the two devices independently generate and report device IDs to the device cloud.

[0132] S509, the device cloud sends the online message of the controlled device to the MQTT remote connection module of the master device.

[0133] S510: The MQTT remote connection module of the master device further sends the online notification of the controlled device to the device management module of the master device.

[0134] S511, the device management module of the master control device caches the ID of the peer device, that is, it caches the ID of the controlled device paired with the master control device.

[0135] S512, The device management module of the main control device notifies the remote app that the establishment of remote transmission channels for the main control device and the controlled device has been launched.

[0136] Steps S501-S512 mainly explain the steps for device online sensing. Device online sensing mainly refers to checking whether the master control device and the controlled device have been prepared to establish a remote data transmission channel.

[0137] S521-S529 provide examples of how the master device can notify the controlled device to start establishing a remote data transmission channel via MQTT messages.

[0138] S521. Start the remote control service in the remote app of the main control device.

[0139] S522, The remote app of the main control device sends a notification message to the remote transmission service of the main control device to start the connection of the remote data transmission channel.

[0140] It should be noted that the master control device can determine the data type to be transmitted here, so the notification message for establishing the remote data transmission channel sent here carries the data type (service type).

[0141] S523. After receiving the above notification information, the remote transmission service of the master control device requests the ID information of the master control device and the controlled device from the device management module.

[0142] S524. The device management module of the main control device returns the ID information of the main control device and the controlled device to the device management module of the main control device.

[0143] It should be understood that when the above device goes online, the device management module of the main control device has already cached the IDs of the main control device and the controlled device. Therefore, it is only necessary to read the ID information of the two devices here.

[0144] S525, the remote transmission service of the master control device sends a notification message to the MQTT remote connection module of the master control device to notify the peer device (controlled device) to establish a remote data transmission channel.

[0145] It should be noted that the notification message in this step carries a data type, such as audio / video data or file data.

[0146] S526, the MQTT remote connection module of the master device sends MQTT messages to the device cloud.

[0147] The MQTT message carries the device ID and service type.

[0148] S527. Based on the received MQTT message, the device cloud forwards the MQTT message to the MQTT remote connection module of the controlled device.

[0149] S528, The MQTT remote connection module of the controlled device transmits the MQTT message to the remote transmission service of the controlled device.

[0150] S529. The remote transmission service of the controlled device sends a notification of the remote data transmission channel to the remote app of the controlled device based on the MQTT message.

[0151] After a user starts the remote control service through the remote control app, the user first needs to rely on the MQTT data channel established during the previous device online, authentication, and binding phases (where MQTT is a short connection on Android devices and a long connection on PC devices) to complete the negotiation process for the dual-end remote desktop control service. The negotiation information may include: dual-end device types, device IDs, and the type of service to be started (the data type to be transmitted), etc.

[0152] Once the preparations are complete, the proposed solution can be implemented for remote data transmission.

[0153] Figure 6 This is a schematic flowchart illustrating another method for remote data sharing according to an embodiment of this application. Figure 6 yes Figure 3 An example of the method shown. Figure 6 The method described in this application is mainly explained from the execution flow of its various internal modules. It should be noted that... Figure 6 Taking audio and video transmission services as an example, where the data to be shared is audio and video data, the transmission channel may be a self-developed P2P channel or a third-party RTC channel. The specific strategy for selecting the target transmission channel is a progressive selection. First, the self-developed P2P channel and the third-party RTC channel are established simultaneously. Then, based on whether the P2P channel is successfully established, it is decided whether to use the self-developed P2P channel or the third-party RTC channel as the target transmission channel.

[0154] S601. Start the remote control service in the remote control app of the main control device.

[0155] The startup method can be in response to a user's selection of a control within the remote control app to initiate remote control services. This application does not limit the startup method.

[0156] S602. The remote control app of the main control device sends a notification to the audio and video call service in the remote transmission service of the main control device to start the audio and video transmission service.

[0157] The notification for this audio and video transmission service can also be understood as informing the audio and video call service of the need to transmit audio and video data.

[0158] S603, the audio and video call service in the main control device sends a notification to the channel management module (the module in the remote transmission service) to start channel connection establishment, that is, to inform the channel management module to start channel connection establishment.

[0159] Since this involves audio and video data, the connection between the third-party RTC channel and the self-developed P2P channel is established simultaneously.

[0160] S604. The channel management module notifies the transmission module in the third-party RTC transmission channel (here, the audio / video transmission channel) to initiate the connection of the third-party RTC channel.

[0161] It should be understood that the remote transmission service of the main control device includes various modules for each remote data transmission channel. Figure 6 When the transmission channel is a third-party RTC channel, the transmission module refers to the audio and video transmission channel, and the third-party RTC channel also includes the third-party SDK; when Figure 6 When the transmission channel is a self-developed P2P channel, the transmission module refers to the data sending and receiving module, excluding third-party SDKs.

[0162] S605, the main control device's transmission module requests the third-party SDK module to join the third-party RTC channel.

[0163] For example, it can be specified using the channel identifier roomID.

[0164] S606, the third-party SDK module of the main control device establishes an end-to-cloud connection with the third-party RTC cloud.

[0165] S607. The third-party RTC cloud in the device cloud sends feedback to the channel management module in the main control device that the third-party RTC channel has been successfully established.

[0166] S604-S607 are examples of the connection establishment process for three-party RTC channels.

[0167] S608, The channel management module of the main control device notifies the transmission module (here, the data sending and receiving module) in the self-developed P2P transmission channel to start the connection establishment for the self-developed 2P channel.

[0168] It should be noted that the two channels mentioned above in this application can start establishing a connection simultaneously, that is, steps S604 and S608 can start executing simultaneously, so as to save response time and establish the transmission channel as soon as possible.

[0169] S609, The transmission module of the main control device requests the establishment of a P2P control channel from the self-developed RTC cloud in the device cloud.

[0170] It should be understood that a P2P channel connection cannot be established directly. Instead, a control channel needs to be established first, followed by NAT detection and hole punching before the channel connection can be completed.

[0171] The S610 main control device's transmission module performs NAT detection to the self-developed RTC based on the STUN microservice.

[0172] The purpose of NAT detection results is mainly to obtain point-to-point information to facilitate subsequent hole punching, such as vendor information and network node information.

[0173] S611. Based on the P2P control channel created above, the transmission module of the master control device and the transmission module of the controlled device exchange information.

[0174] The information here refers to the information detected by NAT.

[0175] S612, the transmission module of the main control device performs NAT traversal based on P2P hole punching technology.

[0176] S608-S612 are examples of the connection establishment process for self-developed P2P channels.

[0177] Subsequently, a suitable target transmission channel is selected based on the P2P connection establishment results. S621-S629 are examples of the execution process for determining the self-developed P2P channel as the target transmission channel when the self-developed P2P channel connection is successfully established; S631-S638 are examples of the execution process for determining the third-party RTC channel as the target transmission channel when the self-developed P2P channel connection fails to establish.

[0178] S621, The transmission module of the main control device detected that the self-developed P2P channel was successfully established.

[0179] S622, The transmission module of the main control device notifies the channel management module of the main control device of the successful establishment of the self-developed P2P channel.

[0180] S623. Based on the received successful connection establishment result, the channel management module of the main control device notifies the transmission module (the transmission module in the third-party RTC transmission channel) to release the third-party RTC channel.

[0181] S624, The main control device's transmission module releases the third-party RTC channel.

[0182] S625, the third-party RTC module of the main control device releases the end-to-cloud connection between the third-party RTC cloud and the main control device.

[0183] S626, the third-party RTC cloud feedback release result is sent to the channel management module of the main control device.

[0184] S627. The channel management module of the main control device determines to enable the self-developed P2P channel. That is, the self-developed P2P channel is designated as the target transmission channel.

[0185] S628, the channel management module of the main control device will send an activation message to the audio and video call service to indicate the activation of the self-developed P2P channel.

[0186] S629, The audio and video call service of the main control device will further notify the remote control app of the above-mentioned activation message.

[0187] S631, The transmission module of the main control device (here, the data sending and receiving module in the self-developed P2P channel) detected a NAT traversal hole punching failure.

[0188] S632, The transmission module of the main control device notifies the channel management module of the main control device of the message that the self-developed P2P channel connection has failed.

[0189] S633. After receiving the above failure message, the channel management module of the main control device notifies the transmission module of the main control device (here, the data sending and receiving module in the self-developed P2P channel) of the resources that the self-developed P2P channel of the coastal defense has established.

[0190] S634, The main control device's transmission module releases the control channel of the self-developed P2P channel and releases the end-to-cloud connection with the self-developed RTC cloud.

[0191] S635, a self-developed RTC cloud feedback release result channel management module that sends the result to the main control device.

[0192] S636, The channel management module of the main control device determines to enable the third-party RTC channel. That is, the third-party RTC channel is designated as the target transmission channel.

[0193] S637, the channel management module of the main control device will send an activation message to the audio and video call service to enable the third-party RTC channel.

[0194] S638, the main control device's audio and video call service will further notify the remote control app of the above-mentioned activation message.

[0195] Figure 6 This mainly takes the transmission of audio and video data as an example. For the transmission of file data, it is only necessary to... Figure 6 The process involves replacing the third-party SDK with the intermediate address allocation module of the self-developed relay channel, and replacing the third-party RTC in the device cloud with the self-developed TURN cloud. The execution steps of the replaced modules are then adaptively adjusted. However, the selection process remains progressive. Simultaneously, connections are established between the self-developed relay channel and the self-developed P2P channel. If the self-developed P2P channel connection fails, the other channel is used as the target transmission channel. Further details are omitted here.

[0196] As mentioned above, transmission parameters and congestion control parameters can be negotiated (agreed) before data transmission, so that data transmission and congestion control can be performed based on the negotiated parameters during data transmission. The following section combines... Figure 7 Please provide an explanation.

[0197] Figure 7 This is a schematic flowchart illustrating a negotiation parameter and data transmission based on the negotiation parameter, according to an embodiment of this application. Figure 7 The explanation mainly uses audio and video data as an example, but it should be understood that the execution process is similar for file data, only the specific content of the agreed parameters is different, so it will not be elaborated on here.

[0198] S701, the remote control app of the main control device sends the negotiation parameters of audio and video data to the video call service of the main control device.

[0199] The negotiation parameters for audio and video data here can include transmission parameters and congestion control parameters. For video data, transmission parameters can include resolution, frame rate, and codec. For audio data, transmission parameters can include sample rate and number of channels. Congestion control parameters can include data size, initial window size, and congestion control strategy.

[0200] S702. The audio and video call service of the master device and the audio and video call service of the controlled device negotiate parameters and determine the parameters to be used in the future. These parameters can be called negotiation parameters (including transmission parameters and congestion control parameters).

[0201] S703, the audio and video call service of the controlled device sends the negotiated parameters to the remote control app of the controlled device.

[0202] Steps S701-S703 are examples of the parameter negotiation process. After the parameters are agreed upon, remote data transmission can begin.

[0203] During data transmission, audio and video data will be acquired, encoded, packaged, transmitted, unpacked, decoded, and rendered according to negotiated parameters. The main control device mainly participates in the transmission, unpacking, decoding, and rendering processes, while the controlled device mainly participates in the acquisition, encoding, packaging, and transmission processes. The two devices will also perform congestion control according to negotiated parameters.

[0204] Congestion control strategies may include slow start plus congestion avoidance strategies and fast recovery strategies. Furthermore, they may include strategies for varying the congestion window size, such as linear or exponential increases. In this application, different congestion control strategies can also be selected based on data type, channel type, and the idle state of the receiver's buffer window.

[0205] S704. The remote control app of the controlled device notifies the audio and video call service of the controlled device to start transmitting the video data of the proposal based on the negotiated parameters.

[0206] S705, The audio and video call service of the controlled device collects, encodes and packages audio and video data based on negotiated parameters.

[0207] S706, The controlled device's audio and video call service sends the packaged audio and video data step by step to the master device's audio and video call service through the target transmission channel.

[0208] S707, the main control device's audio and video call service unpacks, decodes, and renders the received audio and video data to obtain a data format that can be presented to the user.

[0209] S708: The remote control app of the main control device obtains the processed audio and video data from the audio and video call service of the main control device through a callback and presents it to the controlled device.

[0210] For example, audio data is played using the audio output module of the controlled device, while video data is displayed on the screen of the controlled device.

[0211] Figure 8 This is a schematic diagram of the window value change strategy for two congestion control strategies in embodiments of this application. Figure 8 Both congestion control strategies adjust the window size in units of frames (audio and video data) or file blocks (file data). In other words, the window size will be a multiple of the frame or file block size. Figure 8 The horizontal axis of the coordinate system represents the number of transmission rounds, and the vertical axis represents the size of the window value.

[0212] like Figure 8 Image (a) shows an example of a slow start plus congestion avoidance strategy, in which each adjustment period includes a slow start interval and a congestion avoidance interval, initially ( Figure 8Position 1 in (a) will first undergo a slow start, so the window size will be gradually increased exponentially until a certain size is reached. Figure 8 In (a) at position 2, to avoid congestion caused by continuous exponential growth, the rate of window increase is slowed down and changed to linear increase, that is, entering the congestion avoidance interval. When another size threshold is reached ( Figure 8 (position 3 in (a)) restarts from the initial size ( Figure 8 In (a), position 4) is adjusted according to the above method of first increasing exponentially and then linearly. It can be seen that the new inflection point ( Figure 8 The window size at position 5 in (a) is not the same as the window size at the turning point of the previous cycle. Figure 8 As can be seen from (a), this application is not a rigid adjustment process that remains unchanged after a preliminary agreement, but rather an adjustment strategy that is dynamically adjusted according to the actual situation.

[0213] like Figure 8 Image (b) shows an example of a fast recovery strategy. In this strategy, the first adjustment period includes a slow start interval and a congestion avoidance interval, while the second adjustment period does not include the slow start interval and starts directly from the congestion avoidance interval. Initially ( Figure 8 Position 1 in (b) will first undergo a slow start, so the window size will be gradually increased exponentially until a certain size is reached. Figure 8 In section (b), position 2) will slow down the rate of window increase to avoid congestion caused by continuous exponential growth, instead increasing linearly, thus entering the congestion avoidance interval. When another size threshold is reached ( Figure 8 If the position is numbered 3 in (b), then from a new initial size ( Figure 8 In (b), position 4) adjusts the window size according to the above method of first increasing exponentially and then linearly. It can be seen that the new initial window size in the next cycle ( Figure 8 Position 4 in (b) is not the same as the initial window size of the previous cycle. This fast recovery means that when reducing the window size, it doesn't start directly from the initial size, but uses a window size larger than the initial size as the initial value for the next cycle. It should also be understood that this new initial window size is different from the turning point of the previous cycle (…). Figure 8 The window size at position 2 in (b) is also different, meaning it may not be the same as the initial window size of the congestion avoidance interval in the previous cycle. Figure 8As can also be seen in (b), this application is not a rigid adjustment process that remains unchanged after a preliminary agreement, but rather an adjustment strategy that is dynamically adjusted according to the actual situation. Figure 8 The main difference between the two strategies lies in whether the initial window size of the next cycle is the same as the initial window size of the previous cycle, and the fact that the fast recovery strategy does not include a slow start interval in every cycle.

[0214] Figure 9 This is a schematic flowchart of a congestion control process according to an embodiment of this application. Figure 9 This mainly takes congestion control during the transmission of audio and video data as an example. If it is file data, it is only necessary to replace the data frames with file blocks. Figure 9 The transmitting end can be the controlled device, and the receiving end can be the master control device.

[0215] S901, The congestion control module at the transmitting end and the congestion control module at the receiving end negotiate the congestion control strategy and related parameters.

[0216] This refers to the negotiation (agreement) of congestion control parameters mentioned above.

[0217] S902, The data acquisition module at the sending end triggers the service, that is, it triggers the remote data transmission service.

[0218] S903, The data acquisition module of the transmitting end notifies the channel decision module of the transmitting end to select the target transmission channel.

[0219] S904. The channel decision module of the sending end notifies the data caching module of the sending end to cache the data to be sent.

[0220] The data to be sent can be understood as the data to be shared mentioned above, which can be audio / video data or file data. Figure 9 This mainly takes audio and video data as an example. When it is audio and video data, it is cached in units of frames; when it is video data, it is cached in units of file blocks.

[0221] S905, the data buffer module of the sending end sends parameter information such as data packet size and window value to the congestion control module of the sending end.

[0222] S906, The data caching module at the sending end sends the cached data to the data sending module at the sending end.

[0223] The cached data here can be understood as an example of the data collected and packaged as described above.

[0224] S907, The data sending module of the sending end transmits data to the data receiving module of the receiving end.

[0225] S908, the receiving end's data receiving module forwards the received data to the receiving end's data buffer module for buffering.

[0226] S909, the data buffering module of the receiving end feeds back the buffering result to the data receiving module of the receiving end and the data receiving module of the sending end.

[0227] Steps S902-S909 illustrate the normal data transmission process.

[0228] Both S910 and S911 are data buffer modules at the sending end that send the data to be transmitted to the data sending module at the sending end.

[0229] If packet loss (data packet / data frame loss) occurs when data is sent in step S910, but no packet loss occurs when data is sent in step S911, then the data without packet loss will continue to be transmitted and buffered.

[0230] S912, The data sending module of the sending end transmits data to the data receiving module of the receiving end.

[0231] S913, The receiving end's data receiving module forwards the received data to the receiving end's data buffer module for buffering.

[0232] S914. The data buffer module of the receiving end performs frame loss detection, and executes step S915 after detecting frame loss.

[0233] S915, the data buffer module at the receiving end feeds back the frame loss information to the congestion control module at the receiving end.

[0234] To facilitate understanding of the scheme in this application, the 20th to 29th frames are taken as the example of frame loss. However, it should be understood that this value is only for the convenience of understanding the scheme and is not a limitation.

[0235] S916, The congestion control module at the receiving end gradually sends frame loss information to the congestion control module at the sending end, and further informs the data buffer module at the sending end through the congestion control module at the sending end.

[0236] S917, The data buffer module at the sending end repackages the data of the lost frames based on the received lost frame information.

[0237] Assuming that the frames that are lost as described above are frames 20-29, then the data repackaged here will be the data of frames 20-29.

[0238] S918, The data caching module of the sending end sends the repackaged data to the data sending module of the sending end.

[0239] S919, the data sending module of the sending end performs data retransmission, and transmits the repackaged data to the data receiving module of the receiving end again.

[0240] S920: The receiving end's data receiving module sends the received retransmitted data to the receiving end's data buffer module for re-buffering.

[0241] Steps S910-S920 are examples of the execution process for retransmission after frame loss.

[0242] In this application's scheme, congestion control can be performed by devices on both sides, rather than only by devices on one side. The following will explain... Figure 9 To continue explaining, Figure 9 Steps S921-S928 are illustrative procedures for the receiving end to actively perform congestion control, while steps S931-S942 are illustrative procedures for the sending end to perform congestion control.

[0243] S921. The data buffer module of the receiving end obtains the size of the buffer free window and reports it to the congestion control module.

[0244] For ease of understanding, we can assume that the size of the free window here is 60.

[0245] S922, The congestion control module at the receiving end re-determines the congestion control strategy and window size adjustment strategy based on the buffer free window size, the current data type (e.g., audio and video data), and the type of the target transmission channel, and can adjust the packet size and window size.

[0246] For ease of understanding, let's assume the original package size is 10, which is adjusted to 20, and the original window size is 20, which is adjusted to 40.

[0247] S923. The congestion control module at the receiving end sends the updated parameters to the congestion control module at the sending end.

[0248] S924. The congestion control module at the sending end updates the congestion control parameters based on the received information.

[0249] S925, The congestion control module of the sending end notifies the data buffer module of the sending end of the updated parameters.

[0250] S926, The data caching module at the sending end packages the subsequent data according to the updated parameters.

[0251] S927, The data caching module of the sending end sends the cached data to the data sending module of the sending end.

[0252] S928, The data sending module of the sending end forwards the data to the data receiving module of the receiving end for caching via the data receiving module of the receiving end.

[0253] S931, The data buffer module of the sending end sends the data to be transmitted to the data sending module of the sending end.

[0254] S932, The data sending module of the sending end transmits data to the data receiving module of the receiving end.

[0255] S933, The receiving end's data receiving module forwards the received data to the receiving end's data buffer module for buffering.

[0256] S934, The data acquisition module of the receiving end requests to read the cached data from the data cache module of the receiving end.

[0257] S935, the data acquisition module at the receiving end processes and renders the cached data it reads.

[0258] S936. The data buffer module at the receiving end performs congestion detection. When congestion is detected, step S937 is executed to inform the congestion control module at the receiving end, and then gradually informs the congestion control module at the sending end.

[0259] S938, The congestion control module at the sending end redetermines the congestion control strategy and the strategy for adjusting the window size.

[0260] The method for redetermining the strategy can be found in the above text and will not be repeated here.

[0261] S939. The congestion control module of the sending end notifies the data buffer module of the sending end of the updated congestion control parameters.

[0262] S940, the data caching module at the sending end packages the subsequent data according to the updated parameters.

[0263] S941, The data buffer module of the sending end sends the buffered data to the data sending module of the sending end.

[0264] S942, The data sending module of the sending end forwards the data to the data receiving module of the receiving end for caching.

[0265] The methods of the embodiments of this application have been described above with reference to the accompanying drawings. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially, these steps are not necessarily executed in the order shown in the figures. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps. The apparatus of the embodiments of this application will now be described with reference to the accompanying drawings.

[0266] Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Figure 10 The electronic devices in this context can be either master control devices or controlled devices. For example... Figure 10 As shown, the electronic device 100 may include a processor 110, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

[0267] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0268] For example, Figure 10 The processor 110 shown may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0269] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0270] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0271] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0272] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0273] Mobile communication module 150 can provide wireless communication solutions for use in electronic device 100. Wireless communication module 160 can provide wireless communication solutions for use in electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and long-range data transmission technologies. Wireless communication module 160 can be one or more devices integrating at least one communication processing module. Wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 110. Wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0274] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 of the electronic device 100 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other electronic devices through wireless communication technology.

[0275] In this application, a remote data transmission channel needs to be established between the two devices through a wireless communication module.

[0276] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0277] In this embodiment of the application, it is necessary to use a display screen to show the running interface of the remote control app and the real-time screen image of the controlled device.

[0278] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs.

[0279] In this application, the controlled device needs to encode the audio and video data, while the main control device needs to decode the audio and video data, which can be done using the aforementioned video codec.

[0280] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0281] In this application, the audio module of the controlled device is mainly used for audio data acquisition and encoding, and the audio module of the main control device is used for audio data decoding.

[0282] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0283] This application can be used not only for touch screen devices but also for non-touch screen devices, and there are no limitations. Therefore, the input module of the electronic device 100 can also include a keyboard, mouse, etc.

[0284] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0285] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0286] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0287] This application also provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the steps in any of the above methods.

[0288] This application also provides a computer-readable storage medium storing instructions that, when executed by an electronic device, can implement any of the methods described above. This computer-readable medium may include at least: any entity or device capable of carrying computer program code (instructions) to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0289] This application also provides a computer program product, which includes a computer program that, when executed by an electronic device, can implement any of the above-described methods. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form.

[0290] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0291] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0292] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0293] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0294] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0295] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0296] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0297] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0298] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for remote data sharing, characterized in that, include: The master control device determines a first data type of the data to be shared, wherein the first data type is audio / video data or file data, the audio / video data includes at least one of the screen display data of the controlled device, real-time audio data or control commands, the file data is a file already stored in the controlled device, and the data to be shared is used to represent the shared data that the controlled device is to transmit to the master control device during the remote collaborative work between the master control device and the controlled device. The master control device is used for remote control of the controlled device; The main control device selects a target transmission channel from multiple remote data transmission channels based on the first data type; The master control device receives the data to be shared from the controlled device through the target transmission channel.

2. The method according to claim 1, characterized in that, The multiple remote data transmission channels include a real-time communication (RTC) channel, a relay channel, and a point-to-point (P2P) channel. The RTC channel is a general-purpose remote transmission channel. The relay channel is a dedicated remote transmission channel of the main control device that needs to be forwarded by the relay server corresponding to the main control device. The P2P channel is a dedicated remote transmission channel of the main control device that needs to be created using P2P hole punching technology. The step of selecting a target transmission channel from multiple remote data transmission channels based on the first data type includes: When the first data type is audio / video data, the target transmission channel is selected from the RTC channel and the P2P channel; or, When the first data type is file data, the target transmission channel is selected from the relay channel and the P2P channel.

3. The method according to claim 2, characterized in that, When the first data type is audio / video data, selecting the target transmission channel from the RTC channel and the P2P channel includes: Check whether the P2P channel has been successfully established; If the P2P channel is successfully established, select the P2P channel as the target transmission channel; or, If the P2P channel fails to be established, the RTC channel is selected as the target transmission channel.

4. The method according to claim 2, characterized in that, When the first data type is file data, selecting the target transmission channel from the relay channel and the P2P channel includes: Check whether the P2P channel has been successfully established; If the P2P channel is successfully established, select the P2P channel as the target transmission channel; or, If the P2P channel fails to be established, the relay channel is selected as the target transmission channel.

5. The method according to any one of claims 1 to 4, characterized in that, The master control device receives the data to be shared from the controlled device through the target transmission channel, including: The master control device sends transmission parameters to the controlled device, and the transmission parameters include at least one of the following: video frame rate, resolution, codec type, audio sampling rate, number of channels, and file block size. The controlled device sends the data to be shared to the master control device based on the transmission parameters; The master control device receives the data to be shared through the target transmission channel.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The master control device sends congestion control parameters to the controlled device. The congestion control parameters include at least one of the following: initial size of data packets, initial size of window value, maximum window value, congestion control strategy, and congestion window size adjustment strategy. The congestion control strategy includes at least one of the following: slow start plus congestion avoidance strategy and fast recovery strategy. The congestion window size adjustment strategy includes at least one of the following: linear increase and exponential increase. During the period when the master control device receives the data to be shared from the controlled device through the target transmission channel, the master control device and the controlled device perform congestion control based on the congestion control parameters.

7. The method according to claim 6, characterized in that, When the service type of the data to be shared is audio or video data, the window value is set in units of frames; when the service type of the data to be shared is file data, the window value is set in units of file blocks. The file block is used to represent any file block among the multiple file blocks into which each file to be transmitted is divided. Each file block includes multiple data packets, and the data packet is the smallest unit of data transmission.

8. The method according to claim 6, characterized in that, During the process of the master control device receiving the data to be shared from the controlled device through the target transmission channel, the master control device and the controlled device perform congestion control based on the congestion control parameters, including: Based on the first data type, the target transmission channel, and the size of the buffer free window, select the target congestion control strategy and the target adjustment strategy from the congestion control strategy and / or the congestion window size adjustment strategy; Congestion control is performed based on the target congestion control strategy and the target adjustment strategy.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: A first application runs on the master control device and the controlled device, the first application being a remote control application; The second application on the controlled device can be remotely controlled through the first application on the main control device. The second application is any application on the controlled device other than the first application. The controlled device transmits the data generated in real time by the second application when it is remotely controlled to the master control device as the data to be shared.

10. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to perform the steps performed by the master device in the method as described in any one of claims 1 to 9, or to perform the steps performed by the controlled device in the method as described in any one of claims 1 to 9.

11. A system for remote data sharing, characterized in that, The system includes a master control device and a controlled device. The master control device is used to remotely control the controlled device. The master control device is capable of executing the steps performed by the master control device in the method of any one of claims 1 to 9. The controlled device is capable of executing the steps performed by the controlled device in the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 9.

Images