Multi-screen sharing and interaction control system and method

Abstract

The invention relates to a multimedia data processing technology, and discloses a multi-screen sharing and interactive control system and a multi-screen sharing and interactive control method, which are used for realizing full-screen sharing of media contents, supporting real-time multi-point touch interaction and reverse control functions and improving user experience. The multi-screen sharing and interaction control method comprises the following steps: S1, establishing connection between receiving equipment and media source equipment; s2, the media source device receives a media playing request from the receiving device; s3, the media source device plays media, captures screen playing content in real time, encodes the screen playing content and sends the encoded screen playing content to all the connected receiving devices at the same time; s4, each receiving device receives the sending data from the media source device, decodes the sending data and synchronously plays the sending data on a screen; s5, after the receiving device captures the touch input of the user, generating a corresponding interaction control instruction and sending the interaction control instruction to the media source device; and S6, after receiving the interaction control instruction, the media source device responds to the instruction and synchronizes the instruction to each receiving device.

Description

Technical Field

[0001] The present invention relates to multimedia data processing technology, and in particular to a multi-screen sharing and interactive control system and method. Background Art

[0002] With the popularization of multimedia content consumption and the use of smart devices, users have put forward higher requirements for seamless interaction and content sharing between devices. Traditional screen sharing technologies are mainly based on wired connections or wireless transmission limited to a single protocol. These methods usually only support one-way data flow, that is, simply transmitting audio and video data from the media source device to the receiving device. In addition, these technologies often do not support interactive control including touch operation, or are limited to very basic remote control functions, failing to fully utilize the advanced input functions of modern smart devices such as multi-touch and dynamic gesture recognition.

[0003] Existing wireless transmission solutions such as Miracast or AirPlay can achieve media sharing between different devices, but they are mostly limited by device compatibility and network conditions, and cannot implement complex reverse control logic, such as screen operation feedback or advanced command transmission. These limitations reduce operational flexibility and user experience, especially in environments that require high levels of interaction and collaboration. Summary of the invention

[0004] The technical problem to be solved by the present invention is to propose a multi-screen sharing and interactive control system and method, which can realize full-screen sharing of media content while supporting real-time multi-touch interaction and reverse control functions to improve user experience.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] In one aspect, the present invention provides a multi-screen sharing and interactive control system, including a media source device and a plurality of receiving devices connected to the same wireless local area network;

[0007] The media source device is configured with a screen content capturing module, an encoding module, an encryption module and a communication module;

[0008] The screen content capture module is used to capture the playing content on the current screen;

[0009] The encoding module is used to encode the captured playing content on the current screen into a data stream;

[0010] The encryption module is used to encrypt the encoded data stream;

[0011] The communication module is used to send the encrypted data stream to each receiving device that establishes a connection;

[0012] The receiving device is configured with a decryption module, a decoder module and a control module;

[0013] The decryption module is used to decrypt the encrypted data received from the media source device;

[0014] The decoder module is used to decode and play the decrypted data;

[0015] The control module is used to capture the touch input of the user on the screen of the receiving device, and generate corresponding interactive control instructions for the media source device and send them to the media source device.

[0016] Furthermore, the communication module supports multiple protocol selections, including but not limited to: Wi-Fi Direct, Miracast, AirPlay, DLNA or VNC protocols.

[0017] Furthermore, the encoding module uses an H.264 / H.265 encoder to convert the captured screen content into a compressed video data stream in real time, and uses an AAC encoder to encode the captured audio data.

[0018] Furthermore, the receiving device is a smart device with a screen and supporting multi-touch technology that can use different operating systems, including but not limited to: smart TVs, tablet computers or smart phones; the different operating systems include but are not limited to: Windows, macOS, iOS or Android systems.

[0019] Furthermore, the media source device also includes an intelligent network adaptation unit responsible for real-time monitoring of network conditions and dynamically adjusting encoding and decoding parameters and transmission strategies.

[0020] On the other hand, based on the above system, the present invention also provides a multi-screen sharing and interactive control method, which includes the following steps:

[0021] S1. The receiving device establishes a connection with the media source device;

[0022] S2, the media source device receives a media playback request from the receiving device;

[0023] S3, the media source device plays the media and captures the screen playback content in real time, encodes it and sends it to all connected receiving devices at the same time;

[0024] S4, each receiving device receives the data sent from the media source device, decodes it and plays it synchronously on the screen;

[0025] S5, when the receiving device captures the user's touch input, it generates a corresponding interactive control instruction and sends it to the media source device;

[0026] S6. After receiving the interactive control instruction, the media source device responds to the instruction and synchronizes to each receiving device.

[0027] Furthermore, in step S1, the manner in which the receiving device establishes a connection with the media source device includes:

[0028] The media source device automatically detects compatible receiving devices in the wireless LAN and registers itself as a discoverable service;

[0029] The receiving device is paired and authenticated with the media source device through a secure protocol;

[0030] The receiving device and the media source device establish a stable connection through a wireless network.

[0031] Furthermore, step S2 also includes: the user selects available media resources provided by the media source device through an application on any receiving device, and after selecting the corresponding media resource, generates a related media play request and sends it to the media source device.

[0032] Furthermore, in step S4, the method of controlling synchronous playback includes: maintaining the consistency of the picture played across devices through frame synchronization technology of different receiving devices; using timestamp and calibration mechanism to adjust the time difference displayed by each device to compensate for the difference in network delay and processing time.

[0033] Furthermore, in step S5, the interactive control instructions include but are not limited to: pause, play, adjust volume, turn pages or select content, etc.

[0034] The beneficial effects of the present invention are:

[0035] (1) Based on the sharing and interaction solution provided by the present invention, not only high-quality full-screen content sharing is achieved, but also real-time reverse control from the receiving device to the sending device is supported, thereby improving the flexibility and efficiency of multi-device interaction.

[0036] (2) The present invention adopts the most advanced H.264 and H.265 encoding and decoding technologies, optimizes the transmission efficiency and quality of video content, and can efficiently encode the screenshots of the real-time captured sending device, ensuring high-definition video output and smooth and accurate display of the real-time screen status of the sending end even in bandwidth-constrained environments.

[0037] (3) By introducing multi-touch technology and complex user interaction logic, multiple users are allowed to interact on their devices at the same time. This multi-touch interaction mode significantly improves the ability of collaboration and participation.

[0038] (4) The system of the present invention can dynamically adjust the encoding parameters and transmission strategies according to the network conditions, ensuring the best transmission effect and the minimum delay in different network environments.

[0039] (5) The system of the present invention is compatible with a variety of operating systems and hardware platforms, so that the system can be widely used in various smart devices, from smart phones to large-screen smart TVs, without the need for specific hardware or software restrictions.

[0040] (6) The solution of the present invention has a wide range of market application scenarios, such as:

[0041] Home entertainment market: The multi-screen interaction and control functions provided by the present invention perfectly meet the trend of modern smart homes;

[0042] Education: The present invention can be used in classroom teaching to provide a more interactive learning experience and support real-time multimedia sharing of distance education and online courses.

[0043] Business application areas: Conferences and presentations can use the solution of the present invention to share content and control conferences, which is particularly suitable for multi-location business meetings and remote working modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the multi-screen sharing and interactive control process in the present invention. DETAILED DESCRIPTION

[0045] The present invention aims to design a solution that can realize full-screen content sharing between multiple smart devices with screens while supporting real-time multi-touch interaction and reverse control functions, thereby improving the efficiency and quality of multimedia content transmission and enhancing the user interaction experience. The specific technical objectives of the design scheme of the present invention include:

[0046] Full-screen content sharing: Enables the ability to wirelessly transmit full screen content, including high-definition video, images, and audio, from one or more source devices to multiple receiving devices. This feature should support a variety of resolutions and formats to ensure consistent and high-quality content display on different devices.

[0047] Real-time interactivity: Ensure that all shared content is not only static but also interactive in real time on the receiving device. Users should be able to modify or operate shared content directly on the receiving end through touch operations, such as annotation, drawing, or other forms of editing.

[0048] Multi-touch support: The system needs to support modern multi-touch technology so that multiple users can control and interact with shared content on their devices at the same time, improving collaboration efficiency and participation.

[0049] Reverse control function: The receiving device can not only display the content, but also send control commands to the media source device to achieve real-time modification or control of the source content, such as playback control, volume adjustment, content selection, etc. This two-way control mechanism increases the flexibility of operation and the breadth of application.

[0050] Cross-platform compatibility and efficient codec: The system should be compatible with multiple operating systems and device types, and use efficient H.264 / H.265 codec technology to optimize network bandwidth usage and minimize transmission delays.

[0051] Based on the above technical objectives, the multi-screen sharing and interactive control system provided by the present invention includes a media source device and multiple receiving devices connected to the same wireless local area network;

[0052] The media source device is configured with a screen content capturing module, an encoding module, an encryption module and a communication module;

[0053] The screen content capture module is used to capture the playing content on the current screen;

[0054] The encoding module is used to encode the captured playing content on the current screen into a data stream;

[0055] The encryption module is used to encrypt the encoded data stream;

[0056] The communication module is used to send the encrypted data stream to each receiving device that establishes a connection;

[0057] The receiving device is configured with a decryption module, a decoder module and a control module;

[0058] The decryption module is used to decrypt the encrypted data received from the media source device;

[0059] The decoder module is used to decode and play the decrypted data;

[0060] The control module is used to capture the touch input of the user on the screen of the receiving device, and generate corresponding interactive control instructions for the media source device and send them to the media source device.

[0061] The multi-screen sharing and interactive control process realized by this system can be found in Figure 1The media source device as a sending device obtains the resource stream from the media library, encodes the resource stream and encrypts it through the encryption module as a security module, and then sends the encrypted data stream to each receiving device. Each receiving device can decrypt the encrypted data stream and then perform synchronous decoding and playback. The receiving device is also configured with a control module, which captures the user's touch input on the screen of the receiving device, generates corresponding interactive control instructions for the media source device, and sends them to the media source device.

[0062] Based on the above system, the specific implementation method for realizing multi-screen sharing and interaction includes the following parts:

[0063] 1. Initialization phase

[0064] (1) Configure the sending device: Install the necessary software on the sending device and configure its network settings to ensure that the devices can connect to the same local area network. This software is responsible for encoding the screen content and managing network communications.

[0065] (2) Automatic discovery and registration: The sending software uses a service discovery protocol (such as mDNS or SSDP) to automatically detect the receiving device in the network and register the sending device as a discoverable service so that the receiving device can easily connect.

[0066] 2. Content Capture and Encoding

[0067] (1) Screen content capture: The sending device continuously captures the screen output, including all dynamic images and audio output.

[0068] (2) Real-time encoding processing: Use the H.264 / H.265 encoder to convert the captured screen content into a compressed video data stream in real time, and use the AAC encoder to encode the captured audio data to optimize transmission efficiency and maintain content quality.

[0069] 3. Data transmission

[0070] (1) Establishing a connection: A stable connection is established between the sender and the receiver via Wi-Fi. According to the selected protocol (Miracast, AirPlay, etc.), necessary handshakes and security authentication are performed.

[0071] (2) Data stream transmission: The encoded data stream is sent to all connected receiving devices through the wireless network. The adaptive flow control mechanism is used to adjust the data transmission rate to adapt to changes in network conditions.

[0072] 4. Content Display and Interaction

[0073] (1) Content decoding and display: After receiving the data stream, the receiving device decodes it and displays it in full screen on the device screen.

[0074] (2) Interaction capture: The receiving device captures all operations performed by the user through the touch screen or other input devices (such as mouse and keyboard).

[0075] 5. Reverse Control

[0076] (1) Command feedback: The receiving device converts the user's operation commands (such as pause, play, adjust volume, turn pages, select content, etc.) into control signals and feeds them back to the sending device through the network.

[0077] (2) Response execution: After receiving the control signal, the sending device adjusts the output content or performs related operations according to the command, such as changing the playback status or adjusting the settings.

[0078] 6. Monitoring and Optimization

[0079] (1) Performance monitoring: The system continuously monitors transmission quality and device response to identify any potential problems or performance bottlenecks.

[0080] (2) Dynamic adjustment: Based on the monitoring results, the system dynamically adjusts the codec parameters and network transmission settings to ensure the best user experience and system performance.

[0081] Example

[0082] Taking the application of the design scheme of the present invention in a home environment as an example, family members want to watch the same film and television content on various devices and have the ability to control the playback from any device. These devices include the smart TV in the living room, the tablet computer in the kitchen, the personal computer in the bedroom, and the smart phones of family members.

[0083] According to the system configuration scheme of the present invention, the smart set-top box in the home is used as the media source device, responsible for the storage and streaming of the media, and the smart TV, tablet computer, personal computer and various mobile devices (such as smart phones) in the home are used as receiving devices.

[0084] The implementation process is as follows:

[0085] S1. System initialization

[0086] a. Configure the smart set-top box and make sure it is connected to your home wireless network.

[0087] b. Detect all compatible receiving devices in the network, including smart TVs, tablets, PCs and smartphones.

[0088] S2. Receiver device registration and authentication

[0089] a. Pairing and authentication with the smart set-top box through security protocols to ensure network security and correct connection of devices.

[0090] b. Each device registers its device type and supported media formats in the system.

[0091] S3. Content selection and command issuance

[0092] a. The user selects media content through the application on any receiving device. The application displays all available media resources, and the user can browse and select the video or audio file they want to play.

[0093] b. After the user selects a file from the list, the device sends the relevant command (including information about the selected media file) to the smart set-top box.

[0094] S4, media processing and streaming

[0095] a. After receiving the play command, the smart set-top box retrieves the specified media file from the storage.

[0096] b. The set-top box uses an H.264 or H.265 encoder to encode video files in real time and convert them into a format suitable for wireless transmission.

[0097] H.264 data is specially processed as follows:

[0098] Each H.264 encoded image data block sent is concatenated with the encoder's configuration information data before being sent to ensure that each H.264 data segment sent starts with the encoder's configuration information data. This approach can achieve cross-platform compatibility for the application because different chip platforms' media encoders generate different configuration information with the same parameters.

[0099] c. The audio data is encoded in real time using the AAC encoder and converted into a format suitable for wireless transmission.

[0100] d. The encoded data stream is sent to all authenticated receiving devices simultaneously via the home wireless network.

[0101] S5. Playback and interactive control

[0102] a. After receiving the video stream, the receiving device starts to display the video content in full screen on the user interface.

[0103] b. Users can control playback on any receiving device, such as pausing, playing, adjusting volume, etc. In addition, users can also operate and control media devices on any receiving device, such as turning pages online, turning pages left and right, selecting, returning, etc.

[0104] The implementation method for the receiving device of the Android system is to implement the processing logic of the sliding event through the general touch screen monitoring of the Android platform, and send the remote control control command that the set-top box can respond to according to the calculated sliding distance.

[0105] c. The device feeds back the control instructions to the smart set-top box in real time. The smart set-top box performs corresponding actions according to the instructions and synchronizes them to all other receiving devices to maintain the consistency of the playback status.

[0106] Based on the implementation scheme of the above embodiment, all receiving devices can display synchronized video content in full screen and can independently control playback, which greatly increases the flexibility of viewing; and by applying efficient encoding and decoding technology, high definition and smoothness can be maintained even when playing on multiple devices simultaneously; in addition, transmission security is ensured by encrypting the video stream, and the system is compatible with devices of various brands and models, including widely used mobile devices.

[0107] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.

Claims

1. A multi-screen sharing and interactive control system, comprising a media source device and multiple receiving devices connected to the same wireless local area network; characterized in that: The media source device is configured with a screen content capturing module, an encoding module, an encryption module and a communication module; The screen content capture module is used to capture the playing content on the current screen; The encoding module is used to encode the captured playing content on the current screen into a data stream; The encryption module is used to encrypt the encoded data stream; The communication module is used to send the encrypted data stream to each receiving device that establishes a connection; The receiving device is configured with a decryption module, a decoder module and a control module; The decryption module is used to decrypt the encrypted data received from the media source device; The decoder module is used to decode and play the decrypted data; The control module is used to capture the touch input of the user on the screen of the receiving device, and generate corresponding interactive control instructions for the media source device and send them to the media source device.

2. A multi-screen sharing and interactive control system as claimed in claim 1, characterized in that: The communication module supports multiple protocol options, including: Wi-Fi Direct, Miracast, AirPlay, DLNA or VNC protocol.

3. A multi-screen sharing and interactive control system as claimed in claim 1, characterized in that: The encoding module uses an H.264 / H.265 encoder to convert the captured screen content into a compressed video data stream in real time, and uses an AAC encoder to encode the captured audio data.

4. A multi-screen sharing and interactive control system as claimed in claim 1, characterized in that: The receiving device is a smart device with a screen and supporting multi-touch technology that can use different operating systems, including: a smart TV, a tablet computer or a smart phone; the different operating systems include: Windows, macOS, iOS or Android systems.

5. A multi-screen sharing and interactive control system according to any one of claims 1 to 4, characterized in that: The media source device also includes an intelligent network adaptation unit, which is responsible for real-time monitoring of network conditions and dynamically adjusting encoding and decoding parameters and transmission strategies.

6. A multi-screen sharing and interactive control method, applied to the multi-screen sharing and interactive control system according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: S1. The receiving device establishes a connection with the media source device; S2, the media source device receives a media playback request from the receiving device; S3, the media source device plays the media and captures the screen playback content in real time, encodes it and sends it to all connected receiving devices at the same time; S4, each receiving device receives the data sent from the media source device, decodes it and plays it synchronously on the screen; S5, when the receiving device captures the user's touch input, it generates a corresponding interactive control instruction and sends it to the media source device; S6. After receiving the interactive control instruction, the media source device responds to the instruction and synchronizes to each receiving device.

7. A multi-screen sharing and interactive control method as claimed in claim 6, characterized in that: In step S1, the manner in which the receiving device establishes a connection with the media source device includes: The media source device automatically detects compatible receiving devices in the wireless LAN and registers itself as a discoverable service; The receiving device is paired and authenticated with the media source device through a secure protocol; The receiving device and the media source device establish a stable connection through a wireless network.

8. A multi-screen sharing and interactive control method as claimed in claim 6, characterized in that: Step S2 also includes: The user selects available media resources provided by the media source device through an application on any receiving device, and after selecting the corresponding media resources, generates a related media playback request and sends it to the media source device.

9. A multi-screen sharing and interactive control method as claimed in claim 6, characterized in that: In step S4, the method of controlling synchronous playback includes: maintaining the consistency of the picture played across devices through frame synchronization technology of different receiving devices; using timestamps and calibration mechanisms to adjust the time difference displayed by each device to compensate for differences in network delays and processing time.

10. A multi-screen sharing and interactive control method according to any one of claims 6 to 9, characterized in that: In step S5, the interactive control instructions include: pause, play, adjust volume, turn pages or select content, etc.

Images