Remote device screen projection interaction system

Abstract

The application discloses a remote device screen projection interaction system, and relates to the technical field of interaction control. The system comprises a remote execution device, a control relay server, a mobile relay terminal and a display terminal. The remote execution device collects and encodes screen audio and video data, establishes a long connection with the control relay server and executes control instructions. The control relay server manages sessions, generates connection information, relays data and instructions. The mobile relay terminal scans a code to establish a two-way connection, collects near-end voice instructions and converts the voice instructions into structured control instructions, and relays audio and video data streams. The display terminal displays connection information and screen projection content. The scheme realizes convenient screen projection and natural interaction across networks through mobile relay and voice control, and improves remote collaboration experience.

Description

Technical Field

[0001] This application relates to the field of interactive control technology, and in particular to a remote device projection interactive system. Background Technology

[0002] In scenarios such as cross-regional collaborative work, remote presentations, and remote teaching, users have an increasingly urgent need for screen projection and interaction on remote devices. Traditional screen projection technologies, such as Miracast and AirPlay, are limited by local area network environments and cannot directly achieve cross-internet remote screen projection. While remote desktop software, such as TeamViewer and AnyDesk, supports cross-network connections, they require the installation of dedicated software on both devices and complex network configurations. Furthermore, they are difficult to connect to large display devices such as conference room screens, making the operation threshold high for ordinary users.

[0003] In terms of control methods, existing solutions mostly rely on manual operation, such as mouse clicks and keyboard input to control remote devices. Users need to frequently operate the terminal when presenting in their local space, interrupting the smoothness of the presentation and failing to achieve hands-free natural interaction. At the same time, existing solutions lack efficient channels to transmit the intentions of nearby users. Users operating devices in one physical space while operating devices in another physical space have significant experience gaps. In addition, remote devices are usually behind private networks (NAT), and there are technical obstacles to discovering and establishing connections directly on the public network. Ordinary users cannot configure network penetration parameters themselves, resulting in poor stability and low success rate of cross-network projection connections. These problems make the cross-space projection process cumbersome, the interaction unnatural, and the network adaptation difficult, which seriously affects the efficiency of remote collaboration and user experience. There is an urgent need for a solution that can easily initiate cross-network projection and supports natural interaction control at the near end. Summary of the Invention

[0004] This application provides a remote device screen projection and interaction system. Through the interaction path of remote execution device, control relay server, mobile relay terminal and display terminal, it realizes convenient screen projection across networks and near-end voice control, reducing the operation threshold and improving interaction efficiency.

[0005] In a first aspect, embodiments of this application provide a remote device screen projection and interaction system, including a remote execution device, a control relay server, a mobile relay terminal, and a display terminal; The remote execution device is used to collect local audio and video data streams for human-computer interaction and establish a long connection with the control relay server. The control relay server is used to manage the sessions of all online devices, generate connection information for the remote execution device, and relay the audio and video data streams through the long connection. The mobile relay terminal is used to establish a session with the remote execution device based on the connection information, and to establish a local wireless projection connection with the display terminal. It also receives the audio and video data stream from the remote execution device with which the session exists, and sends it to the display terminal for presentation through the local wireless projection connection.

[0006] Furthermore, the mobile relay terminal is also used to collect near-end voice commands and convert them into the control commands, and send the control commands to the remote execution device with a session through the control relay server.

[0007] Furthermore, the control commands include one or more of the following: simulated keyboard operation commands, simulated mouse operation commands, application start / stop commands, media playback control commands, and system setting adjustment commands.

[0008] Furthermore, the mobile relay terminal integrates a voice processing module, which includes a voice acquisition unit, a voice recognition unit, and a command parsing unit; The voice acquisition unit is used to capture voice commands in the near-end environment; The speech recognition unit is used to convert the speech command into text information, which supports local recognition or cloud recognition service calls; The instruction parsing unit is used to convert the text information into control instructions containing operation types and parameters according to preset mapping rules.

[0009] Furthermore, the remote execution device includes a data acquisition module, an encoding and compression module, a communication module, and an instruction execution module; The data acquisition module is used to capture the audio and video data streams locally for human-computer interaction in real time; The encoding and compression module is used to encode and compress the acquired audio and video data streams using audio and video encoding standards. The communication module is used to establish the long connection with the control relay server through a network transmission protocol, and to receive the audio and video data stream transmission instructions and the control instructions; The instruction execution module is used to call the operating system interface to simulate the execution of user input or program operations corresponding to the control instructions.

[0010] Furthermore, the long connection is a full-duplex communication channel based on a network transmission protocol. The long connection includes a heartbeat keep-alive mechanism and an automatic reconnection mechanism after disconnection. The heartbeat keep-alive mechanism and the automatic reconnection mechanism after disconnection are used for the remote execution device to maintain real-time data transmission when it is located in a private network. The long connection is also used for the remote execution device to receive and execute control commands forwarded by the control relay server.

[0011] Furthermore, the control relay server includes a session management module, a connection information generation module, a data relay module, and a command routing module; The session management module is used to manage the connection status, device identification, and session association relationships of all online devices; The connection information generation module is used to generate dynamic identification information, including device identifier, server address and session token, for the successfully registered remote execution device; The data relay module is used to receive the audio and video data stream encoded by the remote execution device and forward it to the associated mobile relay terminal; The instruction routing module is used to receive control instructions from the mobile relay terminal and forward them to the corresponding remote execution device according to the device identifier.

[0012] Furthermore, the local wireless screen projection connection between the mobile relay terminal and the display terminal is established based on DLNA, AirPlay, Miracast, or Chromecast protocols; The network communication connection between the mobile relay terminal and the control relay server is established based on the mobile Internet or wireless local area network and supports encrypted data transmission.

[0013] Furthermore, the connection information includes the device identifier of the remote execution device, the access address of the control relay server, and session token security verification information. After scanning and parsing the connection information, the mobile relay terminal initiates a session establishment request and device association process.

[0014] Furthermore, the connection information can be any one of a QR code, a near-field communication tag, a Bluetooth broadcast frame, an ultrasonic beacon, or a short URL. Any one of the connection information is used by the mobile relay terminal to parse and establish a session with the control relay server.

[0015] This application's embodiments, by constructing a four-party collaborative architecture involving a remote execution device, a control relay server, a mobile relay terminal, and a display terminal, bring several beneficial effects. Firstly, cross-network projection is more convenient and efficient. Session establishment is triggered by scanning a code on the mobile relay terminal, eliminating the need for manual network parameter configuration by the user. The control relay server maintains a long connection with the remote execution device, and with the device identifier and security information included in the connection information, instant connection via code scanning is achieved, simplifying the operation process and solving the problems of complex cross-network projection processes and difficult configuration in traditional solutions. Ordinary users can quickly initiate cross-internet projection without professional knowledge. Secondly, the interactive experience is more natural and fluid. Voice command collection is placed in the near-end space. The mobile relay terminal uses voice recognition and command parsing technology to convert user voice into structured control commands, achieving WYSIWYG (what you say is what you get). Users do not need to manually operate the terminal and can focus on the presentation and interaction, avoiding the problem of traditional manual control interrupting the presentation rhythm, and improving the smoothness and immersion of remote presentations. Thirdly, network adaptability is stronger and robustness is higher. The architecture of the remote execution device actively connecting to the public network control relay server cleverly avoids NAT traversal. First, it avoids complex configurations for home or business private networks, while its long-connection mechanism and reconnection design ensure connection stability during network fluctuations, making the system adaptable to different network environments and more universal. Second, it ensures service reliability through an end-to-end closed loop. The system integrates multiple technical aspects such as device registration, session management, audio and video encoding and transmission, voice recognition, and command routing, forming a loosely coupled service closed loop. Each component has a clear division of labor and efficient data flow, ensuring low-latency transmission of screen data streams and accurate execution of control commands. This avoids compatibility issues caused by multi-system integration and improves the overall reliability and stability of the service. Third, it enhances collaboration efficiency through multi-scenario adaptability. This solution is suitable for various scenarios such as remote office reporting, remote teaching demonstrations, and cross-regional technical support, freeing up users' hands and enhancing the sense of presence and interactivity of remote collaboration. By simplifying the screen projection process and optimizing the interaction method, it reduces the operation time and communication costs in the collaboration process and improves the overall efficiency of cross-regional collaboration. Attached Figure Description

[0016] Figure 1 This is a structural diagram of a remote device screen projection and interaction system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the interaction of a remote device screen projection system provided in an embodiment of this application; Figure 3 This is a structural diagram of the remote execution device provided in the embodiments of this application; Figure 4 This is a structural diagram of the control relay server provided in the embodiments of this application; Figure 5 This is a structural diagram of the mobile relay terminal provided in the embodiments of this application; Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but additional steps not included in the drawings may also be present. The above processes can correspond to methods, functions, procedures, subroutines, subroutines, etc.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] This application relates to a remote device projection and interaction system, electronic device, and computer-readable storage medium, which is suitable for scenarios such as cross-regional collaborative work, remote presentations, and remote teaching. Through the collaboration of a remote execution device, a control relay server, a mobile relay terminal, and a display terminal, it enables convenient cross-network projection and natural near-end voice interaction, solving problems such as local area network limitations, complex operation, unnatural interaction, and network adaptation difficulties in traditional solutions, thereby improving the efficiency of remote collaboration and user experience.

[0020] Figure 1 This is a structural diagram of a remote device screen projection and interaction system provided in an embodiment of this application. Please refer to it. Figure 1 The remote device projection and interaction system of this application comprises a remote execution device 101, a control relay server 102, a mobile relay terminal 103, and a display terminal 104. These components work collaboratively through a network to form a closed-loop system encompassing data acquisition, relay, projection, and control. Specifically, it includes: In one embodiment, the remote execution device 101 is a remote content source and instruction execution carrier, deployed in a remote fixed location, such as a home or office. The remote execution device 101 can be a personal computer, workstation, or other computing device. The remote execution device 101 is used to collect local audio and video data streams, establish a long connection with the control relay server 102, and receive and execute control instructions.

[0021] The remote execution device 101 captures local screen images, audio data, and other audio and video data streams for human-computer interaction in real time, covering scenarios such as desktop operation, application interface, and media playback content. The acquisition frequency can be flexibly configured.

[0022] Long-lived connections are established by actively establishing a full-duplex long-lived connection with the control relay server 102 via the Internet, based on network transmission protocols such as WebSocket and TCP. It includes a heartbeat keep-alive mechanism (sending a heartbeat packet every 30 seconds by default) and an automatic reconnection mechanism (reconnection is initiated within 1 second after disconnection, and the reconnection interval is gradually increased to 30 seconds) to ensure continuous communication in the private network environment.

[0023] The remote execution device 101 executes commands by receiving control commands forwarded by the control relay server 102, and by calling the operating system interface, it simulates user operations such as keyboard key presses, mouse clicks / movements, and application start / stop, ensuring the accuracy and real-time performance of command execution.

[0024] The control relay server 102 is deployed on a server cluster in a public cloud, serving as the system communication hub, session management center, and data relay core. It manages device sessions, generates connection information, and relays audio and video data streams and control commands. Optionally, the session management of the control relay server 102 maintains the connection status, device identifiers, and session associations of all online remote execution devices 101 and mobile relay terminals 103, supports concurrent device access, and stores session data in real time using databases such as Redis to ensure consistent session states.

[0025] Connection information generation generates unique connection information for the successfully registered remote execution device 101, such as a QR code or near-field communication tag. This information includes the device identifier, server access address, and session token security verification information. The validity period is configurable to prevent unauthorized access.

[0026] The control relay server 102 receives the encoded audio and video data stream from the remote execution device 101 and forwards it to the corresponding mobile relay terminal 103 according to the session association. It also supports data stream fragmentation transmission and breakpoint resumption, adapting to different network bandwidth environments. At the same time, it receives control commands from the mobile relay terminal 103 and accurately routes them to the target remote execution device 101 according to the device identifier.

[0027] The mobile relay terminal 103 is a smart mobile device carried by the user, such as a smartphone or tablet, which serves as a connection bridge between the local and remote locations. It is used to establish dual-path connections, collect voice commands, convert control commands, and relay audio and video data streams.

[0028] The dual-path connection is established based on the connection information and control relay server 102 to establish a network communication connection (supporting 4G / 5G and Wi-Fi), and at the same time, a wireless projection connection is established with the local display terminal 104 (based on protocols such as DLNA and AirPlay) to realize the bridging of remote data streams and local displays.

[0029] The mobile relay terminal 103 integrates a voice processing module, which is used to collect near-end voice commands through a microphone, convert them into text through voice recognition, and then parse them into structured control commands. It supports local offline recognition and cloud recognition service calls, and is adapted to different network environments.

[0030] The mobile relay terminal 103 receives audio and video data streams forwarded by the control relay server 102, decodes them, and pushes them to the display terminal 104 via local screen projection. It also sends structured control commands to the remote execution device 101 via the control relay server 102 to complete the interaction of voice input, command execution, and screen feedback.

[0031] Display terminal 104 is located in the user's local space, such as a display device in a conference room, such as a smart TV, conference screen, or projector. Display terminal 104 is used to initially display connection information and receive and present projected content. Display terminal 104 receives and displays connection information, such as QR codes, pushed by remote execution device 101 or control relay server 102. Display terminal 104 supports full-screen display and timed refresh to ensure user convenience in scanning codes.

[0032] Display terminal 104 receives decoded audio and video data streams pushed by mobile relay terminal 103, renders and displays the screen content of remote devices in real time, and supports high-definition resolution and low latency presentation to ensure smooth presentation.

[0033] The collaborative architecture consisting of remote execution device 101, control relay server 102, mobile relay terminal 103 and display terminal 104 breaks the limitations of local area network. Mobile relay terminal 103 realizes dual-path connection and voice control bridging, control relay server 102 solves the problems of NAT traversal and session management, and display terminal 104 provides an intuitive presentation carrier. Overall, it realizes convenient screen projection and natural interaction across networks, reduces the operation threshold and enhances the immersive experience of remote collaboration.

[0034] Figure 2 This is a flowchart illustrating the interaction of a remote device screen projection system according to an embodiment of this application. Please refer to [link / reference]. Figure 2The complete interactive process of this application system, from remote device registration to screen projection and voice control, forms a standard execution chain, specifically including: Step 201: The remote execution device is used to collect local audio and video data streams for human-computer interaction and establish a long connection with the control relay server.

[0035] In one embodiment, after the remote execution device 101 starts up, it automatically completes registration and establishes a long connection with the control relay server 102. Optionally, after the client software on the remote execution device 101 starts up, it automatically collects the hardware identifiers of the remote execution device 101, such as the CPU serial number and network card MAC address, generates a unique device ID, and sends a registration request to the control relay server 102 via the HTTPS protocol. The registration request carries the capability information of the remote execution device 101, such as the supported encoding formats and maximum acquisition resolution.

[0036] After successful registration, a full-duplex connection is established with the control relay server 102 via the WebSocket protocol to negotiate data transmission parameters, such as fragment size and encoding level. A heartbeat keep-alive mechanism is initiated, periodically sending heartbeat packets to the control relay server 102. The control relay server 102 responds to the heartbeat request to maintain the connection. If the network is interrupted, the client automatically initiates a disconnection reconnection mechanism. After successful reconnection, the session context is restored without manual intervention from the user.

[0037] The remote execution device 101 client interface displays real-time connection status (online / offline), control relay server 102 address, session ID, and other information, making it easy for users to check device connection status. Automatic registration and long-connection mechanisms eliminate the need for manual user configuration, while heartbeat keep-alive and disconnection reconnection ensure connection stability, resolving issues of complex network configuration and easy connection interruptions, and improving system robustness.

[0038] Step 202: The control relay server is used to manage the sessions of all online devices, generate connection information for the remote execution device, and relay the audio and video data streams through the long connection.

[0039] In one embodiment, the control relay server 102 generates connection information for the successfully registered remote execution device 101 and supports pushing it to the display terminal 104 for display.

[0040] The session management module 1021 generates a unique session token based on the device ID of the remote execution device 101. The connection information generation module 1022 integrates the device identifier, the access address of the control relay server 102, the session token and the validity period to generate connection information. The default format is a QR code, but it can be switched to other formats such as near-field communication tags.

[0041] Users can trigger connection information push through the client of the remote execution device 101, control the relay server 102 to send the connection information to the designated display terminal 104 (supports IP address specification and device name search selection), or the user can directly project the screen content of the remote execution device 101 to the display terminal 104 and display the connection information.

[0042] Connection information has a default expiration date and automatically expires after that. The control relay server 102 generates new connection information to prevent long-unused connection information from being misused, thus improving security. Dynamically generated connection information includes a security verification mechanism, supports multiple presentation formats and expiration management, ensuring both convenience and improved access security, and solving the problems of easy leakage and lack of expiration time limits for connection information.

[0043] Step 203: The mobile relay terminal is used to establish a session with the remote execution device based on the connection information, and to establish a local wireless projection connection with the display terminal. It also receives the audio and video data stream from the remote execution device with which the session exists, and sends it to the display terminal for presentation through the local wireless projection connection.

[0044] In one embodiment, a user scans connection information via mobile relay terminal 103 to establish a network connection with control relay server 102 and a local projection connection with display terminal 104, and enables voice control. Optionally, the user opens a dedicated app on mobile relay terminal 103, activates the camera to scan a QR code on display terminal 104, and the app parses information such as the device ID of remote execution device 101, the server address of control relay server 102, and session token to verify the validity of the connection information, such as verifying the validity period and signature.

[0045] Based on the resolved server address of the control relay server 102, the App establishes an encrypted network connection with the control relay server 102 via mobile Internet or Wi-Fi, and uses TLS 1.3 protocol encryption to send a session establishment request. The session establishment request carries the device ID of the remote execution device 101, the session token, and the capability information of the mobile relay terminal 103, such as the supported decoding format and screen projection protocol.

[0046] The app automatically scans for display terminals 104 on the local area network, supports DLNA, AirPlay, Miracast, and Chromecast protocols, and lists available devices. Users can manually select the target display terminal 104, or the app can automatically match the nearest display terminal 104 to establish a screen mirroring connection and negotiate screen mirroring parameters such as resolution and frame rate.

[0047] After verifying the validity of the session token, the control relay server 102 associates the mobile relay terminal 103 with the remote execution device 101, sends a successful association notification to both ends, and completes the establishment of a dual-path connection. Scanning a code to trigger the connection simplifies the operation process, encrypted network connections ensure data security, and automatic scanning and matching of display terminals improves the convenience of screen projection, solving the problems of cumbersome connection establishment and low security in traditional solutions.

[0048] After successful session association, the remote execution device 101 acquires audio and video data streams, which are then relayed to the mobile relay terminal 103 via the control relay server 102, and finally projected onto the display terminal 104. Optionally, the data acquisition module 1011 of the remote execution device 101 initiates screen capture (Windows system calls GDI+ and DirectX interfaces, Linux system calls X11 interface) and audio acquisition (calls the system audio interface to capture speaker output), generating raw audio and video data streams.

[0049] The encoding and compression module 1012 uses the H.264 / H.265 video encoding standard and the OPUS / AAC audio encoding standard to encode and compress the original data stream. The encoding parameters can be dynamically adjusted according to the network bandwidth. When the bandwidth is sufficient, a high bit rate and a low compression ratio are used. When the bandwidth is insufficient, the bit rate is automatically reduced. The compressed data stream is sent to the control relay server 102 through a long connection.

[0050] The data relay module 1023 of the control relay server 102 receives the encoded data stream, forwards it to the corresponding mobile relay terminal 103 according to the session association, and supports data stream fragmentation transmission. The size of each fragment can be configured from 1KB to 10KB to avoid large file transmissions occupying too much bandwidth.

[0051] After receiving the data stream, the mobile relay terminal 103 decodes it using its built-in decoder, supporting hardware decoding acceleration, to obtain the original audio and video data. This data is then pushed to the display terminal 104 via a local screen mirroring connection. The display terminal 104 renders and presents the data in real time, achieving low-latency screen mirroring of remote screen content. This system employs audio and video encoding standards to ensure compatibility, dynamically adapts encoding parameters to different network environments, accelerates processing efficiency through hardware decoding, and ensures smooth presentation through low-latency screen mirroring, thus resolving issues such as high latency, poor compatibility, and weak network adaptability.

[0052] Users control remote execution devices through near-end voice commands, forming a complete interaction. Optionally, the voice acquisition unit 1031 (microphone array) of the mobile relay terminal 103 monitors the near-end ambient voice in real time, supports voice wake-up, such as the wake-up word of the screen mirroring assistant and on-demand acquisition (the user clicks the App acquisition button), and uses noise reduction algorithms to filter ambient noise and improve the clarity of voice commands.

[0053] The speech recognition unit 1032 converts the collected audio data into text information, supports local offline recognition and cloud recognition service calls, and can switch to offline recognition in the absence of network environment to ensure the availability of basic functions.

[0054] The instruction parsing unit 1033 converts text information into structured control instructions according to preset mapping rules. The control instruction types include simulated keyboard operation instructions, such as mapping the next page to the right-click instruction, simulated mouse operation instructions, such as mapping zoom to the mouse wheel up instruction, application start / stop instructions, such as mapping opening Excel to the start Excel program instruction, media playback control instructions, such as mapping pause to the media pause instruction, and system setting adjustment instructions, such as mapping increasing volume to the system volume increase instruction.

[0055] The structured control command is sent to the control relay server 102 via an encrypted network connection. The command routing module 1024 forwards the command to the target remote execution device 101 based on the device ID. After receiving the command, the command execution module 1014 of the remote execution device 101 calls the operating system interface to simulate execution. The Windows system calls the pywin32 library, and the Linux system calls the XTest library to complete the corresponding operation.

[0056] After the remote execution device 101 executes the command, the screen content changes. The updated audio and video data stream is collected, encoded, relayed, decoded, and projected along the original path, and presented in real time on the display terminal 104, completing a closed loop of voice control interaction. It supports users to continuously send voice commands for continuous control.

[0057] Scanning a QR code to trigger a connection simplifies the operation process, encrypted network connections ensure data security, and automatic scanning to match display terminals enhances the convenience of screen projection. Mainstream audio and video encoding standards ensure compatibility, dynamic encoding parameters adapt to different network environments, hardware decoding acceleration improves processing efficiency, and low-latency screen projection ensures smooth presentations. Voice wake-up and noise reduction improve the accuracy of command acquisition, local and cloud dual-mode recognition adapts to different network environments, command types cover multiple scenario needs, and low-latency feedback ensures smooth interaction, achieving a natural, WYSIWYG interaction and solving problems such as cumbersome connection establishment, low security, high screen projection latency, and cumbersome manual control.

[0058] Figure 3 This is a structural diagram of the remote execution device provided in the embodiments of this application. Please refer to it. Figure 3 The remote execution device comprises a data acquisition module 1011, an encoding and compression module 1012, a communication module 1013, and an instruction execution module 1014. These modules work together to achieve data acquisition, encoding, transmission, and instruction execution, specifically including: In one embodiment, the data acquisition module 1011 is used to capture local audio and video data streams for human-computer interaction in real time, providing raw data for encoded transmission. Optionally, the screen capture unit uses a system-level interface to capture screen images. Windows systems support three capture methods: GDI+, DirectX, and Duplication API; Linux systems support X11 and XRandR interfaces; and macOS systems support the Quartz Display Services interface. The optimal capture method can be automatically selected based on device performance and system version. The capture resolution supports adaptive scaling (up to 4K), the capture frequency is configurable (15fps-60fps), and full-screen capture and specified area capture are supported. Users can set the coordinates of the capture area on the client side.

[0059] The audio acquisition unit calls the system audio interface (Windows WaveIn, Linux ALSA, macOS CoreAudio) to capture speaker output audio and microphone input audio, supports mono / stereo acquisition, has a configurable sampling rate (44.1kHz-48kHz), and a bit depth of 16bit to ensure clear audio quality.

[0060] The data synchronization unit adds timestamps to the collected audio and video data, achieving audio and video synchronization based on these timestamps. This avoids audio-visual desynchronization issues during screen projection, with synchronization errors controlled within 10ms. Multiple interfaces adapt to different operating systems, supporting flexible configuration of resolution and frame rate. The audio-visual synchronization mechanism ensures a smooth projection experience, resolving issues of poor acquisition compatibility and audio-visual desynchronization.

[0061] The encoding and compression module 1012 uses audio and video encoding standards to encode and compress the acquired data stream, reducing the bandwidth usage.

[0062] The video encoding unit integrates the FFmpeg encoding library, supporting both H.264 and H.265 (HEVC) video encoding standards. H.264 encoding ensures compatibility, while H.265 encoding reduces bandwidth consumption at the same image quality. Users can manually select the encoding standard on the client side, or the system can automatically switch based on network bandwidth. The encoding bitrate supports dynamic adjustment (1Mbps-20Mbps), employing both CBR (Constant Bitrate) and VBR (Variable Bitrate) adaptive modes. When the network is stable, VBR is used to improve image quality, while CBR is used to ensure smoothness when the network fluctuates.

[0063] The audio encoding unit supports OPUS and AAC audio encoding standards. OPUS encoding is suitable for different bandwidth scenarios (6kbps-510kbps) and offers better sound quality than AAC. OPUS encoding is used by default, but it automatically switches to AAC encoding when compatible with terminals that do not support OPUS. The encoded audio and video data are encapsulated into a TS stream (Transport Stream) for easy transmission and synchronization.

[0064] The compression optimization unit further compresses the encoded data stream, reducing the amount of data transmitted while preserving data integrity. Decompression does not affect audio and video quality. Mainstream encoding standards ensure compatibility, dynamic bitrate and encoding switching adapt to different network environments, and audio and video encapsulation and compression optimizations reduce bandwidth usage, solving the problems of high bandwidth consumption and poor network adaptability.

[0065] Communication module 1013 is used to establish a long connection with control relay server 102 to realize the transmission of data streams and commands, specifically including: The connection establishment unit supports both WebSocket and TCP network transport protocols, with WebSocket (based on HTTP, offering strong firewall penetration capabilities) as the default. In network-restricted scenarios, it can switch to TCP. During connection establishment, TLS 1.3 is used to encrypt data transmission and verify server certificates and session tokens to prevent data eavesdropping and tampering.

[0066] The long connection maintenance unit implements a heartbeat keep-alive mechanism, sending a heartbeat packet (containing device ID, session ID, and connection status) every 30 seconds. After the control relay server 102 responds to the heartbeat request, it updates the connection status. If no response is received from the control relay server 102, the disconnection reconnection mechanism is initiated, with reconnection intervals of 1s, 3s, 5s, 10s, and 30s respectively. After 5 failed reconnections, it continues to reconnect every 30 seconds until the connection is restored.

[0067] The data transmission unit employs a segmented transmission mechanism, dividing the encoded audio and video data stream into 1KB-10KB data packets, adding packet sequence numbers, checksums, timestamps, and other information, and sending them to the control relay server 102 via a long connection. It also supports breakpoint resumption; if some data packets fail to transmit, the control relay server 102 notifies the client to retransmit the corresponding packets, avoiding retransmission of the entire data stream and improving transmission efficiency. Simultaneously, it receives control commands forwarded by the control relay server 102, parses them, and transmits them to the command execution module 1014. Multi-protocol support enhances penetration capability, encrypted transmission ensures data security, heartbeat keep-alive and disconnection reconnection ensure connection stability, and segmented transmission and breakpoint resumption improve transmission efficiency, solving the problems of easy connection interruption, insecure data transmission, and low efficiency in traditional solutions.

[0068] The instruction execution module 1014 is used to receive control instructions and call the operating system interface to simulate the execution of corresponding operations, specifically including: The instruction parsing unit 1033 receives control instructions transmitted by the communication module 1013, parses the instruction type (such as keyboard operation, mouse operation, application start / stop, etc.) and parameters (such as keyboard key values, mouse coordinates, application path, etc.), verifies the legality of the instruction format, and discards illegal instructions directly and records them in the log.

[0069] The operation simulation unit calls the corresponding operating system interface according to the instruction type to simulate user input and program operation.

[0070] Keyboard operation simulation uses the keybd_event function called by the Windows system, the XTestFakeKeyEvent function called by the Linux system, and the CGEventCreateKeyboardEvent function called by the macOS system to simulate keyboard key presses and releases. It supports common function keys such as up, down, left, and right arrow keys, Enter, Esc, and character keys.

[0071] Mouse operation simulation uses the mouse_event function in Windows, the XTestFakeMotionEvent and XTestFakeButtonEvent functions in Linux, and the CGEventCreateMouseEvent function in macOS to simulate mouse movement, left-click / double-click, right-click, and scroll wheel scrolling, supporting both absolute and relative coordinate positioning.

[0072] Application startup and shutdown involve calling system process interfaces (Windows CreateProcess, Linux fork / exec, macOS NSTask) to launch an application at a specified path or terminate an application with a specified process ID. It supports startup and shutdown control for common office software (Word, Excel, PPT), media players, browsers, and other applications.

[0073] Media control calls the system media control interface to implement media playback (Play), pause (Pause), stop (Stop), volume adjustment and other operations. It supports mainstream media players such as Windows Media Player, VLC and PotPlayer.

[0074] The system settings have been adjusted to call the system settings interface to perform operations such as screen brightness adjustment, system volume adjustment, and monitor resolution switching, adapting to the setting mechanisms of different operating systems.

[0075] The execution result feedback unit records the instruction execution status (success / failure), execution time, and operation content, and feeds this information back to the control relay server 102 via the communication module 1013. This facilitates the mobile relay terminal 103 to synchronize the execution results and allows users to view the instruction execution status. Multiple instruction types cover various scenarios, and calling underlying system interfaces ensures operational accuracy. Instruction verification and logging enhance security and maintainability. Execution result feedback achieves closed-loop management, solving the problems of single instruction execution, poor accuracy, and lack of feedback.

[0076] Figure 4 This is a structural diagram of the control relay server provided in the embodiments of this application. Please refer to it. Figure 4 The internal modules of the control relay server 102 include a session management module 1021, a connection information generation module 1022, a data relay module 1023, and a command routing module 1024. These modules work together to implement session management, connection information generation, data relay, and command routing. Specifically, they include: In one embodiment, the session management module 1021 is used to manage the connection status, device identifier, and session association relationships of all online devices, specifically including: The device registration management unit receives registration requests from remote execution device 101 and mobile relay terminal 103, verifies the legality of registration information such as device ID format and session token validity, assigns a unique session ID to the new device, stores basic device information (device type, model, supported protocols and encoding formats) to the Redis database, and sets the device online status to online.

[0077] The connection status monitoring unit monitors the device connection status in real time and verifies the device's online status through heartbeat packets. If no heartbeat packets are received from the device for 3 consecutive times (default 90 seconds), the device status is marked as offline, and the session information is retained for 30 minutes. The session can be quickly restored after the device comes back online. The device session information that has been offline for more than 30 minutes is cleaned up periodically to free up server resources.

[0078] The session association unit receives the session establishment request from the mobile relay terminal 103, verifies the validity of the session token and device ID, queries the status of the corresponding remote execution device 101, and if the remote execution device 101 is online, establishes a session association relationship between the mobile relay terminal 103 and the remote execution device 101, stores the association mapping table (session ID → mobile terminal ID + remote device ID), and sends a successful association notification to both ends; if the remote execution device 101 is offline, returns a session establishment failure response and prompts the user to check the status of the remote device.

[0079] The session destruction unit receives a device's active disconnection request or detects a device's offline timeout, destroys the corresponding session association, updates the device's online status, and releases related resources, such as cached data streams and instruction queues, ensuring reasonable allocation of server resources. Registration and status management ensure standardized device access, the session association mechanism achieves precise binding between mobile relay terminals and remote execution devices, and session destruction and resource cleanup improve server operating efficiency, resolving issues of chaotic session management and excessive resource consumption.

[0080] The connection information generation module 1022 is used to generate connection information containing security verification information for the successfully registered remote execution device 101, specifically including: The identifier generation unit generates a random session token (32-bit hexadecimal string) based on the unique device ID of the remote execution device 101. Combined with the server domain name / IP address and session validity period (default 3600 seconds), the token forms connection information data. The data is encrypted using the AES-256 encryption algorithm to prevent tampering.

[0081] The presentation format conversion unit converts the encrypted connection information into various presentation formats for users to choose from. Optionally, the QR code adopts the QR Code encoding standard to generate a QR code image containing encrypted connection information, supporting different sizes (100×100 pixels to 500×500 pixels), which users can download and save or directly push to the display terminal for display.

[0082] The Near Field Communication (NFC) Tag writes encrypted connection information into the NFC tag data area. The mobile relay terminal 103 can read the connection information through NFC sensing without scanning a code, making it suitable for scenarios where it is inconvenient to display a QR code.

[0083] Bluetooth broadcast frames transmit encrypted connection information via Bluetooth Low Energy (BLE). Mobile relay terminal 103 can receive broadcast data and parse the connection information after Bluetooth is turned on, making it suitable for short-range scenarios without visual display.

[0084] The ultrasonic beacon converts connection information into ultrasonic signals (20kHz-22kHz), which are played through the audio device associated with the server. The mobile relay terminal 103 receives the ultrasonic signals through the microphone and parses them to obtain the connection information, making it suitable for scenarios where multiple devices can access the network simultaneously.

[0085] The short link URL stores encrypted connection information in the server database and generates a unique short link URL. Users can display the short link through the display terminal 104, or enter the URL or scan the short link QR code in the mobile relay terminal 103 to obtain connection information.

[0086] The validity management unit records the generation time, validity period, and usage status of connection information. Within the validity period, it allows only one binding to the mobile relay terminal 103. After binding, the connection information automatically expires to prevent reuse. Users can manually refresh the connection information to generate a new session token and QR code; the old connection information immediately becomes invalid, improving security. Multiple presentation formats adapt to different use cases. Encryption and validity period management ensure connection information security, and automatic expiration after binding prevents abuse, addressing the issues of limited connection information formats and low security.

[0087] The data relay module 1023 is used to receive audio and video data streams from the remote execution device 101 and forward them to the corresponding mobile relay terminal 103 according to the session association relationship. Specifically, it includes: The data stream receiving unit listens to the data stream transmission port of the remote execution device 101, receives the encoded audio and video TS stream data packets, verifies the data packet checksum and sequence number, discards erroneous data packets, and notifies the remote execution device 101 to retransmit, ensuring the integrity of the data stream.

[0088] The data stream caching unit uses a circular buffer to cache the received data stream. The buffer capacity is configurable to avoid data loss due to network fluctuations. The cached data stream is sorted by timestamp to ensure the correct forwarding order and avoid screen stuttering or errors during projection.

[0089] The data stream forwarding unit queries the session association mapping table to determine the mobile relay terminal 103 associated with the remote execution device 101, and forwards the cached data stream to the mobile relay terminal 103 in the original fragment format. It supports batch forwarding and on-demand forwarding. On-demand forwarding is triggered when the bandwidth of the mobile relay terminal 103 is insufficient. During the forwarding process, the timestamp, checksum and other information of the data packets remain unchanged to ensure that the mobile relay terminal 103 can correctly decode and synchronize.

[0090] The bandwidth adaptation unit monitors the network bandwidth status of the mobile relay terminal 103 in real time (by receiving bandwidth data reported by the mobile terminal). When bandwidth is insufficient, it notifies the remote execution device 101 to reduce the encoding rate; when bandwidth recovers, it notifies the remote execution device 101 to increase the encoding rate, thus achieving dynamic adaptation between data stream transmission and network bandwidth. Packet verification and retransmission ensure data integrity, caching and sorting ensure the correctness of forwarding order, dynamic bandwidth adaptation improves network adaptability, and batch and on-demand forwarding optimizes transmission efficiency, solving problems such as data loss, disordered order, and poor network adaptability during data transfer.

[0091] The instruction routing module 1024 is used to receive control instructions from the mobile relay terminal 103 and accurately forward them to the corresponding remote execution device 101 according to the device identifier. Specifically, this includes: The instruction receiving unit receives encrypted control instructions sent by the mobile relay terminal 103, decrypts the instruction data using the AES-256 decryption algorithm, verifies the validity of the instruction signature (to prevent tampering) and the session ID, and discards illegal instructions directly and records them in the log.

[0092] The command parsing and routing unit parses and decrypts the control command, extracts the target remote device ID, command type and parameters, queries the session association mapping table, and confirms whether the target remote execution device 101 is online and bound to the mobile relay terminal 103. If the device is online and bound, the control command is added to the command queue of the remote execution device 101 and forwarded according to the first-come, first-served principle. If the device is offline or not bound, a command forwarding failure response is returned to the mobile relay terminal 103, prompting the user to check the device status or re-establish the session.

[0093] The instruction queue management unit maintains an independent instruction queue for each online remote execution device 101. The queue length is configurable. When the queue is full, the latest instruction is discarded and a queue full response is returned. Instructions that have timed out and have not been executed are cleaned up periodically to avoid instruction backlog. Instruction priority settings are supported, such as emergency control instructions having higher priority than ordinary instructions to ensure that critical instructions are executed first.

[0094] The forwarding status feedback unit records the command forwarding status (success / failure, queue waiting, executed) and feeds the status back to the mobile relay terminal 103, allowing users to understand the command execution progress in real time and solving the problem of no feedback during command execution. Encryption and signature verification ensure command security, precise routing ensures the correct delivery of commands, queue management avoids command backlog, and priority settings and status feedback improve the reliability of command execution, solving the problems of chaotic command forwarding, lack of security, and lack of status feedback.

[0095] Figure 5 This is a structural diagram of the mobile relay terminal provided in the embodiments of this application. Please refer to it. Figure 5 The mobile relay terminal integrates a voice processing module, including a voice acquisition unit 1031, a voice recognition unit 1032, and a command parsing unit 1033. It also integrates a communication module and a screen projection module to achieve voice control, data transmission, and local screen projection functions. Specifically, it includes: In one embodiment, the voice acquisition unit 1031 is used to capture voice commands in a near-end environment, filter noise, and improve command clarity, specifically including: The audio capture unit calls the microphone array interface of the mobile terminal, supports single microphone and multi-microphone array, and collects near-end audio data. The sampling rate is configurable (44.1kHz-48kHz), the bit depth is 16bit, and the number of channels supports mono / stereo. The default is mono acquisition to reduce the amount of data.

[0096] The noise filtering unit integrates adaptive noise suppression algorithms, such as spectral subtraction and Wiener filtering, to filter environmental noise, such as the sound of air conditioning in the conference room, conversations, and background music. It also employs an echo cancellation algorithm to eliminate echo interference from the sound played on the display terminal, ensuring clear voice commands. Furthermore, it supports voice wake-up functionality. When the user speaks a preset wake-up word, such as "screen mirroring assistant," voice command acquisition is initiated. When not woken up, it operates in a low-power listening state, reducing terminal power consumption.

[0097] The audio preprocessing unit performs gain adjustment (automatically adapting to voice volume at different distances), DC offset removal, and audio truncation (removing silence segments before and after the speech) on the acquired audio data to generate standardized audio data, providing high-quality input for recognition. Multi-microphone support and noise filtering improve voice acquisition quality, voice wake-up enables low power consumption and convenient triggering, and audio preprocessing ensures standardized input, solving the problems of environmental interference and cumbersome triggering in voice acquisition.

[0098] The speech recognition unit 1032 is used to convert audio data into text information, supporting both local offline recognition and cloud-based recognition service calls, specifically including: The recognition mode switching unit automatically switches the recognition mode according to the network status of the mobile relay terminal 103. When the network is available, it prioritizes the use of cloud recognition services, which has a high recognition accuracy and supports multilingual and dialect recognition. When there is no network, it switches to local offline recognition, which supports voice recognition of commonly used control commands to ensure the availability of basic functions.

[0099] The cloud-based recognition unit encodes the pre-processed audio data into PCM format and sends it to the cloud recognition service via HTTPS protocol, carrying recognition parameters (language type, recognition accuracy). It also receives the recognition text results returned by the cloud, supporting real-time streaming recognition and reducing recognition latency.

[0100] The local offline recognition unit loads the locally trained recognition model, supports a vocabulary of common control commands such as next page, open Excel, and pause, performs feature extraction (MFCC features) and pattern matching on the audio data, and outputs the recognized text results.

[0101] The recognition result optimization unit performs post-processing on the recognized text, including misspelling correction (fuzzy matching based on the control command vocabulary), semantic completion (supplementing omitted command keywords), and redundant information filtering (removing interjections and meaningless words), improving text accuracy and providing input for command parsing. Dual-mode recognition adapts to different network environments: cloud-based recognition ensures accuracy and multi-language support, while local offline recognition ensures availability even without a network connection. Optimizing the recognition results improves text quality, addressing the issues of network dependence and low accuracy in speech recognition.

[0102] The instruction parsing unit 1033 is used to convert text information into structured control instructions containing operation types and parameters according to preset mapping rules, specifically including: The instruction mapping library management unit has a built-in standardized instruction mapping library that stores the mapping relationship between text instructions and structured control instructions. The mapping library supports user-defined extensions, and users can add personalized instruction mappings in the App, such as mapping page turning to the down arrow on the keyboard. The mapping relationship is stored in JSON format for easy maintenance and updates.

[0103] The text semantic parsing unit performs semantic analysis on the optimized text information, extracts core instruction keywords, such as extracting the keyword "next page" for "next page" and extracting the keywords "open" and "Excel" for "open Excel", matches the corresponding entries in the instruction mapping library, and determines the instruction type, such as keyboard operation, application start / stop, and related parameters, such as keyboard key values ​​and application package name / path.

[0104] The structured instruction generation unit generates standardized structured control instructions based on the matching results. The instruction format is JSON, which includes fields such as instruction version, instruction type, parameter list, timestamp, and signature. Supported instruction types include keyboard operation instructions, application start / stop instructions, media playback control instructions, and system setting adjustment instructions.

[0105] The parameters of keyboard operation commands include the key name, such as Up, Down, Enter and Esc, and the operation type (press, release, click).

[0106] The parameters of mouse operation commands include operation type (move, left_click, left_double_click, right_click, scroll), coordinate information (absolute coordinates x / y or relative offset dx / dy), and scroll distance.

[0107] The parameters of the application start / stop command include the operation type (start, stop) and the application identifier (package name, path, process name).

[0108] The parameters of media playback control commands include the operation type (play, pause, stop, previous, next, volume up, volume down).

[0109] The parameters of the system settings adjustment command include the operation type (screen brightness adjustment, system volume adjustment, resolution switching) and the adjustment value (absolute value or relative ratio).

[0110] The instruction verification unit performs format verification and signature generation on the generated structured control instructions to ensure instruction integrity and legality, preventing illegal instructions from being sent to the server. An extensible instruction mapping library supports customized needs, semantic parsing and structured instruction generation ensure instruction accuracy, instruction verification ensures security, and multiple instruction types cover various control scenarios, solving the problems of single instruction parsing and poor flexibility.

[0111] The communication module is used to establish an encrypted network connection with the control relay server 102, transmit structured control commands, and receive audio and video data streams, specifically including: The connection establishment unit supports 4G / 5G and Wi-Fi (2.4GHz / 5GHz) network connections. Based on the resolved server address and session token, it establishes an encrypted connection with the control relay server 102 via the TLS1.3 protocol, verifies the validity of the server certificate, and prevents man-in-the-middle attacks.

[0112] The data transmission unit sends structured control commands to the control relay server 102 via an encrypted connection, employing a segmented transmission (suitable for long commands) and retransmission mechanism to ensure reliable command transmission; it receives audio and video data streams forwarded by the control relay server 102, stores them in a local buffer, and provides data for decoding and screen projection.

[0113] The network status monitoring unit monitors the network connection status and bandwidth in real time. It automatically reconnects when the network is interrupted and sends a bandwidth alarm to the server when the bandwidth is insufficient, triggering the remote execution device 101 to reduce the bit rate and ensure smooth screen projection.

[0114] The screen mirroring module is used to establish a wireless screen mirroring connection with the local display terminal 104, and pushes the decoded audio and video data to the display terminal 104, specifically including: The display terminal discovery unit supports DLNA, AirPlay, Miracast, and Chromecast screen mirroring protocols. It automatically scans for available display terminals 104 on the local area network and lists information such as device name, supported protocols, and resolution for users to select. It also supports automatic matching of the nearest display terminal 104.

[0115] The screen projection connection establishment unit establishes a wireless screen projection connection based on the user-selected display terminal 104 and the supported protocol, negotiates screen projection parameters (resolution, frame rate, bit rate), supports 1080P / 4K high-definition screen projection, and uses adaptive resolution by default to match the optimal display effect of display terminal 104.

[0116] The audio and video decoding unit integrates hardware decoding engines, such as Android MediaCodec and iOS AVFoundation, to decode the received audio and video data streams. It supports H.264 / H.265 video decoding and OPUS / AAC audio decoding, with a decoding latency of ≤50ms, ensuring audio and video synchronization.

[0117] The projection unit pushes the decoded audio and video data to the display terminal 104 via the projection connection, using real-time streaming protocols such as RTSP and RTP to ensure low data transmission latency. Projection latency is controlled within 200ms, meeting the real-time requirements of remote presentations. It supports pause, resume, and terminate projection operations, allowing users to control the projection status at any time. Multi-protocol support improves display terminal compatibility, hardware decoding ensures decoding efficiency and low latency, high-definition projection and low-latency transmission enhance the presentation experience, and flexible projection control improves user convenience, solving problems such as poor projection protocol compatibility, high latency, and inflexible control.

[0118] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Please refer to it. Figure 6 The hardware structure of the electronic device described in this application, which can integrate the remote device projection and interaction system of this application, ensures the efficient operation of the system through the collaborative work of hardware components, specifically including: In one embodiment, the electronic device includes a processor 901, a memory 902, an input device 903, an output device 904, and a communication module (integrating network communication and screen projection communication functions). The components are connected via an industrial-grade bus, such as PCIe or USB 3.0, to achieve data transmission and command interaction.

[0119] The processor 901 is used to run the software programs and modules in the memory 902, coordinate the work of various hardware components, and realize the various functions of the remote device projection and interaction system.

[0120] Select the appropriate processor based on the device type. The remote execution device 101 uses a high-performance multi-core CPU, such as Intel Core i5 / i7 or AMD Ryzen 5 / 7, which supports multi-threaded concurrent processing to meet the high-performance requirements of screen capture, encoding, and instruction execution. The control relay server 102 uses a server-grade CPU, such as Intel Xeon or AMD EPYC, with ≥16 cores and a main frequency of ≥2.4GHz, supporting concurrent access and data relay for millions of devices. The mobile relay terminal 103 uses a mobile-grade CPU, such as Qualcomm Snapdragon 8 series or Apple A series, which balances performance and power consumption and supports efficient operation of voice processing, decoding, and screen projection.

[0121] Its functions include running software programs for data acquisition, encoding and compression, communication transmission, command execution, voice processing, and screen projection control, controlling the collaborative work of each module, such as coordinating the data flow transmission between the data acquisition module and the encoding module, scheduling the network connection and data transmission and reception of the communication module, and allocating hardware resources, such as GPUs for video encoding and decoding acceleration, to ensure that the various functions of the system are implemented efficiently and collaboratively.

[0122] Processor 901 supports hardware acceleration technology. Remote execution device 101 and mobile relay terminal 103 utilize GPUs, such as NVIDIA CUDA and AMD OpenCL, to accelerate video encoding and decoding, reducing CPU utilization. Control relay server 102 utilizes network card multi-queue and hardware interrupt balancing technology to improve network data processing efficiency. Mobile relay terminal 103 utilizes DSP (Digital Signal Processor) to accelerate speech recognition and noise filtering, improving speech processing speed and accuracy. Differentiated processor selection adapts to different device requirements, multi-threading and hardware acceleration ensure high-performance system operation, and resource scheduling optimization improves component collaboration efficiency, solving the problems of insufficient hardware performance and unreasonable resource allocation in traditional devices.

[0123] The memory 902 is used to store software programs, configuration information, cached data, log data, etc., providing data for the processor 901 and functional modules. Optionally, an appropriate memory is configured according to the device type. The remote execution device 101 is configured with RAM ≥ 16GB (high-speed DDR4 / DDR5) and SSD ≥ 512GB to ensure the needs of screen capture caching, temporary storage of encoded data, and software operation. The control relay server 102 is configured with RAM ≥ 64GB and SSD ≥ 2TB to meet the needs of large amounts of session data, data stream caching, and log storage. The mobile relay terminal 103 is configured with RAM ≥ 8GB and flash memory ≥ 128GB to support the storage of voice models, cached data, and applications.

[0124] The storage is divided into a system area (storing the operating system and core software), an application area (storing dedicated client / server programs), a cache area (storing audio and video data stream caches and voice recognition caches), a log area (storing device operation logs, instruction execution logs, and error logs), and a configuration area (storing system configurations, protocol parameters, and instruction mapping libraries). This partitioned management improves the orderliness and security of storage and avoids interference between different types of data.

[0125] The storage device 902 supports encrypted data storage, such as SSD hardware encryption and software AES encryption, to protect configuration information, log data, and cached audio and video data from unauthorized access. It also supports data backup and recovery functions, automatically backing up critical configuration and log data periodically to prevent data loss. The control relay server 102 adopts a distributed storage architecture to ensure high availability of session data and data streams. Large-capacity, high-speed storage meets the storage needs of different devices; partition management improves storage orderliness; encryption and backup ensure data security; and distributed storage enhances server data availability, solving the problems of insufficient storage and lack of data security in traditional devices.

[0126] The input device 903 is used to receive user input and external data, providing a human-computer interaction interface for system operation.

[0127] The remote execution device 101 includes input devices such as a keyboard, mouse, and touch screen, which are used by users to manually configure client parameters, such as server address, acquisition resolution, start / stop screen projection service, and manually trigger connection information generation.

[0128] The input devices for the control relay server 102 include a keyboard, a mouse, and a remote management terminal, which are used by the administrator to configure server parameters, such as the maximum number of concurrent connections, data storage path, connection information validity period, monitor server operating status, troubleshoot, and perform other operations.

[0129] The mobile relay terminal 103's input devices include a touchscreen, microphone, camera, NFC module, and Bluetooth module. The touchscreen is used for user operation of the app, such as launching QR code scanning, selecting display terminals, and customizing command mappings. The microphone is used to collect voice commands, the camera is used to scan QR codes, the NFC module is used to sense and read connection information, and the Bluetooth module is used to receive connection information broadcast via Bluetooth. This diverse input device adapts to the operational needs of different devices. The mobile relay terminal's multimodal input enhances the convenience of connection and control, solving the problems of traditional devices having limited input methods and cumbersome operation.

[0130] The output device 904 is used to output the device's operating status, configuration information, and projected content, providing users with visual and audible feedback.

[0131] The output device of the remote execution device 101 includes a display and a speaker. The display is used to show the client's running status (connection status, acquisition frame rate, encoding bit rate), connection information preview, and log information; the speaker is used to play operation prompts (such as connection successful, command execution successful).

[0132] The control relay server 102 output device includes a display and indicator lights. The display is used to show the server operation monitoring interface (CPU / memory usage, number of concurrent connections, data transmission rate) and log information; the indicator lights are used to visually show the server operation status (power light, network light, operation status light).

[0133] The mobile relay terminal 103 output device includes a touch screen and a speaker. The touch screen is used to display the App interface (scanning interface, display terminal list, screen casting control interface, command execution status); the speaker is used to play voice recognition result prompts and screen casting status prompts, such as screen casting successful or screen casting disconnected.

[0134] The display terminal 104 output device includes a high-definition display screen and a speaker. The display screen is used to show connection information (QR code, short link) and the screen content of the remote device being projected; the speaker is used to play the audio content of the projected screen and supports stereo sound output. This multi-format output device provides rich feedback methods, while the high-definition display and audio output ensure a smooth projection experience. Status indicator lights and prompts enhance operational convenience, solving the problems of limited device output information and unintuitive feedback.

[0135] In one embodiment, the computer-readable storage medium provided in this application adopts a high-speed SSD, industrial-grade flash memory, or distributed cloud storage architecture, with capacity configurations adapted to different device scenarios. The remote execution device 101 and the mobile relay terminal 103 are adapted to a capacity of ≥512GB, and the control relay server 102 is adapted to a capacity of ≥2TB. It supports a wide operating temperature range of -40℃ to 85℃, and has industrial-grade characteristics such as power failure prevention, electromagnetic interference resistance, and high write endurance. It can operate stably in multiple scenarios such as offices, conference rooms, and industrial control.

[0136] The computer-readable instructions stored on the storage medium are organized in a modular structure, including execution code for functions such as data acquisition, encoding and compression, communication transmission, instruction processing, screen projection control, and session management. It also pre-stores basic data such as system configuration parameters, a standardized instruction mapping library, device identification templates, and security encryption keys. The instruction modules are loosely coupled, supporting independent updates and expansions. Corresponding functional modules can be flexibly loaded according to the device type (remote execution device / control relay server / mobile relay terminal).

[0137] When the processor executes the above instructions, it initiates differentiated operating logic according to the device role. The remote execution device 101 initializes the data acquisition and encoding module, automatically completes device registration and establishes a long connection with the control relay server 102, continuously acquires and encodes audio and video data streams for transmission, and simultaneously receives and forwards control commands and calls system interfaces for execution. The control relay server 102 initiates the session management and data relay module, dynamically generates connection information containing security tokens, maintains device associations, and accurately routes audio and video data streams and control commands. The mobile relay terminal 103 activates the voice processing, dual-path connection, and decoding projection module, parses the connection information, establishes network communication and local projection connections, completes the acquisition, recognition, parsing, and forwarding of voice commands, and simultaneously decodes audio and video data and pushes it to the display terminal 104.

[0138] This storage medium supports hot-swapping and online expansion. The remote execution device 101 and the mobile relay terminal 103 can expand the storage capacity through interfaces such as USB and SD cards. The control relay server 102 supports the dynamic addition of distributed storage nodes. It uses the AES-256 encryption algorithm to protect the stored data, combined with CRC check and bad block management mechanisms to ensure the integrity of configuration information, command data and logs. It is compatible with multiple operating systems such as Windows, Linux, Android, and iOS, and can be connected to various terminal devices without additional drivers. It effectively supports the stable operation of the remote device projection and interaction system, and improves deployment flexibility and maintenance convenience.

[0139] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments provided herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A remote device screen projection and interaction system, characterized in that, This includes remote execution devices, control relay servers, mobile relay terminals, and display terminals; The remote execution device is used to collect local audio and video data streams for human-computer interaction and establish a long connection with the control relay server. The control relay server is used to manage the sessions of all online devices, generate connection information for the remote execution device, and relay the audio and video data streams through the long connection. The mobile relay terminal is used to establish a session with the remote execution device based on the connection information, and to establish a local wireless projection connection with the display terminal. It also receives the audio and video data stream from the remote execution device with which the session exists, and sends it to the display terminal for presentation through the local wireless projection connection.

2. The remote device projection and interaction system according to claim 1, characterized in that, The mobile relay terminal is also used to collect near-end voice commands and convert them into the control commands, and send the control commands to the remote execution device with a session through the control relay server.

3. The remote device projection and interaction system according to claim 2, characterized in that, The control commands include one or more of the following: simulated keyboard operation commands, simulated mouse operation commands, application start / stop commands, media playback control commands, and system setting adjustment commands.

4. The remote device projection and interaction system according to claim 1, characterized in that, The mobile relay terminal integrates a voice processing module, which includes a voice acquisition unit, a voice recognition unit, and a command parsing unit. The voice acquisition unit is used to capture voice commands in the near-end environment; The speech recognition unit is used to convert the speech command into text information, which supports local recognition or cloud recognition service calls; The instruction parsing unit is used to convert the text information into control instructions containing operation types and parameters according to preset mapping rules.

5. The remote device projection and interaction system according to claim 1, characterized in that, The remote execution device includes a data acquisition module, an encoding and compression module, a communication module, and an instruction execution module; The data acquisition module is used to capture the audio and video data streams locally for human-computer interaction in real time; The encoding and compression module is used to encode and compress the acquired audio and video data streams using audio and video encoding standards. The communication module is used to establish the long connection with the control relay server through a network transmission protocol, and to receive the audio and video data stream transmission instructions and the control instructions; The instruction execution module is used to call the operating system interface to simulate the execution of user input or program operations corresponding to the control instructions.

6. The remote device projection and interaction system according to claim 1, characterized in that, The long connection is a full-duplex communication channel based on a network transmission protocol. The long connection includes a heartbeat keep-alive mechanism and an automatic reconnection mechanism after disconnection. The heartbeat keep-alive mechanism and the automatic reconnection mechanism after disconnection are used for the remote execution device to maintain real-time data transmission when it is located in a private network. The long connection is also used for the remote execution device to receive and execute control commands forwarded by the control relay server.

7. The remote device projection and interaction system according to claim 1, characterized in that, The control relay server includes a session management module, a connection information generation module, a data relay module, and a command routing module; The session management module is used to manage the connection status, device identification, and session association relationships of all online devices; The connection information generation module is used to generate dynamic identification information, including device identifier, server address and session token, for the successfully registered remote execution device; The data relay module is used to receive the audio and video data stream encoded by the remote execution device and forward it to the associated mobile relay terminal; The instruction routing module is used to receive control instructions from the mobile relay terminal and forward them to the corresponding remote execution device according to the device identifier.

8. The remote device projection and interaction system according to claim 1, characterized in that, The local wireless screen projection connection between the mobile relay terminal and the display terminal is established based on DLNA, AirPlay, Miracast or Chromecast protocols; The network communication connection between the mobile relay terminal and the control relay server is established based on the mobile Internet or wireless local area network and supports encrypted data transmission.

9. The remote device projection and interaction system according to claim 1, characterized in that, The connection information includes the device identifier of the remote execution device, the access address of the control relay server, and session token security verification information. After scanning and parsing the connection information, the mobile relay terminal initiates a session establishment request and device association process.

10. The remote device projection and interaction system according to claim 1, characterized in that, The connection information can be any one of the following: QR code, near-field communication tag, Bluetooth broadcast frame, ultrasonic beacon, or short URL. Any one of the connection information is used by the mobile relay terminal to parse and establish a session with the control relay server.

Images