A method for implementing reverse control of wired screen mirroring support

Abstract

The application discloses a kind of wired screen support reverse control implementation method, comprising: screen receiving end reads own USB mode type;If USB mode type is slave, USB port is switched to slave mode, USB device driving is started, and host access is waited;If USB mode type is host, USB port is switched to host mode, USB Host driving is started, and slave access is waited;USB mode type is host, and the system type of smart terminal is detected and identified, and corresponding function is started according to system type;USB mode type is slave, and whether to request HID feature data to slave is set HID working mode;USB mode type is host, and the slave that access is apple terminal, and HID working mode is set to apple slave;Screen receiving end starts screen, and starts HID peripheral event reception and processing.The application realizes the USB composite interface of HID and screen interface, identifies android and apple system, and realizes reverse control to smart terminal according to host-slave mode type, HID working mode type.

Description

Technical Field

[0001] This invention relates to the field of screen mirroring technology, and in particular to a method for implementing wired screen mirroring with reverse control. Background Technology

[0002] Due to their powerful computing capabilities, vast array of applications, and rich multimedia content, smartphones have become indispensable personal electronic devices. Consequently, the application scenarios of screen mirroring from smartphones to portable screens, televisions, projectors, and other devices are increasingly common, such as in conference displays, educational displays, and selfie sticks. Current technology allows for the convenient synchronization of content (meeting content, multimedia files, game screenshots, movies, and videos) from smart terminals like smartphones or tablets onto any other display device, providing greater convenience and enjoyment for life, work, and entertainment. However, with increasingly diverse interactive needs, facilitating convenient human-computer interaction with the smartphone becomes crucial for user experience in screen mirroring scenarios. For example, when a user operates the interface on the mirrored display device, they can in turn control the user interface of the smart terminal to perform synchronized operations.

[0003] Existing mobile phone screen mirroring and control methods are mainly achieved through two approaches:

[0004] (1) Using Bluetooth HID protocol

[0005] The screen mirroring receiver integrates Bluetooth HID functionality. The mobile phone connects to the Bluetooth device in the screen mirroring receiver via Bluetooth. The screen mirroring receiver then sends human-computer interaction data to the mobile phone via the Bluetooth device to achieve interaction and reverse control with the mobile phone.

[0006] (2) via USB HUB expansion interface

[0007] Add a USB hub to the screen mirroring receiver, connect the mobile phone to the USB hub as the host, and connect the USB and USB HID devices (human-machine interface devices) of the screen mirroring receiver to the USB interface of the hub as slave devices. The USB HID devices directly send human-machine interaction data to the mobile phone to realize interaction and reverse control with the mobile phone.

[0008] Existing solutions have the following drawbacks: 1. Both Bluetooth HID and USB hub expansion interfaces require corresponding hardware modules, increasing hardware costs. 2. Bluetooth HID is susceptible to 2.4GHz band signal interference, leading to packet loss, or may interfere with Wi-Fi devices. 3. Bluetooth HID requires the phone to pair with a Bluetooth device, resulting in poor usability. 4. With USB hub expansion interfaces, the phone can only act as a USB host, not a USB slave. When the phone acts as a USB host, it needs to supply power to the peripheral, accelerating battery drain. 5. Neither method can distinguish between Android and Apple phones at the HID device end. Since these two types of phones handle touch-based HID differently, compatibility issues arise. Summary of the Invention

[0009] This invention provides a method for wired screen mirroring with reverse control. In scenarios where a mobile phone is connected to a screen mirroring device via USB for screen mirroring, the phone can be controlled via the screen mirroring receiver without increasing costs. It supports multiple input devices such as a mouse, keyboard, and touchscreen. The screen mirroring receiver reads input signals from these devices and then sends them to the phone via USB to achieve reverse control. This method allows for accurate touch control of the phone on touchscreens of different sizes and orientations, while also resolving compatibility issues between Apple and Android phones.

[0010] A method for supporting reverse control in wired screen mirroring includes the following steps:

[0011] Step 1: The receiving end reads its own USB mode type and performs subsequent steps according to the USB mode type.

[0012] Step 2: If the USB mode type is slave mode, switch the USB port of the screen mirroring receiver to slave mode, start the USB device driver, and wait for the USB host device to connect; if the USB mode type is host mode, switch the USB port of the screen mirroring receiver to host mode, start the USB host driver, and wait for the USB slave device to connect.

[0013] Step 3: When the USB mode type is host mode, detect and identify the smart terminal system type, and start the corresponding function according to the smart terminal system type;

[0014] Step 4: When the USB mode type is slave mode, set the HID working mode according to whether the connected smart terminal acting as the master requests HID feature data from the screen mirroring receiver acting as the slave during the USB enumeration process; if the USB mode type is master mode and the connected slave is an Apple terminal, then set the HID working mode stored in the screen mirroring receiver to Apple slave mode.

[0015] Step 5: The screen mirroring receiver starts the screen mirroring process to begin screen mirroring, and at the same time starts the HID event handler to begin receiving and processing peripheral events.

[0016] Furthermore, step 3, which involves detecting and identifying the type of the smart terminal system and activating the corresponding function based on the type of the smart terminal system, further includes: if the inserted smart terminal's USB Vendor ID is detected to be 0x05AC, then the smart terminal is identified as an Apple terminal, the USB port of the screen mirroring receiver is switched to USB slave mode, and the USB device driver is activated; if the inserted smart terminal is detected to be a non-Apple terminal, then the screen mirroring receiver sends a control command through the USB port to activate the smart terminal's AOA HID function.

[0017] Furthermore, step 4, setting the HID working mode, further includes: if the connected smart terminal, acting as the host, requests HID feature data from the screen-sharing receiver, acting as the slave, during the USB enumeration process, then the HID working mode stored on the screen-sharing receiver is set to Android slave mode; if no HID feature data is requested from the slave during the USB enumeration process, then the HID working mode stored on the screen-sharing receiver is set to Apple slave mode; after the AOA mode of the screen-sharing receiver is successfully started, the HID working mode is set to Android AOA mode.

[0018] Furthermore, the USB device driver implements the combined functions of multiple USB interfaces, including a vendor-defined class for screen mirroring, a mouse HID class, a keyboard HID class, and a touch device HID class. The vendor-defined class includes a Bulk transmission input endpoint and a Bulk transmission output endpoint, used to complete the screen mirroring protocol and data transmission. The USB device driver combines three HID class interfaces based on the vendor-defined class to implement the HID functions of the mouse, keyboard, and touch device, respectively. Through class interface combination, the combined functions of the screen mirroring class interface and multiple HID class interfaces are implemented on the USB port of the screen mirroring receiving end.

[0019] Furthermore, the HID class interface definition for touch devices includes a report descriptor for the HID device, describing parameters such as the width, height, and number of touch points of the touch device. When the smart terminal acting as the host enumerates the screen mirroring receivers acting as slaves, it obtains the report descriptor to identify the HID function provided by the screen mirroring receiver. It also includes HID feature data describing the touch device model and sensitivity parameters. The screen mirroring receiver determines whether the currently connected smart terminal is an Apple terminal or an Android terminal by whether the smart terminal reads the HID feature data during the enumeration process.

[0020] Furthermore, after completing the master-slave mode settings for the screen mirroring receiver and the smart terminal, and identifying the system type of the smart terminal, the screen mirroring function is activated. The screen mirroring receiver receives the screen mirroring video stream, decodes it, and displays the video. The screen mirroring receiver then begins to accept input events from the HID peripheral.

[0021] If the input event received from the HID peripheral comes from the touch device, the touch coordinates are mapped through the coordinate transformation program, converting the coordinate values ​​received from the touch device into coordinate values ​​in the HID coordinate system, and mapping the touch coordinates to the coordinate system of the smart terminal.

[0022] If the input event received from the HID peripheral comes from the keyboard or remote control, it is converted into a keyboard HID message according to the HID specification; if the input event received from the HID peripheral comes from the mouse, it is converted into a mouse HID message according to the HID specification.

[0023] Furthermore, if the input event received from the HID peripheral comes from a touch device and the connected smart terminal is an Android system, the event after coordinate transformation is packaged into a multi-touch HID message according to the HID specification, and the length of the multi-touch message is determined by the number of touch points.

[0024] If the input event received from the HID peripheral comes from a touch device and the connected mobile phone is an Apple system, the event after coordinate transformation will be packaged into an HID message in absolute coordinate mouse mode.

[0025] Furthermore, the system type of the smart terminal is determined. If the smart terminal is an Apple system, or an Android system and the USB port of the screen mirroring receiver has been switched to slave mode, then the USB Device driver is selected to send HID messages to the smart terminal. If the smart terminal is an Android system and the USB port of the screen mirroring receiver has been switched to host mode, then the HID messages are sent to the smart terminal via the AOA protocol. The screen mirroring receiver sends the HID messages to the smart terminal via the USB port and performs reverse control.

[0026] Technical effects of the present invention:

[0027] 1. The screen mirroring receiver directly implements the functionality of a USB composite interface that combines HID and screen mirroring capabilities. This means it creates a "HID + screen mirroring interface" USB composite device on a single USB port, eliminating the need for a hub. The phone can directly connect to the screen mirroring receiver via a single USB port to achieve both screen mirroring and HID reverse control functions, without requiring additional hardware or operations. This reduces hardware costs and improves ease of use.

[0028] 2. When the screen mirroring receiver acts as a USB slave, it implements a USB device driver that combines various HID and screen mirroring interfaces, and distinguishes between Android and Apple phones based on the actions taken by the phone (as the USB host) during enumeration, so as to perform different subsequent processing.

[0029] 3. When the receiving end acts as a USB host, it distinguishes the phone system type based on the phone's USB Vendor ID and implements the corresponding HID interaction function according to the phone type.

[0030] 4. The screen mirroring receiver is software-configurable to support both USB slave and USB host modes, allowing users to switch between USB master and slave modes according to their usage scenarios, thereby improving mobile phone compatibility. Attached Figure Description

[0031] Figure 1 This is a system block diagram of the present invention, which supports reverse control for wired screen mirroring.

[0032] Figure 2 This is a flowchart of the wired screen mirroring support for reverse control process of the present invention;

[0033] Figure 3 This is a flowchart of the adaptive HID event processing of the screen-sharing receiver in this invention;

[0034] Figure 4 This is a schematic diagram of the touch coordinate transformation of the present invention; Detailed Implementation

[0035] This invention provides a composite device that supports multiple HID device types and screen mirroring functionality interfaces on a single USB port in the screen mirroring receiver. Unlike existing technologies where the USB port of the screen mirroring receiver only has a screen mirroring interface, this invention adds an HID interface to this USB port. The software implements USB master / slave mode configuration, adaptively and dynamically adjusting the HID protocol between the phone and the screen mirroring receiver based on the USB master / slave mode and the phone type, thereby achieving USB wired screen mirroring and reverse control functionality compatible with both Android and Apple phones. The embodiments use Android and Apple phones as examples; however, those skilled in the art will understand that this solution is applicable to other smart terminal devices operating on Android or Apple operating systems, such as tablets.

[0036] This invention adds a composite HID interface to the existing USB wired screen mirroring, adaptively switching between USB modes and adaptively selecting HID event handling methods. The system block diagram is shown below. Figure 1 .

[0037] The meanings of the relevant parameters in Figure (1) are explained as follows:

[0038] HID: Human-Machine Interface Device

[0039] HID feature: A feature item in a human-machine interface device, used to convey device information and status.

[0040] VID: USB Manufacturer ID

[0041] AOA: Android Open Accessories Protocol

[0042] USB Device Driver: USB Slave Mode Driver

[0043] USB Host Driver: USB Host Mode Driver

[0044] The screen mirroring application on the mobile / smart terminal device transmits screen mirroring data to its own USB module. The mobile USB module then sends the screen mirroring data to the screen mirroring receiver. After receiving the data, the screen mirroring receiver's USB driver module transmits it to the screen mirroring processing program for conversions such as video information and screen display information. Finally, the audio and video data are transmitted to the display peripherals. When HID peripherals on the screen mirroring receiver, such as keyboards, mice, and touchscreens, generate HID events, they are processed by the HID event receiving program and the HID event handling program, respectively. The screen mirroring processing program passes the video frame and screen information (width, height, and position) to the HID event handling program. The HID event handling program uses this information to perform coordinate transformations for touch events, generates HID messages, and sends these messages to the mobile / smart terminal device's USB module via the screen mirroring receiver's USB driver module. The mobile / smart terminal device parses and processes the HID messages and generates a system response.

[0045] After receiving input from the HID peripheral of the touch device, the screen mirroring receiver needs to receive video and screen display information from the screen mirroring process at the HID event handler when controlling the mobile phone / smart terminal device. This is because coordinate transformation is required. The touch events received from the touch device are coordinates on the screen, but the mobile phone screen is not full-screen or the orientation is not the same as the screen orientation. For example, when the mobile phone / smart terminal device screen is displayed in the middle of the screen, it should only respond to touch events within the display area of ​​the mobile phone / smart terminal device screen, and the received touch coordinates need to be converted into the screen coordinates of the mobile phone / smart terminal device. Therefore, the coordinate transformation algorithm needs to know the screen width and height, the width and height of the mobile phone / smart terminal device screen, and its position, etc., in order to calculate the coordinates of the touch point on the mobile phone / smart terminal device screen, and then generate an HID message to be transmitted to the mobile phone / smart terminal device to achieve reverse control.

[0046] The implementation of the screen mirroring receiver with a composite USB HID interface and the adaptive switching of USB modes is shown in Figure (2). The meanings of the relevant parameters in Figure (2) are explained as follows:

[0047] `mode`: HID working mode, a variable in the screen mirroring receiver program, used to record the HID working mode of the currently connected device. `mode` is the final working mode of the USB port determined during operation based on the "USB mode type" of the screen mirroring receiver and the actual type of connected device. It can be dynamically adjusted and set according to actual working conditions. The following embodiment uses a mobile phone as an example; those skilled in the art will understand that the solution is applicable to other smart terminal devices with Android or Apple operating systems, such as tablets.

[0048] Step 1: The receiving end reads its own USB mode type and performs subsequent steps according to the USB mode type.

[0049] Step 2: If the USB mode type is slave mode, switch the USB port of the screen mirroring receiver to slave mode, start the USB device driver, and wait for the USB host device to connect; if the USB mode type is host mode, switch the USB port of the screen mirroring receiver to host mode, start the USB host driver, and wait for the USB slave device to connect.

[0050] Step 3: When the USB mode type is host mode, detect and identify the smart terminal system type, and start the corresponding function according to the smart terminal system type;

[0051] The function of starting the corresponding function according to the type of the smart terminal system further includes: if the USB Vendor ID of the inserted smart terminal is detected to be 0x05AC, then the smart terminal is identified as an Apple terminal, the USB port of the screen mirroring receiver is switched to USB slave mode, and the USB device driver is started; if the inserted smart terminal is detected to be a non-Apple terminal, then the screen mirroring receiver sends a control command through the USB port to start the AOA HID function of the smart terminal.

[0052] Step 4: When the USB mode type is slave mode, set the HID working mode according to whether the connected smart terminal acting as the master requests HID feature data from the screen mirroring receiver acting as the slave during the USB enumeration process; if the USB mode type is master mode and the connected slave is an Apple terminal, then set the HID working mode stored in the screen mirroring receiver to Apple slave mode.

[0053] If the connected smart terminal, acting as the host, requests HID feature data from the screen mirroring receiver, acting as the slave, during the USB enumeration process, then the HID working mode stored on the screen mirroring receiver is set to Android slave mode; if no HID feature data is requested from the slave during the USB enumeration process, then the HID working mode stored on the screen mirroring receiver is set to Apple slave mode; after the AOA mode of the screen mirroring receiver is successfully started, the HID working mode is set to Android AOA mode.

[0054] Step 5: The screen mirroring receiver starts the screen mirroring process to begin screen mirroring, and at the same time starts the HID event handler to begin receiving and processing peripheral events.

[0055] The more specific implementation process is as follows:

[0056] S1: The screen mirroring receiver application reads the USB mode type of the screen mirroring receiver from the system parameters area of ​​the non-volatile memory (the system parameters are stored in the flash memory of the screen mirroring receiver and are read by the screen mirroring receiver program; the entire process is initiated by the screen mirroring receiver program). The USB mode type parameter is set by the screen mirroring receiver device before leaving the factory or by the user in the user interface and is stored in the system parameters area of ​​the non-volatile memory.

[0057] S2: If the USB mode type is slave mode, switch the USB port of the USB screen mirroring receiver to slave mode and start the USB device driver. The USB device driver implements multiple composite functions of USB interfaces, including a vendor-specific class for screen mirroring, an HID class for mouse functionality, an HID class for keyboard functionality, and an HID class for touchscreen functionality. The specific implementation method is as follows:

[0058] (1) The USB device driver registers a Vendor-specific class interface, which provides a Bulk In Endpoint and a Bulk Out Endpoint for screen mirroring protocol and data transmission. The Endpoint descriptor and Interface descriptor of the interface are represented by the following pseudocode:

[0059]

[0060]

[0061] Here, um_intf is the descriptor content of the interface, which describes the interface as a user-defined function (USB_CLASS_VENDOR_SPEC). It contains two Bulk endpoints: fs_um_source_desc describes the input endpoint, and fs_um_sink_desc describes the output endpoint.

[0062] (2) The USB device driver, based on the Vendor-specific class interface, further incorporates three HID class interfaces to implement the HID functions of the mouse, keyboard, and touch device, respectively. Taking the touch device as an example, its interface function descriptor is represented by the following pseudocode:

[0063]

[0064]

[0065] Among them, `report_desc` is the report descriptor of the HID device, which describes parameters such as the width, height, and number of touch points of the touch device. Smart terminals obtain these function descriptors to identify the functions provided by the device when enumerating USB slave devices. `us_feature_data_0` and `us_feature_data_1` describe parameters such as the touch device model and sensitivity. The receiving end can determine whether the currently connected smart terminal is an Apple or Android phone by whether the smart terminal reads this data during the enumeration process. Mouse and keyboard HID interfaces also have corresponding function descriptors.

[0066] (3) Using the above method, the USB device can implement the combined functions of screen mirroring interface and multiple HID interface on one USB port of the screen mirroring receiver.

[0067] S3: If the USB mode type is host mode, then switch the USB port of the USB screen mirroring receiver to host mode and start the USB Host driver.

[0068] S4 / S5: The screen mirroring receiver is waiting for a USB device to connect.

[0069] S6: If the USB mode type is host mode and the inserted phone's USB Vendor ID is detected as 0x05AC, then this phone is identified as an Apple phone. The USB port of the screen mirroring receiver is switched to USB slave mode, and the USB device driver is started. The implementation method of the USB device driver is the same as that described in S2. Due to the characteristics of Apple phones, reverse control can only be achieved by switching the USB port of the screen mirroring receiver to USB slave mode. The current solution can adaptively switch through dynamic identification.

[0070] S7: If the USB mode type is host mode and the inserted phone is detected to be a non-Apple phone, the screen mirroring receiver will start the AOA HID function through the host control command. AOA HID is a function supported by the Android system.

[0071] S8: If the USB mode type is slave mode and the connected host requests feature data in the HID function descriptor from the slave during the USB enumeration process, then set the HID working mode to Android slave mode.

[0072] S9: If the USB mode type is slave mode and the connected host does not request HID feature data from the slave during USB enumeration, then set the HID working mode to Apple slave mode. If the USB mode type is host mode and the connected slave is an Apple mobile phone, then set the HID working mode to Apple slave mode.

[0073] S10: After successfully starting AOA mode in S7, set the HID working mode to Android AOA mode.

[0074] S11: Start the screen mirroring program to begin the phone's screen mirroring function, and at the same time start the HID event handler to begin receiving and processing peripheral events.

[0075] The HID working mode (mode) indicates the final HID working mode of the screen mirroring receiver, which is determined based on the USB mode type parameter of the screen mirroring receiver and the type of the currently inserted smart terminal device. The correspondence between the above implementation steps can be summarized in the following table:

[0076]

[0077] In this article, the screen mirroring receiver integrates screen mirroring and display functions, collectively referred to as the screen mirroring receiver. The aforementioned implementation allows the user's smart terminal (phone) to connect to the screen mirroring receiver via USB. The receiver then switches between master and slave modes, identifies whether the connected phone is an Android or Apple phone, and accordingly sets the phone to the mode corresponding to the screen mirroring receiver.

[0078] Because different mobile phone operating systems support touch-based HID devices (HID devices here refer only to touchscreens, excluding HID devices such as mice and keyboards; Android phones and Apple phones process touchscreen signals differently), it is necessary to perform corresponding HID event processing and conversion based on the current USB mode and the type of mobile phone connected, in order to ensure compatibility with mobile phones on different operating systems.

[0079] The adaptive HID event processing flow of the receiving end on the screen mirroring device is as follows: Figure 3 As shown:

[0080] S12: After completing the master-slave mode settings for the screen mirroring receiver and the mobile phone, and identifying the mobile phone system type, the screen mirroring function is activated.

[0081] S13: The receiving end on the same screen begins reading input events from the HID peripheral.

[0082] S14: After receiving the screen mirroring video stream, the receiving end decodes and displays the video. If the width or height of the video frame changes, it generates the changed width and height information through touchscreen coordinate transformation. The width and height of the video can be obtained through the screen mirroring protocol or video driver detection. The receiving end simultaneously performs coordinate transformation on the width and height of the screen.

[0083] S15: If the input event received from the HID peripheral is from a touch device, a coordinate transformation program is used to map the touch coordinates, converting the coordinate values ​​received from the touch device into coordinate values ​​in the HID coordinate system. The transformation algorithm needs to calculate the display area of ​​the mobile phone screen based on the width and height of the video frame, the width and height of the screen, and the orientation, thereby mapping the touch coordinates to the mobile phone's coordinate system. Figure 4 This is a diagram illustrating touch coordinate transformation. The transformation algorithm converts the coordinate values ​​received from the touch device into coordinate values ​​in the HID coordinate system. The specific implementation method is illustrated using a landscape display scenario as an example:

[0084] Figure 4 In the diagram, w and h represent the width and height of the touch device; x and y represent the coordinates of the touch point received from the touch device; w v The width of the video is obtained from the screen mirroring protocol and the video decoding end; w1 and h1 are the width and height of the HID coordinate system. The coordinate conversion program mainly uses this information to calculate the current touch coordinates converted to HID coordinates x1 and y1. The specific algorithm is as follows:

[0085]

[0086] The smart terminal obtains the width and height of the HID coordinate system through the HID descriptor, and can then map the coordinates of the HID coordinate system to the coordinates of the mobile phone screen. This part is done by the smart terminal.

[0087] S16: If an input event from a HID peripheral is received from the keyboard or remote control, it is converted into a keyboard HID message according to the HID specification.

[0088] S17: If an input event from a HID peripheral is received from a mouse, it is converted into a mouse HID message according to the HID specification.

[0089] More specifically, S15 includes,

[0090] S18: If the input event received from the HID peripheral comes from a touch device and the connected phone is an Android system, then the event in S15, after coordinate transformation, is packaged into a multi-touch HID message according to the HID specification. The length of the multi-touch message is determined by the number of touch points, and the general format is as follows:

[0091] Byte 0: Report ID

[0092] Byte 1: Validity and status of touch point 1

[0093] Byte 2: Index ID of touch point 1

[0094] Byte 3 / Byte 4: Low and high bytes of the X-axis coordinates of touch point 1

[0095] Byte 5 / Byte 6: Low and high bytes of the Y-axis coordinates of touch point 1

[0096] Byte 7 / Byte 8: Low and high bytes of the width of touch point 1

[0097] Byte 9 / Byte 10: Low and high bytes of the height of touch point 1

[0098] …(Other touch point information)

[0099] Byte N: Total number of valid touch points

[0100] Byte N+1: Low byte of scan time

[0101] Byte N+2: High byte of scan time

[0102] S19: If the input event received from the HID peripheral comes from a touch device and the connected mobile phone is an Apple system, then the event in S15 after coordinate transformation is packaged into an absolute coordinate mouse mode HID message. The message is 6 bytes long, and the description of each byte is as follows:

[0103] Byte 0: Report ID

[0104] Byte 1: Touch Switch (touch state)

[0105] Byte 2: Low byte of X-axis coordinate

[0106] Byte 3: High byte of X-axis coordinate

[0107] Byte 4: Low byte of Y-axis coordinate

[0108] Byte 5: High byte of Y-axis coordinate

[0109] S20: If the phone is an Apple system, or the phone is an Android system and the USB receiving end of the screen mirroring is in slave mode, then select the USB Device driver to send HID messages to the phone.

[0110] S21: If the phone is an Android system and the USB receiver for screen mirroring is in host mode, then select to send HID messages to the phone via the AOA protocol.

[0111] S22: The screen mirroring receiver sends HID messages to the mobile phone via the USB interface.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for implementing reverse control via wired screen mirroring, comprising the following steps: Step 1: The receiving end reads its own USB mode type and performs subsequent steps according to the USB mode type. Step 2: If the USB mode type is slave mode, switch the USB port of the screen mirroring receiver to slave mode, start the USB device driver, and wait for the USB host device to connect; if the USB mode type is host mode, switch the USB port of the screen mirroring receiver to host mode, start the USB host driver, and wait for the USB slave device to connect. Step 3: When the USB mode type is host mode, detect and identify the smart terminal system type, and start the corresponding function according to the smart terminal system type; Step 4: When the USB mode type is slave mode, set the HID working mode according to whether the connected smart terminal acting as the master requests HID feature data from the screen mirroring receiver acting as the slave during the USB enumeration process; if the USB mode type is master mode and the connected slave is an Apple terminal, then set the HID working mode stored in the screen mirroring receiver to Apple slave mode. Step 5: The screen mirroring receiver starts the screen mirroring process to begin screen mirroring, and at the same time starts the HID event handler to begin receiving and processing peripheral events.

2. The method for implementing wired screen mirroring with reverse control as described in claim 1, wherein step 3, detecting and identifying the smart terminal system type and activating the corresponding function according to the smart terminal system type, further includes: if the inserted smart terminal's USB Vendor ID is detected to be 0x05AC, then the smart terminal is identified as an Apple terminal, the USB port of the screen mirroring receiver is switched to USB slave mode, and the USB device driver is activated; if the inserted smart terminal is detected to be a non-Apple terminal, then the screen mirroring receiver sends a control command through the USB port to activate the smart terminal's AOA HID function.

3. The method for implementing wired screen mirroring with reverse control as described in claim 2, wherein setting the HID working mode in step 4 further includes: if the connected smart terminal acting as the host requests HID feature data from the screen mirroring receiver acting as the slave during the USB enumeration process, then the HID working mode stored on the screen mirroring receiver is set to Android slave mode; if no HID feature data is requested from the slave during the USB enumeration process, then the HID working mode stored on the screen mirroring receiver is set to Apple slave mode; after the AOA mode of the screen mirroring receiver is successfully started, the HID working mode is set to Android AOA mode.

4. The wired screen mirroring method supporting reverse control as described in claim 3, wherein the USB device driver implements the composite function of multiple USB interfaces, including a manufacturer-defined class for screen mirroring, a mouse function HID class, a keyboard function HID class, and a touch device HID class. The manufacturer-defined class includes a Bulk transmission input endpoint and a Bulk transmission output endpoint for completing the screen mirroring protocol and data transmission. The USB device driver composites three HID class interfaces based on the manufacturer-defined class to implement the HID functions of the mouse, keyboard, and touch device, respectively. Through class interface composite, the composite function of the screen mirroring class interface and multiple HID class interfaces is implemented on the USB port of the screen mirroring receiving end.

5. The wired screen mirroring method supporting reverse control as described in claim 4, wherein the touch device HID class interface definition includes a report descriptor for the HID device, describing parameters such as the width, height, and number of touch points of the touch device. When the smart terminal acting as the host enumerates the screen mirroring receiver acting as the slave, it obtains the report descriptor to identify the HID function provided by the screen mirroring receiver. It also includes HID feature data describing the touch device model and sensitivity parameters. The screen mirroring receiver determines whether the currently connected smart terminal is an Apple terminal or an Android terminal by whether the smart terminal reads the HID feature data during the enumeration process.

6. The method for implementing wired screen mirroring with reverse control as described in any one of claims 1-5, comprising: After completing the master-slave mode settings for the screen mirroring receiver and the smart terminal, and identifying the system type of the smart terminal, the screen mirroring function is activated. The screen mirroring receiver receives the screen mirroring video stream, decodes it, and displays the video. The screen mirroring receiver then begins to accept input events from the HID peripheral. If the input event received from the HID peripheral comes from the touch device, the touch coordinates are mapped through the coordinate transformation program, converting the coordinate values ​​received from the touch device into coordinate values ​​in the HID coordinate system, and mapping the touch coordinates to the coordinate system of the smart terminal. If an input event from a HID peripheral is received from the keyboard or remote control, it is converted into a keyboard HID message according to the HID specification. If an input event from an HID peripheral is received from the mouse, it is converted into a mouse HID message according to the HID specification.

7. The method for implementing wired screen mirroring with reverse control as described in claim 6, further comprising: If the input event received from the HID peripheral comes from a touch device and the connected smart terminal is an Android system, the event after coordinate transformation will be packaged into a multi-touch HID message according to the HID specification. The length of the multi-touch message is determined by the number of touch points. If the input event received from the HID peripheral comes from a touch device and the connected mobile phone is an Apple system, the event after coordinate transformation will be packaged into an HID message in absolute coordinate mouse mode.

8. The method for implementing wired screen mirroring with reverse control as described in claim 7, further comprising: The system determines the type of the smart terminal. If the smart terminal is an Apple system, or an Android system and the USB port of the screen mirroring receiver has been switched to slave mode, then the USB Device driver is selected to send HID messages to the smart terminal. If the smart terminal is an Android system and the USB port of the screen mirroring receiver has been switched to host mode, then the HID messages are sent to the smart terminal via the AOA protocol. The screen mirroring receiver sends the HID messages to the smart terminal via the USB port and performs reverse control.

Images