Cross-apparatus control method, electronic apparatus and storage medium

Abstract

Provided in the embodiments of the present application are a cross-apparatus control method, an electronic apparatus and a storage medium. By means of an existing data transmission channel in a screen-mirroring connection state, first peripheral operation data generated by a user's interaction operation on a peripheral apparatus matched with a screen-mirroring display apparatus is transmitted to a screen-mirroring device; the screen-mirroring device forwards the first peripheral operation data to a screen-mirroring source-end apparatus; a screen-mirroring auxiliary application of the screen-mirroring source-end apparatus converts the first peripheral operation data on the basis of hardware parameters of the screen-mirroring display apparatus and the screen-mirroring source-end apparatus, so as to obtain second peripheral operation data that is adapted to the hardware status thereof, and then transmits same to the screen-mirroring device; and the screen-mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain a HID message that the screen-mirroring source-end apparatus can directly use, and transmits the HID message to the screen-mirroring source-end apparatus for response. The present application achieves cross-apparatus control over the screen-mirroring source-end apparatus during explanations centered on the screen-mirroring display apparatus, thus ensuring the continuity of multi-person communication, and improving the information transmission effect.

Description

Cross-device control methods, electronic devices and storage media

[0001] This application claims priority to Chinese Patent Application No. 202410563401.1, filed on May 8, 2024, entitled "Cross-Device Control Method, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communication technology, and more particularly to cross-device control methods, electronic devices, and storage media. Background Technology

[0003] With the continuous development and popularization of information technology, various types of electronic devices are constantly emerging, and the ways in which these devices connect have also become more diverse. For example, in conference rooms built based on information technology, interactive whiteboards, microphones, speakers, mice, keyboards, and other electronic devices are usually configured. These devices are interconnected and work together as peripherals, with the interactive whiteboard at the core, to enable information collection, transmission, and presentation during multi-person communication.

[0004] When using interactive whiteboards to present information, the information source may not be the interactive whiteboard itself, but rather other terminal devices in the meeting room. Screen sharing technology is used to replicate the information from the source device on the display device. In this application scenario, the interactive whiteboard acts as the display device, and the other terminal devices are the source devices. To achieve information transfer between the source and display devices, a dedicated intermediate device is typically needed to establish a data channel for data transmission. This intermediate device, specifically designed to establish data channels between different devices for information replication, can be named a screen projector, screen sharer, wireless screen projector, or wireless screen sharer, etc. The term "screen sharer" will be used consistently from now on.

[0005] The inventors analyzed the working process of existing screen sharing devices and found that, during information presentation, the focal device is the screen sharing display device, but the source of the information content is the screen sharing source device. In multi-person communication scenarios, when presenting information based on pre-prepared presentation slides, the speaker can only maintain the screen sharing source device as the center, or move between the screen sharing source device and the screen sharing display device, in order to operate and switch the information presentation content on the screen sharing source device. They cannot maintain the screen sharing display device as the focal point of information presentation for the presentation, which affects the continuity of multi-person communication and reduces the effectiveness of information transmission.

[0006] Application content

[0007] The present application provides a cross-device control method, an electronic device, and a storage medium, so as to solve the technical problem that in the existing screen sharing process, it is impossible to maintain the focus of information presentation centered on the screen sharing display device for explanation, which affects the continuity of multi-person communication and reduces the effect of information transmission.

[0008] In a first aspect, an embodiment of the present application provides a cross-device control method for a screen sharing device. The cross-device control method includes:

[0009] In the screen sharing connection state, receive first peripheral operation data sent by a screen sharing display device through a wireless network. The first peripheral operation data is generated by a peripheral device of the screen sharing display device;

[0010] Send the first peripheral operation data to a screen sharing source device through a wired interface, so that a screen sharing auxiliary application of the screen sharing source device converts the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain second peripheral operation data and send it to the screen sharing device;

[0011] Receive the second peripheral operation data, and encapsulate the second peripheral operation data based on the USB HID protocol to obtain a HID message;

[0012] Send the HID message to the screen sharing source device for the screen sharing source device to respond according to the HID message.

[0013] As described above, through the existing data transmission channel in the screen sharing connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen sharing display device is sent to the screen sharing device. The screen sharing device forwards the first peripheral operation data to the screen sharing source device. The screen sharing auxiliary application in the screen sharing source device converts it according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain second peripheral operation data adapted to its own hardware state and then sends it to the screen sharing device. The screen sharing device encapsulates the second peripheral operation data according to the USB HID protocol to obtain a HID message directly available to the screen sharing source device, and sends the HID message to the screen sharing source device for response, realizing cross-device control of the screen sharing source device by the screen sharing display device when explaining centered on the screen sharing display device as the focus of information presentation, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0014] Among them, the cross-device control method further includes:

[0015] When establishing a screen sharing connection, obtain the hardware parameters of the screen sharing display device and send them to the screen sharing source device for caching.

[0016] As described above, the caching of the hardware parameters of the screen sharing source device for the screen sharing display device can be quickly converted according to its own and cached hardware parameters when receiving the first peripheral operation data.

[0017] Among them, the first peripheral operation data is synchronously carried with the hardware parameters of the screen sharing display device.

[0018] As described above, when the screen sharing source device obtains the first peripheral operation data, it obtains the hardware parameters of the screen sharing display device, and accordingly quickly performs conversion in combination with its own hardware parameters.

[0019] Among them, the hardware parameters include display parameters.

[0020] As described above, through the conversion corresponding to the display parameters, it can ensure that the operations detected between two devices with different display screen sizes are accurately and correspondingly responded.

[0021] In a second aspect, an embodiment of the present application provides a cross-device control method for a screen sharing source device. The cross-device control method includes:

[0022] In the screen sharing connection state, receive the first peripheral operation data from the screen sharing device through a wired interface. The first peripheral operation data is received by the screen sharing device from the screen sharing display device through a wireless network connection, and the first peripheral operation data is generated by a peripheral device of the screen sharing display device;

[0023] Convert the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain second peripheral operation data and send it to the screen sharing device, so that the screen sharing device encapsulates the second peripheral operation data based on the USB HID protocol to obtain a HID message and sends it to the screen sharing source device;

[0024] Respond according to the HID message.

[0025] As described above, through the existing data transmission channel in the screen sharing connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen sharing display device is sent to the screen sharing device. The screen sharing device forwards the first peripheral operation data to the screen sharing source device. The screen sharing auxiliary application in the screen sharing source device converts it according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain second peripheral operation data adapted to its own hardware state and then sends it to the screen sharing device. The screen sharing device encapsulates the second peripheral operation data according to the USB HID protocol to obtain a HID message directly available to the screen sharing source device, and sends the HID message to the screen sharing source device for response, realizing cross-device control of the screen sharing source device by the screen sharing display device when explaining with the screen sharing display device as the center of information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0026] This cross-device control method also includes:

[0027] When establishing a screen sharing connection, the hardware parameters obtained by the screen sharing device from the screen sharing display device are acquired and cached.

[0028] Accordingly, the first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device, including:

[0029] The first peripheral operation data is converted according to the conversion relationship between the cached hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0030] The aforementioned caching of hardware parameters of the screen sharing display device by the source device can be quickly converted based on its own and the cached hardware parameters when receiving operation data from the first peripheral device.

[0031] Among them, the first peripheral operation data synchronously carries the hardware parameters of the screen display device;

[0032] Accordingly, the first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device, including:

[0033] The first peripheral operation data is converted according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0034] As described above, by converting the display parameters accordingly, it is possible to ensure that operations detected between two devices with different display screen sizes can be accurately responded to.

[0035] Thirdly, embodiments of this application provide an electronic device, which includes:

[0036] One or more processors;

[0037] Memory, used to store one or more computer programs;

[0038] When one or more computer programs are executed by one or more processors, electronic devices enable cross-device control methods as described in either the first or second aspect.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a cross-device control method as described in either the first or second aspect.

[0040] The electronic device of the third aspect and the computer-readable storage medium of the fourth aspect can be used to perform the cross-device control method provided in any of the above embodiments, and have corresponding functions and beneficial effects. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 is a flowchart of a cross-device control method provided in an embodiment of this application;

[0043] Figure 2 is a schematic diagram of the application scenarios of interactive flat panels and screen sharing devices;

[0044] Figure 3 is a schematic diagram of data processing and transmission during the implementation of the cross-device control method shown in Figure 1;

[0045] Figure 4 is a flowchart of another cross-device control method provided in an embodiment of this application;

[0046] Figure 5 is a flowchart of another cross-device control method provided in an embodiment of this application;

[0047] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of this application. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not the entire structure.

[0049] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0050] The embodiments are described in detail below.

[0051] In the process of information presentation in the screen mirroring scenario, the focus device for information presentation is the screen mirroring display device, but the source that determines the content of information presentation is the screen mirroring source device. When presenting information based on a pre-prepared presentation in a multi-person communication scenario, the presenter can only center around the screen mirroring source device or move between the screen mirroring source device and the screen mirroring display device to switch the information presentation content by operating on the screen mirroring source device. The presenter cannot center around the focus of information presentation, i.e., the screen mirroring display device, to give an explanation, which affects the continuity of multi-person communication and reduces the effect of information transmission. For example, during the information presentation process, when using communication auxiliary functions such as the handwriting annotation of the screen mirroring display device on the screen mirroring display device, when switching the presented information, the presenter can only leave the screen mirroring display device and move to the screen mirroring source device for operation, moving back and forth between two devices at different positions, which obviously affects the continuity of multi-person communication and reduces the effect of information transmission.

[0052] To solve the above technical problems, the embodiment of the present application proposes a cross-device control method. Through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device converts it according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain the HID message directly available to the screen mirroring source device, and sends the HID message to the screen mirroring source device for response, realizing the cross-device control of the screen mirroring source device by the screen mirroring display device when giving an explanation centered around the focus of information presentation, i.e., the screen mirroring display device, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0053] The screen mirroring display device applied in this solution is exemplarily an interactive flat panel. The interactive flat panel can be an integrated device that manipulates the content displayed on the display screen through touch technology and realizes human-computer interaction operations, and integrates one or more functions such as a projector, an electronic whiteboard, a curtain, a speaker, a television, a video conference terminal, and a wireless router. The interactive flat panel can be flat, curved, or a splicing of multiple planes, etc.

[0054] Typically, interactive flat panels are equipped with at least one operating system, including but not limited to Android, Linux, Windows, and Huawei HarmonyOS. This operating system controls and coordinates the interactive flat panel and peripheral devices, enabling the various independent hardware components within the interactive flat panel to work together as a stable whole. An interactive flat panel includes at least one display screen, such as a touch-sensitive display. This touch-sensitive display screen can be an infrared display, capacitive display, resistive display, electromagnetic display, or pressure-sensitive display, etc. On a touch-sensitive display screen, users can perform touch operations by touching the screen with their fingers or a stylus. The interactive flat panel detects the touch location and responds accordingly to achieve the touch function. Different touch sensing modules used on touch-sensitive displays will collect different raw touch signals, resulting in different converted touch signals. In interactive flat panels, to allow quick access to frequently used functions without exiting the current page, sidebars are usually placed on both sides of the display screen. These sidebars display multiple function controls, such as a camera control to activate the camera or a writing control to quickly start writing.

[0055] For infrared displays, the touch sensing module is an infrared touch frame. The tactile signals it collects can include signals indicating infrared light obstruction. The converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position. For capacitive displays, the touch sensing module is a capacitive touchpad. The tactile signals it collects can include the current flowing through the electrodes of the touchscreen. The converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position. For resistive displays, the touch sensing module is a resistive touchpad. The tactile signals it collects can include the voltage at the touch position. The converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position. For electromagnetic displays, the touch sensing module is an electromagnetic plate. The tactile signals it collects can include the change in magnetic flux and the frequency of the received electromagnetic signal. The converted touch signals can include a position touch signal corresponding to the change in magnetic flux and a pressure-sensitive signal corresponding to the frequency. The position touch signal can include the X and Y coordinates of the touch position; the pressure-sensitive signal can include a pressure value. For pressure-sensitive displays, the touch sensing module is a pressure sensor. The touch signals it collects can include pressure signals, and the converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position.

[0056] The interactive flat panel described in this application embodiment may also be referred to as an all-in-one machine, smart flat panel, conference flat panel, conference large board, complete machine, etc. It should be understood that electronic devices used in multi-person communication scenarios to support information, presentation, recording and sharing functions in the process of multi-person communication, and with a matching screen sharing device for sending screen sharing data, are all within the scope of interactive flat panel.

[0057] Figure 1 is a flowchart of a cross-device control method provided in an embodiment of this application. As shown in Figure 1, the cross-device control method includes, but is not limited to, steps S110-S160. Steps S110-S160, and the supplementary description centered on steps S110-S150, are a comparative description of the independent data processing steps performed by each of the screen-sharing source device 31, the screen-sharing device 21, and the screen-sharing display device 11 during the cross-device control process of the screen-sharing display device 11 over the screen-sharing source device 31, as well as the data processing steps performed in cooperation between the two. Additionally, Figure 3 can be referenced simultaneously to understand the data processing and transmission process of the screen-sharing source device 31, the screen-sharing device 21, and the screen-sharing display device 11 when implementing the embodiment of this application.

[0058] Step S110: When the screen sharing device is in the screen sharing connection state, it receives the first peripheral operation data sent by the screen sharing display device through the wireless network. The first peripheral operation data is generated by the peripheral device of the screen sharing display device.

[0059] In a screen sharing scenario, as shown in Figure 2, it generally includes a screen sharing source device 31, a screen sharing device 21, and a screen sharing display device 11. When the screen sharing function is needed to support multi-person communication, the user first inserts the screen sharing device 21 into the screen sharing source device 31 through a wired interface (such as a USB Type-A interface and a USB Type-C interface), completing the physical connection between the screen sharing device 21 and the screen sharing source device 31 and establishing a data transmission channel. The screen sharing device 21 also establishes a wireless network connection with the screen sharing display device 11 based on the wireless hotspot provided by the screen sharing display device 11, using the network connection information obtained by pairing with the screen sharing display device 11 beforehand. This establishes a screen sharing connection from the screen sharing source device 31 through the screen sharing device 21 to the screen sharing display device 11. After the screen sharing device 21 is connected to the screen sharing source device 31 via a wired interface, it can directly trigger the screen sharing source device 31 to start its own installed screen sharing auxiliary application according to the device type of the screen sharing device 21, or load the screen sharing auxiliary application to the screen sharing source device 31 from the storage path specified in the screen sharing device 21.

[0060] In the screen sharing connection state, as shown in Figure 3, the most basic data transmission requirement is that the screen sharing source device 31 generates screen sharing data based on its own display screen and played audio, first sending it to the screen sharing device 21 via a wired interface, and then the screen sharing device 21 sends the screen sharing data to the screen sharing display device 11 for output. It should be understood that, without the user terminating or pausing the screen sharing, the screen sharing data is generated and continuously sent in real time, unaffected by the operation of peripheral devices on the screen sharing display device 11.

[0061] During screen sharing, the user may operate the peripheral devices of the screen sharing display device 11, and various operation signals will be detected accordingly to generate operation data. For example, the screen sharing display device 11 may be equipped with peripheral devices such as a touch module, mouse, and keyboard, each of which may generate corresponding operation data. For example, the touch module may generate operation data corresponding to the touch position and touch duration, while the mouse and keyboard may generate operation data corresponding to the triggering of specific keys. The screen sharing display device 11 itself can also respond to operation data. However, in the screen sharing connection state, if the currently displayed screen is from the screen sharing source device 31, it means that the operation needs to be responded to by the screen sharing source device 31 that actually generated the screen, and the operation data will be sent to the screen sharing device 21 via the wireless network. To distinguish the processing results of the same data at different stages in subsequent processing, this operation data is defined here as the first peripheral operation data.

[0062] Step S120: The screen transmitter sends the first peripheral operation data to the screen transmission source device through the wired interface.

[0063] As shown in Figure 3, the screen sharing device 21 does not process the operation data of the first peripheral device, but directly sends it to the screen sharing source device 31 through the wired interface used to establish the screen sharing connection.

[0064] Step S130: The screen sharing auxiliary application of the screen sharing source device converts the first peripheral operation data into the second peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device, and sends it to the screen sharing device.

[0065] Depending on the peripheral device, the peripheral operation data also varies. Some peripheral operation data can be directly responded to in all electronic devices. For example, the peripheral operation data generated by mouse and keyboard key signals, and the key signals of the left mouse button and the Del key on the keyboard, can be directly responded to in different electronic devices according to system definitions. Some peripheral operation data cannot be directly responded to in all electronic devices. For example, the peripheral operation data generated by the touch module actually records the touch position. Different electronic devices have different touch screen sizes, and the same position may point to different targets when displaying the same image. To ensure that the peripheral operation data generated by the touch module of the screen-sharing display device can be correctly responded to in screen-sharing source devices of different sizes, it is necessary to convert the first peripheral operation data generated by the touch module into second peripheral operation data according to the display parameters of the screen-sharing source device and the screen-sharing display device. Of course, considering that the peripheral operation data of various peripheral devices may have different response methods in different electronic devices, step S130 is not limited to the first peripheral operation data generated by the touch module, but all first peripheral operation data generated by various peripheral devices need to be converted. Of course, the conversion and processing of entity data in the first peripheral operation data may remain unchanged when the response method is the same.

[0066] For conversion processes related to display parameters, if the aspect ratios of the source and display devices are the same, the touch position can be directly converted proportionally. For example, if the source device is 960×540 and the display device is 3840×2160. If the aspect ratios of the source and display devices are different, the screen mirroring method (e.g., full-screen or proportional) needs to be considered. The specific processing method is still proportional conversion, but it is necessary to determine whether the black bars on both sides of the display device need to be taken into account during screen mirroring. Overall, by converting the display parameters accordingly, it can be ensured that operations detected by two devices with different display screen sizes can be accurately responded to.

[0067] In one optional implementation, when establishing a screen sharing connection, the screen sharing device obtains the hardware parameters of the display device and sends them to the source device for caching. Correspondingly, when establishing a screen sharing connection, the source device obtains the hardware parameters from the display device and caches them. During data conversion, the source device converts the first peripheral operation data according to the conversion relationship between the cached hardware parameters and the source device's hardware parameters to obtain the second peripheral operation data, which is then sent to the screen sharing device. The source device's caching of the display device's hardware parameters allows for rapid conversion based on its own and the cached hardware parameters upon receiving the first peripheral operation data.

[0068] In another optional implementation, the screen-sharing display device simultaneously carries its hardware parameters when sending the first peripheral operation data. When the screen-sharing source device performs data conversion, it converts the first peripheral operation data according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen-sharing source device, obtaining the second peripheral operation data, which is then sent to the screen-sharing device. By corresponding to the conversion of display parameters, it is possible to ensure accurate and corresponding responses to operations detected by two devices with different display screen sizes.

[0069] Corresponding to the data transmission and reception process of the screen sharing device, when the screen sharing source device is in the screen sharing connection state, it receives the first peripheral operation data from the screen sharing device through a wired interface. The first peripheral operation data is received by the screen sharing device from the screen sharing display device through a wireless network connection. The first peripheral operation data is generated by the peripheral device of the screen sharing display device. The screen sharing source device converts the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and sends it to the screen sharing device. This allows the screen sharing device to encapsulate the second peripheral operation data according to the USB HID (Human Interface Device) protocol to obtain an HID message and send it to the screen sharing source device.

[0070] Step S140: The screen transmitter receives the operation data of the second peripheral device and encapsulates the operation data of the second peripheral device based on the USB HID protocol to obtain an HID message.

[0071] After receiving the operation data from the second peripheral device, the screen sharing device encapsulates the operation data according to the USB HID protocol. In this embodiment, the first peripheral device operation data, the second peripheral device operation data, and the HID message are essentially the result of encoding the physical data, i.e., the operation data generated by the user's operation of the peripheral device of the screen sharing display device, using different data packet protocols. There are corresponding encoding and decoding processes between various conversions, which are the basic implementation of data transmission and will not be elaborated here.

[0072] Step S150: The screen sharing device sends the HID message to the screen sharing source device.

[0073] The screen sharing device establishes a wired connection with the screen sharing source device, such as a wired connection established through a USB Type-A interface or a USB Type-C interface. The HID message is sent to the screen sharing source device through the wired connection accordingly.

[0074] Step S160: The source device of the screen transmission responds according to the HID message.

[0075] For the source device in a screen-sharing application, the screen-sharing auxiliary application may simply be a plugin to assist the screen-sharing device in implementing the screen-sharing function. Its core functionality primarily involves audio and video data processing related to screen-sharing and may lack the ability to communicate with the source device's various interfaces. However, for the source device, HID packets are data that can be directly processed and allocated by the operating system. Therefore, when the screen-sharing device encapsulates the second peripheral operation data into HID packets and sends them to the remote screen-sharing device via a wired interface, the remote device's operating system can quickly recognize them as data for interactive control and then allocate and respond according to the standard processing flow. This means that implementing this solution only requires adding corresponding data sending and conversion functionality to existing screen-sharing applications. Compared to the screen-sharing device directly forwarding the first peripheral operation data to the source device for processing and response, there is no need to design the connection and sending logic between the screen-sharing auxiliary application and the system data interface of the remote screen-sharing device. Furthermore, the potentially continuous conversions are handled by the powerful data processing capabilities of the source device, effectively reducing the data processing burden on the screen-sharing device.

[0076] It should be understood that during screen sharing, the screen sharing data is generated and continuously transmitted in real time, unaffected by operations on the peripheral devices of the screen sharing display device 11. The screen sharing source device 31 responds according to the HID message, causing changes in the displayed image and / or audio output. These changes are directly reflected in the output of the screen sharing display device 11. From the user's perspective, this operation and the corresponding output change represent an operation on the content displayed on the screen sharing display device 11, with a corresponding response from the device. However, the actual underlying response logic is that the screen sharing display device 11 feeds back the peripheral operation data generated by the user's operation to the screen sharing source device 31 for response through this embodiment. The response result of the screen sharing source device 31 is presented to the screen sharing display device 11 through the basic screen sharing function. The screen sharing data at the bottom of Figure 3 does not indicate that screen sharing resumes only after the intermediate peripheral operation data processing is completed; it is only used to emphasize that the user's operation on the peripheral devices of the screen sharing display device 11 will be fed back to the screen sharing source device 31 for response, ultimately causing changes in the output content of the screen sharing display device 11.

[0077] As described above, through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring assistant application in the screen mirroring source device converts it according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain the HID message directly available to the screen mirroring source device, and sends the HID message to the screen mirroring source device for response, realizing the cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0078] FIG. 4 is a flowchart of a cross-device control method provided by an embodiment of the present application. As shown in FIG. 4, the cross-device control method includes, but is not limited to, steps S210-step S240. Steps S210-step S240, and the supplementary description centered on steps S210-step S240, are descriptions of the process of cross-device control from the perspective of the screen mirroring device. As shown in FIG. 4, the cross-device control method includes:

[0079] Step S210: In the screen mirroring connection state, receive the first peripheral operation data sent by the screen mirroring display device through a wireless network. The first peripheral operation data is generated by the peripheral device of the screen mirroring display device.

[0080] Step S220: Send the first peripheral operation data to the screen mirroring source device through a wired interface, so that the screen mirroring assistant application of the screen mirroring source device converts the first peripheral operation data according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data and send it to the screen mirroring device.

[0081] Step S230: Receive the second peripheral operation data and encapsulate the second peripheral operation data based on the USB HID protocol to obtain an HID message.

[0082] Step S240: Send the HID message to the screen mirroring source device for the screen mirroring source device to respond according to the HID message.

[0083] Overall, through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operations performed by the user on the peripheral devices supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device converts it according to the hardware parameters of the screen mirroring display device and the screen mirroring source device, obtains the second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain the HID message directly available to the screen mirroring source device, and sends the HID message to the screen mirroring source device for response, realizing cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0084] Based on the above embodiments, the cross-device control method further includes:

[0085] When establishing the screen mirroring connection, obtain the hardware parameters of the screen mirroring display device and send them to the screen mirroring source device for caching. The caching of the hardware parameters of the screen mirroring display device by the screen mirroring source device can perform conversion quickly according to its own and the cached hardware parameters when receiving the first peripheral operation data.

[0086] Based on the above embodiments, the first peripheral operation data synchronously carries the hardware parameters of the screen mirroring display device. When the screen mirroring source device obtains the first peripheral operation data, it obtains the hardware parameters of the screen mirroring display device and quickly performs conversion in combination with its own hardware parameters.

[0087] Based on the above embodiments, the hardware parameters include display parameters. Through the conversion corresponding to the display parameters, it can ensure accurate corresponding responses to the operations detected between two devices with different display screen sizes.

[0088] Based on the above embodiments, the peripheral devices include one or more of a touch control module, a mouse, and a keyboard.

[0089] Based on the above embodiments, the wired interface is a USB Type-A interface or a USB Type-C interface.

[0090] The specific execution process and the corresponding beneficial effects in the embodiments of the present application are described correspondingly in the embodiments centered around Figure 1, and will not be repeated here.

[0091] FIG. 5 is a flowchart of a cross-device control method provided by an embodiment of the present application. As shown in FIG. 5, the cross-device control method includes, but is not limited to, steps S310 to S330. Steps S310 to S330, as well as the supplementary descriptions centered around steps S310 to S330, describe the process of cross-device control from the perspective of the screen-casting source device. As shown in FIG. 5, the cross-device control method includes:

[0092] Step S310: In the screen-casting connection state, receive first peripheral device operation data from the screen-casting device through a wired interface. The first peripheral device operation data is received by the screen-casting device from the screen-casting display device through a wireless network connection, and the first peripheral device operation data is generated by a peripheral device of the screen-casting display device.

[0093] Step S320: Convert the first peripheral device operation data according to the hardware parameters of the screen-casting display device and the screen-casting source device, obtain second peripheral device operation data, and send it to the screen-casting device, so that the screen-casting device encapsulates the second peripheral device operation data based on the USB HID protocol to obtain a HID message and send it to the screen-casting source device.

[0094] Step S330: Respond according to the HID message.

[0095] Overall, through the existing data transmission channel in the screen-casting connection state, the first peripheral device operation data generated by the interaction operation of the user on the peripheral device supporting the screen-casting display device is sent to the screen-casting device. The screen-casting device forwards the first peripheral device operation data to the screen-casting source device. The screen-casting auxiliary application in the screen-casting source device converts the first peripheral device operation data according to the hardware parameters of the screen-casting display device and the screen-casting source device, obtains second peripheral device operation data adapted to its own hardware state, and then sends it to the screen-casting device. The screen-casting device encapsulates the second peripheral device operation data according to the USB HID protocol to obtain a HID message directly available to the screen-casting source device, and sends the HID message to the screen-casting source device for response, realizing cross-device control of the screen-casting source device by the screen-casting display device when explaining with the screen-casting display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0096] Based on the above embodiment, the cross-device control method further includes:

[0097] When establishing a screen-casting connection, obtain the hardware parameters obtained by the screen-casting device from the screen-casting display device and cache them;

[0098] Correspondingly, converting the first peripheral device operation data according to the hardware parameters of the screen-casting display device and the screen-casting source device, obtaining second peripheral device operation data, and sending it to the screen-casting device includes:

[0099] The first peripheral operation data is converted according to the conversion relationship between the cached hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0100] The source device of the screen sharing device caches the hardware parameters of the screen sharing display device, and can quickly convert them according to its own and cached hardware parameters when it receives the operation data of the first peripheral device.

[0101] Based on the above embodiments, the first peripheral operation data synchronously carries the hardware parameters of the screen transmission display device;

[0102] Accordingly, the first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device, including:

[0103] The first peripheral operation data is converted according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0104] By converting the display parameters accordingly, it is possible to ensure that operations detected by two devices with different display screen sizes can be accurately responded to.

[0105] The specific execution process and corresponding beneficial effects in the embodiments of this application are described in the embodiments centered on Figure 1, and will not be repeated here.

[0106] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The specific product form of this electronic device can be an interactive flat panel or a screen sharing device. As shown in Figure 6, the electronic device includes a processor 310 and a memory 320. The interactive flat panel and conference control device mentioned above are both types of electronic devices in terms of hardware architecture. In a common product form of electronic devices, it may also include an input device 330, an output device 340, and a communication device 350. The number of processors 310 in the electronic device can be one or more. Figure 6 shows one processor 310 as an example. The processor 310, memory 320, input device 330, output device 340, and communication device 350 in the electronic device can be connected through a bus or other means. Figure 6 shows a connection via a bus as an example.

[0107] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the cross-device control method in the embodiments of this application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby realizing the aforementioned cross-device control method.

[0108] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] Input device 330 can be used to receive network configuration information. Output device 340 may include a display device such as a screen.

[0110] The aforementioned electronic equipment can be used to execute any cross-device control method, possessing corresponding functions and beneficial effects.

[0111] This application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform relevant operations in the cross-device control method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0112] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0113] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0114] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A cross-device control method, wherein, For a screen sharing device, the cross-device control method includes: In the screen sharing connection state, the system receives first peripheral operation data sent by the screen sharing display device via a wireless network. The first peripheral operation data is generated by the peripheral device of the screen sharing display device. The first peripheral operation data is sent to the screen transmission source device through a wired interface, so that the screen transmission auxiliary application of the screen transmission source device can convert the first peripheral operation data into second peripheral operation data according to the hardware parameters of the screen transmission display device and the screen transmission source device and send it to the screen transmission device. Receive the operation data of the second peripheral device, and encapsulate the operation data of the second peripheral device based on the USB HID protocol to obtain an HID message; The HID message is sent to the screen transmission source device so that the screen transmission source device can respond according to the HID message.

2. The cross-device control method according to claim 1, wherein, Also includes: When establishing a screen sharing connection, the hardware parameters of the screen sharing display device are obtained and sent to the screen sharing source device for caching.

3. The cross-device control method according to claim 1, wherein, The first peripheral operation data synchronously carries the hardware parameters of the screen display device.

4. The cross-device control method according to any one of claims 1-3, wherein, The hardware parameters include display parameters.

5. The cross-device control method according to any one of claims 1-3, wherein, The peripheral devices include one or more of a touch module, a mouse, and a keyboard.

6. The cross-device control method according to any one of claims 1-3, wherein, The wired interface is either a USB Type-A interface or a USB Type-C interface.

7. Cross-device control method, wherein, The cross-device control method for screen sharing source devices includes: In the screen sharing connection state, the first peripheral operation data is received from the screen sharing device through the wired interface. The first peripheral operation data is received by the screen sharing device from the screen sharing display device through the wireless network connection. The first peripheral operation data is generated by the peripheral device of the screen sharing display device. The first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and send it to the screen sharing device, so that the screen sharing device encapsulates the second peripheral operation data based on the USB HID protocol to obtain HID message and sends it to the screen sharing source device. Respond according to the HID message.

8. The cross-device control method according to claim 7, wherein, Also includes: When establishing a screen sharing connection, the hardware parameters obtained by the screen sharing device from the screen sharing display device are acquired and cached. Accordingly, the step of converting the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and sending it to the screen sharing device includes: The first peripheral operation data is converted according to the conversion relationship between the cached hardware parameters and the hardware parameters of the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device.

9. The cross-device control method according to claim 7, wherein, The first peripheral operation data synchronously carries the hardware parameters of the screen display device. Accordingly, the step of converting the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and sending it to the screen sharing device includes: The first peripheral operation data is converted according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

10. Electronic equipment, of which, include: One or more processors; Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the cross-device control method as described in any one of claims 1-9.

11. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the cross-device control method as described in any one of claims 1-9.

Images