Cross-device control method, electronic device and storage medium

Abstract

The embodiment of the invention discloses a cross-device control method, electronic equipment and a storage medium. Through an existing data transmission channel in a screen transmission connection state, first peripheral operation data generated by interaction operation performed by a user on peripheral equipment matched with the screen transmission display equipment is sent to a screen transmission device, and the screen transmission device forwards the first peripheral operation data to screen transmission source end equipment. The screen transmission auxiliary application in the screen transmission source end equipment performs conversion according to the hardware parameters of the screen transmission display equipment and the screen transmission source end equipment to obtain second peripheral operation data adaptive to the hardware state of the screen transmission auxiliary application, and then sends the second peripheral operation data to the screen transmission device; and the screen transmission device carries out packaging according to the USB HID protocol according to the second peripheral operation data to obtain an HID message directly available for the screen transmission source end equipment, and sends the HID message to the screen transmission source end equipment for response, so that cross-equipment control on the screen transmission source end equipment when explanation is carried out by taking the screen transmission display equipment as the center is realized, continuity of multi-person communication is ensured, and the user experience is improved. And the information transmission effect is improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and more particularly to cross-device control methods, electronic devices, and storage media. Background Technology

[0002] With the continuous development and popularization of information technology, various types of electronic devices are constantly emerging, and the ways in which these devices connect have also become more diverse. For example, in conference rooms built based on information technology, interactive whiteboards, microphones, speakers, mice, keyboards, and other electronic devices are usually configured. These devices are interconnected and work together as peripherals, with the interactive whiteboard at the core, to enable information collection, transmission, and presentation during multi-person communication.

[0003] When using interactive whiteboards to present information, the information source may not be the interactive whiteboard itself, but rather other terminal devices in the meeting room. Screen sharing technology is used to replicate the information from the source device on the display device. In this application scenario, the interactive whiteboard acts as the display device, and the other terminal devices are the source devices. To achieve information transfer between the source and display devices, a dedicated intermediate device is typically needed to establish a data channel for data transmission. This intermediate device, specifically designed to establish data channels between different devices for information replication, can be named a screen projector, screen sharer, wireless screen projector, or wireless screen sharer, etc. The term "screen sharer" will be used consistently from now on.

[0004] The inventors analyzed the working process of existing screen sharing devices and found that, during information presentation, the focal device is the screen sharing display device, but the source of the information content is the screen sharing source device. In multi-person communication scenarios, when presenting information based on pre-prepared presentation slides, the speaker can only maintain the screen sharing source device as the center, or move between the screen sharing source device and the screen sharing display device, in order to operate and switch the information presentation content on the screen sharing source device. They cannot maintain the screen sharing display device as the focal point of information presentation for the presentation, which affects the continuity of multi-person communication and reduces the effectiveness of information transmission. Summary of the Invention

[0005] This invention provides a cross-device control method, electronic device, and storage medium to solve the technical problem that existing screen sharing processes cannot maintain the focus of the information presentation on the screen display device, which affects the continuity of multi-person communication and reduces the effectiveness of information transmission.

[0006] In a first aspect, embodiments of this application provide a cross-device control method for a screen sharing device, the cross-device control method comprising:

[0007] In the screen mirroring connection state, receive the first peripheral operation data sent by the screen mirroring display device through a wireless network. The first peripheral operation data is generated by the peripheral device of the screen mirroring display device;

[0008] Send the first peripheral operation data to the screen mirroring source device through a wired interface, so that the screen mirroring auxiliary application of the screen mirroring source device can convert the first peripheral operation data according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data and send it to the screen mirror;

[0009] Receive the second peripheral operation data, and encapsulate the second peripheral operation data based on the USB HID protocol to obtain a HID message;

[0010] Send the HID message to the screen mirroring source device for the screen mirroring source device to respond according to the HID message.

[0011] As described above, through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen mirroring display device is sent to the screen mirror. The screen mirror forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device converts it according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data adapted to its own hardware state and then sends it to the screen mirror. The screen mirror encapsulates the second peripheral operation data according to the USB HID protocol to obtain a HID message directly available to the screen mirroring source device and sends the HID message to the screen mirroring source device for response, realizing cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the center of information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0012] Among them, the cross-device control method further includes:

[0013] When establishing a screen mirroring connection, obtain the hardware parameters of the screen mirroring display device and send them to the screen mirroring source device for caching.

[0014] As described above, the screen mirroring source device can quickly convert according to its own and cached hardware parameters when receiving the first peripheral operation data.

[0015] Among them, the first peripheral operation data carries the hardware parameters of the screen mirroring display device synchronously.

[0016] As described above, the screen mirroring source device obtains the hardware parameters of the screen mirroring display device when obtaining the first peripheral operation data, and quickly converts in combination with its own hardware parameters accordingly.

[0017] Among them, the hardware parameters include display parameters.

[0018] As described above, through the conversion corresponding to the display parameters, it is possible to ensure an accurate corresponding response to the operations detected between two devices with different display screen sizes.

[0019] In a second aspect, an embodiment of the present application provides a cross-device control method for a screen mirroring source device. The cross-device control method includes:

[0020] In the screen mirroring connection state, receive first peripheral operation data from a screen mirroring device through a wired interface. The first peripheral operation data is received by the screen mirroring device from a screen mirroring display device through a wireless network connection, and the first peripheral operation data is generated by a peripheral device of the screen mirroring display device;

[0021] Convert the first peripheral operation data according to the hardware parameters of the screen mirroring display device and the screen mirroring source device, obtain second peripheral operation data and send it to the screen mirroring device, so that the screen mirroring device encapsulates the second peripheral operation data based on the USB HID protocol to obtain a HID message and send it to the screen mirroring source device;

[0022] Respond according to the HID message.

[0023] As described above, through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device converts the first peripheral operation data according to the hardware parameters of the screen mirroring display device and the screen mirroring source device, obtains second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain a HID message directly available to the screen mirroring source device, and sends the HID message to the screen mirroring source device for response, realizing cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0024] Among them, the cross-device control method further includes:

[0025] When establishing a screen mirroring connection, obtain the hardware parameters obtained by the screen mirroring device from the screen mirroring display device and cache them;

[0026] Correspondingly, converting the first peripheral operation data according to the hardware parameters of the screen mirroring display device and the screen mirroring source device, obtaining second peripheral operation data and sending it to the screen mirroring device includes:

[0027] The first peripheral operation data is converted according to the conversion relationship between the cached hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0028] The aforementioned caching of hardware parameters of the screen-sharing display device by the source device can be quickly converted based on its own and the cached hardware parameters when receiving operation data from the first peripheral device.

[0029] Among them, the first peripheral operation data synchronously carries the hardware parameters of the screen display device;

[0030] Accordingly, the first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device, including:

[0031] The first peripheral operation data is converted according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0032] As described above, by converting the display parameters accordingly, it is possible to ensure that operations detected by two devices with different display screen sizes can be accurately responded to.

[0033] Thirdly, embodiments of this application provide an electronic device, which includes:

[0034] One or more processors;

[0035] Memory, used to store one or more computer programs;

[0036] When one or more computer programs are executed by one or more processors, electronic devices enable cross-device control methods as described in either the first or second aspect.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a cross-device control method as described in either the first or second aspect.

[0038] The electronic device of the third aspect and the computer-readable storage medium of the fourth aspect can be used to perform the cross-device control method provided in any of the above embodiments, and have corresponding functions and beneficial effects. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating a cross-device control method provided in this application embodiment;

[0041] Figure 2 This is a schematic diagram illustrating application scenarios for interactive flat panels and screen sharing devices.

[0042] Figure 3 To implement Figure 1 The diagram illustrates the data processing and transmission process in the cross-device control method.

[0043] Figure 4 A flowchart illustrating another cross-device control method provided in this application embodiment;

[0044] Figure 5 A flowchart illustrating another cross-device control method provided in this application embodiment;

[0045] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all of the structures.

[0047] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0048] The embodiments are described in detail below.

[0049] In the process of information presentation in the screen mirroring scenario, the focus device for information presentation is the screen mirroring display device, but the source determining the content of information presentation is the screen mirroring source device. When presenting information based on a pre-prepared presentation in a multi-person communication scenario, the presenter can only center around the screen mirroring source device or move between the screen mirroring source device and the screen mirroring display device to switch the information presentation content by operating on the screen mirroring source device. The presenter cannot center around the focus of information presentation, i.e., the screen mirroring display device, for the explanation, which affects the continuity of multi-person communication and reduces the effect of information transmission. For example, during the information presentation process, when it is necessary to use communication auxiliary functions such as the handwritten annotation of the screen mirroring display device on the screen mirroring display device, when switching the presented information, the presenter can only leave the screen mirroring display device and move to the screen mirroring source device for operation, moving back and forth between two devices at different positions, which obviously affects the continuity of multi-person communication and reduces the effect of information transmission.

[0050] To solve the above technical problems, the embodiment of the present application proposes a cross-device control method. Through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device converts it according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain the HID message directly available to the screen mirroring source device, and sends the HID message to the screen mirroring source device for response, realizing the cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the focus of information presentation, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0051] The screen mirroring display device applied in this solution is exemplarily an interactive flat panel. The interactive flat panel can be an integrated device that manipulates the content displayed on the display screen through touch technology and realizes human-computer interaction operations, and integrates one or more functions such as a projector, an electronic whiteboard, a curtain, a speaker, a television, a video conferencing terminal, and a wireless router. The interactive flat panel can be flat, curved, or a splicing of multiple planes, etc.

[0052] Typically, interactive flat panels are equipped with at least one operating system, including but not limited to Android, Linux, Windows, and Huawei HarmonyOS. This operating system controls and coordinates the interactive flat panel and peripheral devices, enabling the various independent hardware components within the interactive flat panel to work together as a stable whole. An interactive flat panel includes at least one display screen, such as a touch-sensitive display. This touch-sensitive display screen can be an infrared display, capacitive display, resistive display, electromagnetic display, or pressure-sensitive display, etc. On a touch-sensitive display screen, users can perform touch operations by touching the screen with their fingers or a stylus. The interactive flat panel detects the touch location and responds accordingly to achieve the touch function. Different touch sensing modules used on touch-sensitive displays will collect different raw touch signals, resulting in different converted touch signals. In interactive flat panels, to allow quick access to frequently used functions without exiting the current page, sidebars are usually placed on both sides of the display screen. These sidebars display multiple function controls, such as a camera control to activate the camera or a writing control to quickly start writing.

[0053] For infrared displays, the touch sensing module is an infrared touch frame. The tactile signals it collects can include signals indicating infrared light obstruction. The converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position. For capacitive displays, the touch sensing module is a capacitive touchpad. The tactile signals it collects can include the current flowing through the electrodes of the touchscreen. The converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position. For resistive displays, the touch sensing module is a resistive touchpad. The tactile signals it collects can include the voltage at the touch position. The converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position. For electromagnetic displays, the touch sensing module is an electromagnetic plate. The tactile signals it collects can include the change in magnetic flux and the frequency of the received electromagnetic signal. The converted touch signals can include a position touch signal corresponding to the change in magnetic flux and a pressure-sensitive signal corresponding to the frequency. The position touch signal can include the X and Y coordinates of the touch position; the pressure-sensitive signal can include a pressure value. For pressure-sensitive displays, the touch sensing module is a pressure sensor. The touch signals it collects can include pressure signals, and the converted touch signals can include position touch signals, which can include the X and Y coordinates of the touch position.

[0054] The interactive flat panel described in this application embodiment may also be referred to as an all-in-one machine, smart flat panel, conference flat panel, conference large board, complete machine, etc. It should be understood that electronic devices used in multi-person communication scenarios to support information, presentation, recording and sharing functions in the process of multi-person communication, and with a matching screen sharing device for sending screen sharing data, are all within the scope of interactive flat panel.

[0055] Figure 1 This is a flowchart of a cross-device control method provided in an embodiment of this application, such as... Figure 1 As shown, this cross-device control method includes, but is not limited to, steps S110-S160. Steps S110-S160, and the supplementary description centered on steps S110-S150, are a comparative description, from the comprehensive perspective of the screen-sharing source device 31, the screen-sharing device 21, and the screen-sharing display device 11, of the independent data processing stages performed by each device, as well as the data processing stages performed in cooperation between the two during the cross-device control process of the screen-sharing display device 11 over the screen-sharing source device 31. Additionally, please refer to... Figure 3 When implementing the embodiments of this application, the data processing and transmission process of the screen transmission source device 31, screen transmission device 21 and screen transmission display device 11 are understood.

[0056] Step S110: When the screen sharing device is in the screen sharing connection state, it receives the first peripheral operation data sent by the screen sharing display device through the wireless network. The first peripheral operation data is generated by the peripheral device of the screen sharing display device.

[0057] In screen sharing scenarios, such as Figure 2 As shown, a typical setup includes a screen sharing source device 31, a screen sharing adapter 21, and a screen sharing display device 11. When screen sharing is needed to support multi-user communication, the user first inserts the screen sharing adapter 21 into the screen sharing source device 31 via a wired interface (e.g., USB Type-A or USB Type-C), completing the physical connection between the screen sharing adapter 21 and the screen sharing source device 31 and establishing a data transmission channel. The screen sharing adapter 21 also establishes a wireless network connection with the screen sharing display device 11 based on the wireless hotspot provided by the screen sharing display device 11, using the network connection information obtained through pre-pairing with the screen sharing display device 11. This establishes a screen sharing connection from the screen sharing source device 31 through the screen sharing adapter 21 to the screen sharing display device 11. After the screen sharing adapter 21 is connected to the screen sharing source device 31 via the wired interface, it can directly trigger the screen sharing source device 31 to launch its own installed screen sharing auxiliary application, or load the screen sharing auxiliary application from the storage path specified in the screen sharing adapter 21 onto the screen sharing source device 31.

[0058] In screen sharing connection state, such as Figure 3As shown, the most basic data transmission requirement is that the source device 31 generates screen sharing data based on its own display screen and played audio, first sending it to the screen sharing receiver 21 via a wired interface, and then the screen sharing receiver 21 sends the screen sharing data to the screen sharing display device 11 for output. It should be understood that, without the user terminating or pausing the screen sharing, the screen sharing data is generated and continuously sent in real time, unaffected by the operation of peripheral devices on the screen sharing display device 11.

[0059] During screen sharing, the user may operate the peripheral devices of the screen sharing display device 11, and various operation signals will be detected accordingly to generate operation data. For example, the screen sharing display device 11 may be equipped with peripheral devices such as a touch module, mouse, and keyboard, each of which may generate corresponding operation data. For example, the touch module may generate operation data corresponding to the touch position and touch duration, while the mouse and keyboard may generate operation data corresponding to the triggering of specific keys. The screen sharing display device 11 itself can also respond to operation data. However, in the screen sharing connection state, if the currently displayed screen is from the screen sharing source device 31, it means that the operation needs to be responded to by the screen sharing source device 31 that actually generated the screen, and the operation data will be sent to the screen sharing device 21 via the wireless network. To distinguish the processing results of the same data at different stages in subsequent processing, this operation data is defined here as the first peripheral operation data.

[0060] Step S120: The screen transmitter sends the first peripheral operation data to the screen transmission source device through the wired interface.

[0061] like Figure 3 As shown, the screen sharing device 21 does not process the operation data of the first peripheral device, but directly sends it to the screen sharing source device 31 through the wired interface used to establish the screen sharing connection.

[0062] Step S130: The screen sharing auxiliary application of the screen sharing source device converts the first peripheral operation data into the second peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device, and sends it to the screen sharing device.

[0063] Depending on the peripheral device, the peripheral operation data also varies. Some peripheral operation data can be directly responded to in all electronic devices. For example, the peripheral operation data generated by mouse and keyboard key signals, and the key signals of the left mouse button and the Del key on the keyboard, can be directly responded to in different electronic devices according to system definitions. Some peripheral operation data cannot be directly responded to in all electronic devices. For example, the peripheral operation data generated by the touch module actually records the touch position. Different electronic devices have different touch screen sizes, and the same position may point to different targets when displaying the same image. To ensure that the peripheral operation data generated by the touch module of the screen-sharing display device can be correctly responded to in screen-sharing source devices of different sizes, it is necessary to convert the first peripheral operation data generated by the touch module into second peripheral operation data according to the display parameters of the screen-sharing source device and the screen-sharing display device. Of course, considering that the peripheral operation data of various peripheral devices may have different response methods in different electronic devices, step S130 is not limited to the first peripheral operation data generated by the touch module, but all first peripheral operation data generated by various peripheral devices need to be converted. Of course, the conversion and processing of entity data in the first peripheral operation data may remain unchanged when the response method is the same.

[0064] For conversion processes related to display parameters, if the aspect ratios of the source and display devices are the same, the touch position can be directly converted proportionally. For example, if the source device is 960×540 and the display device is 3840×2160. If the aspect ratios of the source and display devices are different, the screen mirroring method (e.g., full-screen or proportional) needs to be considered. The specific processing method is still proportional conversion, but it is necessary to determine whether the black bars on both sides of the display device need to be taken into account during screen mirroring. Overall, by converting the display parameters accordingly, it can be ensured that operations detected by two devices with different display screen sizes can be accurately responded to.

[0065] In one optional implementation, when establishing a screen sharing connection, the screen sharing device obtains the hardware parameters of the display device and sends them to the source device for caching. Correspondingly, when establishing a screen sharing connection, the source device obtains the hardware parameters from the display device and caches them. During data conversion, the source device converts the first peripheral operation data according to the conversion relationship between the cached hardware parameters and the source device's hardware parameters to obtain the second peripheral operation data, which is then sent to the screen sharing device. The source device's caching of the display device's hardware parameters allows for rapid conversion based on its own and the cached hardware parameters upon receiving the first peripheral operation data.

[0066] In another optional implementation, the screen-sharing display device simultaneously carries its hardware parameters when sending the first peripheral operation data. When the screen-sharing source device performs data conversion, it converts the first peripheral operation data according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen-sharing source device, obtaining the second peripheral operation data, which is then sent to the screen-sharing device. By corresponding to the conversion of display parameters, it is possible to ensure accurate and corresponding responses to operations detected by two devices with different display screen sizes.

[0067] Corresponding to the data transmission and reception process of the screen sharing device, when the screen sharing source device is in the screen sharing connection state, it receives the first peripheral operation data from the screen sharing device through a wired interface. The first peripheral operation data is received by the screen sharing device from the screen sharing display device through a wireless network connection. The first peripheral operation data is generated by the peripheral device of the screen sharing display device. The screen sharing source device converts the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and sends it to the screen sharing device. This allows the screen sharing device to encapsulate the second peripheral operation data according to the USB HID (Human Interface Device) protocol to obtain an HID message and send it to the screen sharing source device.

[0068] Step S140: The screen transmitter receives the operation data of the second peripheral device and encapsulates the operation data of the second peripheral device based on the USB HID protocol to obtain an HID message.

[0069] After receiving the operation data from the second peripheral device, the screen sharing device encapsulates the operation data according to the USB HID protocol. In this embodiment, the first peripheral device operation data, the second peripheral device operation data, and the HID message are essentially the result of encoding the physical data, i.e., the operation data generated by the user's operation of the peripheral device of the screen sharing display device, using different data packet protocols. There are corresponding encoding and decoding processes between various conversions, which are the basic implementation of data transmission and will not be elaborated here.

[0070] Step S150: The screen sharing device sends the HID message to the screen sharing source device.

[0071] The screen sharing device establishes a wired connection with the screen sharing source device, such as a wired connection established through a USB Type-A interface or a USB Type-C interface. The HID message is sent to the screen sharing source device through the wired connection accordingly.

[0072] Step S160: The source device of the screen transmission responds according to the HID message.

[0073] For the source device in a screen-sharing application, the screen-sharing auxiliary application may simply be a plugin to assist the screen-sharing device in implementing the screen-sharing function. Its core functionality primarily involves audio and video data processing related to screen-sharing and may lack the ability to communicate with the source device's various interfaces. However, for the source device, HID packets are data that can be directly processed and allocated by the operating system. Therefore, when the screen-sharing device encapsulates the second peripheral operation data into HID packets and sends them to the remote screen-sharing device via a wired interface, the remote device's operating system can quickly recognize them as data for interactive control and then allocate and respond according to the standard processing flow. This means that implementing this solution only requires adding corresponding data sending and conversion functionality to existing screen-sharing applications. Compared to the screen-sharing device directly forwarding the first peripheral operation data to the source device for processing and response, there is no need to design the connection and sending logic between the screen-sharing auxiliary application and the system data interface of the remote screen-sharing device. Furthermore, the potentially continuous conversions are handled by the powerful data processing capabilities of the source device, effectively reducing the data processing burden on the screen-sharing device.

[0074] It should be understood that in the screen sharing connection state, the screen sharing data is a real-time generation and continuous transmission process, unaffected by the operation of peripheral devices on the screen sharing display device 11. The screen sharing source device 31 responds according to the HID message, which causes changes in the displayed screen and / or audio output. These changes are directly reflected in the output of the screen sharing display device 11 based on the screen sharing. This operation and the corresponding output change appear to the user as an operation on the content displayed on the screen sharing display device 11, with the screen sharing display device 11 responding accordingly. However, the actual underlying response logic is that the screen sharing display device 11 feeds back the peripheral operation data generated by the user operation to the screen sharing source device 31 for response through this application embodiment. The response result of the screen sharing source device 31 is presented to the screen sharing display device 11 through the basic screen sharing function. Figure 3 The screen sharing data at the bottom does not indicate that the screen sharing will resume only after the processing of peripheral operation data in the middle is completed. It is only used to emphasize that the operation performed by the user on the peripheral device of the screen sharing display device 11 will be fed back to the screen sharing source device 31 for response, and will ultimately cause the output content of the screen sharing display device 11 to change.

[0075] As described above, through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operations performed by the user on the peripheral devices supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device performs conversion based on the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain the HID message directly available to the screen mirroring source device, and sends the HID message to the screen mirroring source device for response, realizing cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0076] Figure 4 It is a flowchart of a cross-device control method provided by an embodiment of the present application. As Figure 4 shown, the cross-device control method includes, but is not limited to, steps S210 - step S240. Steps S210 - step S240, and the supplementary description centered on steps S210 - step S240, are descriptions of the process of cross-device control from the perspective of the screen mirroring device. As Figure 4 shown, the cross-device control method includes:

[0077] Step S210: In the screen mirroring connection state, receive the first peripheral operation data sent by the screen mirroring display device through a wireless network. The first peripheral operation data is generated by the peripheral devices of the screen mirroring display device.

[0078] Step S220: Send the first peripheral operation data to the screen mirroring source device through a wired interface, so that the screen mirroring auxiliary application in the screen mirroring source device converts the first peripheral operation data according to the hardware parameters of the screen mirroring display device and the screen mirroring source device to obtain the second peripheral operation data and sends it to the screen mirroring device.

[0079] Step S230: Receive the second peripheral operation data, and encapsulate the second peripheral operation data based on the USB HID protocol to obtain the HID message.

[0080] Step S240: Send the HID message to the screen mirroring source device for the screen mirroring source device to respond according to the HID message.

[0081] Overall, through the existing data transmission channel in the screen mirroring connection state, the first peripheral operation data generated by the interaction operations performed by the user on the peripheral devices supporting the screen mirroring display device is sent to the screen mirroring device. The screen mirroring device forwards the first peripheral operation data to the screen mirroring source device. The screen mirroring auxiliary application in the screen mirroring source device converts it according to the hardware parameters of the screen mirroring display device and the screen mirroring source device, obtains the second peripheral operation data adapted to its own hardware state, and then sends it to the screen mirroring device. The screen mirroring device encapsulates the second peripheral operation data according to the USB HID protocol to obtain the HID packet directly available to the screen mirroring source device, and sends the HID packet to the screen mirroring source device for response, achieving cross-device control of the screen mirroring source device by the screen mirroring display device when explaining with the screen mirroring display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0082] Based on the above embodiments, the cross-device control method further includes:

[0083] When establishing the screen mirroring connection, obtain the hardware parameters of the screen mirroring display device and send them to the screen mirroring source device for caching. The caching of the hardware parameters of the screen mirroring display device by the screen mirroring source device can be quickly converted according to its own and the cached hardware parameters when receiving the first peripheral operation data.

[0084] Based on the above embodiments, the first peripheral operation data is synchronously carried with the hardware parameters of the screen mirroring display device. When the screen mirroring source device obtains the first peripheral operation data, it obtains the hardware parameters of the screen mirroring display device, and accordingly quickly performs conversion in combination with its own hardware parameters.

[0085] Based on the above embodiments, the hardware parameters include display parameters. Through the conversion corresponding to the display parameters, it can ensure that the operations detected between two devices with different display screen sizes are accurately and correspondingly responded to.

[0086] Based on the above embodiments, the peripheral devices include one or more of a touch control module, a mouse, and a keyboard.

[0087] Based on the above embodiments, the wired interface is a USB Type-A interface or a USB Type-C interface.

[0088] In the embodiments of the present application, the specific execution process and the corresponding beneficial effects are described in the embodiments centered on Figure 1 and will not be repeated here.

[0089] Figure 5 is the method flowchart of a cross-device control method provided by the embodiments of the present application. As shown in Figure 5As shown, the cross-device control method includes, but is not limited to, steps S310 to S330. Steps S310 to S330, and the supplementary description centered around steps S310 to S330, describe the process of cross-device control from the perspective of the screen-casting source device. As Figure 5 As shown, the cross-device control method includes:

[0090] Step S310: In the screen-casting connection state, receive first peripheral operation data from the screen-caster through a wired interface. The first peripheral operation data is received by the screen-caster from the screen-casting display device through a wireless network connection, and the first peripheral operation data is generated by a peripheral device of the screen-casting display device.

[0091] Step S320: Convert the first peripheral operation data according to the hardware parameters of the screen-casting display device and the screen-casting source device, obtain second peripheral operation data, and send it to the screen-caster, so that the screen-caster encapsulates the second peripheral operation data based on the USB HID protocol to obtain a HID message and send it to the screen-casting source device.

[0092] Step S330: Respond according to the HID message.

[0093] Overall, through the existing data transmission channel in the screen-casting connection state, the first peripheral operation data generated by the interaction operation of the user on the peripheral device supporting the screen-casting display device is sent to the screen-caster. The screen-caster forwards the first peripheral operation data to the screen-casting source device. The screen-casting auxiliary application in the screen-casting source device converts the first peripheral operation data according to the hardware parameters of the screen-casting display device and the screen-casting source device, obtains second peripheral operation data adapted to its own hardware state, and then sends it to the screen-caster. The screen-caster encapsulates the second peripheral operation data according to the USB HID protocol to obtain a HID message directly available to the screen-casting source device, and sends the HID message to the screen-casting source device for response, realizing cross-device control of the screen-casting source device by the screen-casting display device when explaining with the screen-casting display device as the center of the information presentation focus, ensuring the continuity of multi-person communication and improving the effect of information transmission.

[0094] Based on the above embodiments, the cross-device control method further includes:

[0095] When establishing a screen-casting connection, obtain the hardware parameters obtained by the screen-caster from the screen-casting display device and cache them;

[0096] Correspondingly, converting the first peripheral operation data according to the hardware parameters of the screen-casting display device and the screen-casting source device, obtaining second peripheral operation data, and sending it to the screen-caster includes:

[0097] The first peripheral operation data is converted according to the conversion relationship between the cached hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0098] The source device of the screen sharing device caches the hardware parameters of the screen sharing display device, and can quickly convert them according to its own and cached hardware parameters when it receives the operation data of the first peripheral device.

[0099] Based on the above embodiments, the first peripheral operation data synchronously carries the hardware parameters of the screen display device;

[0100] Accordingly, the first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device, including:

[0101] The first peripheral operation data is converted according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

[0102] By converting the display parameters accordingly, it is possible to ensure that operations detected by two devices with different display screen sizes can be accurately responded to.

[0103] The specific execution process and corresponding beneficial effects in the embodiments of this application are described below. Figure 1 The corresponding embodiments centered on this topic are described in detail here, and will not be repeated here.

[0104] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The specific product form of this electronic device can be an interactive flat panel or a screen sharing device. Figure 6 As shown, the electronic device includes a processor 310 and a memory 320. The interactive flat panel and conference control device mentioned earlier are both types of electronic devices in terms of hardware architecture. In a common form of electronic device, it may also include an input device 330, an output device 340, and a communication device 350. The number of processors 310 in the electronic device can be one or more. Figure 6 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, output device 340, and communication device 350 in the electronic device can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0105] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the cross-device control method in the embodiments of this application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby realizing the aforementioned cross-device control method.

[0106] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0107] Input device 330 can be used to receive network configuration information. Output device 340 may include a display device such as a screen.

[0108] The aforementioned electronic equipment can be used to execute any cross-device control method, possessing corresponding functions and beneficial effects.

[0109] This application also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform relevant operations in the cross-device control method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0110] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0111] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0113] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0115] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A cross-device control method, characterized in that, For a screen sharing device, the cross-device control method includes: In the screen sharing connection state, the system receives first peripheral operation data sent by the screen sharing display device via a wireless network. The first peripheral operation data is generated by the peripheral device of the screen sharing display device. The first peripheral operation data is sent to the screen transmission source device through a wired interface, so that the screen transmission auxiliary application of the screen transmission source device can convert the first peripheral operation data into second peripheral operation data according to the hardware parameters of the screen transmission display device and the screen transmission source device and send it to the screen transmission device. Receive the operation data of the second peripheral device, and encapsulate the operation data of the second peripheral device based on the USB HID protocol to obtain an HID message; The HID message is sent to the screen transmission source device so that the screen transmission source device can respond according to the HID message.

2. The cross-device control method according to claim 1, characterized in that, Also includes: When establishing a screen sharing connection, the hardware parameters of the screen sharing display device are obtained and sent to the screen sharing source device for caching.

3. The cross-device control method according to claim 1, characterized in that, The first peripheral operation data synchronously carries the hardware parameters of the screen display device.

4. The cross-device control method according to any one of claims 1-3, characterized in that, The hardware parameters include display parameters.

5. The cross-device control method according to any one of claims 1-3, characterized in that, The peripheral devices include one or more of a touch module, a mouse, and a keyboard.

6. The cross-device control method according to any one of claims 1-3, characterized in that, The wired interface is either a USB Type-A interface or a USB Type-C interface.

7. A cross-device control method, characterized in that, The cross-device control method for screen sharing source devices includes: In the screen sharing connection state, the first peripheral operation data is received from the screen sharing device through the wired interface. The first peripheral operation data is received by the screen sharing device from the screen sharing display device through the wireless network connection. The first peripheral operation data is generated by the peripheral device of the screen sharing display device. The first peripheral operation data is converted according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and send it to the screen sharing device, so that the screen sharing device encapsulates the second peripheral operation data based on the USB HID protocol to obtain HID message and sends it to the screen sharing source device. Respond according to the HID message.

8. The cross-device control method according to claim 7, characterized in that, Also includes: When establishing a screen sharing connection, the hardware parameters obtained by the screen sharing device from the screen sharing display device are acquired and cached. Accordingly, the step of converting the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and sending it to the screen sharing device includes: The first peripheral operation data is converted according to the conversion relationship between the cached hardware parameters and the hardware parameters of the screen sharing source device to obtain the second peripheral operation data, which is then sent to the screen sharing device.

9. The cross-device control method according to claim 7, characterized in that, The first peripheral device operation data synchronously carries the hardware parameters of the screen display device. Accordingly, the step of converting the first peripheral operation data according to the hardware parameters of the screen sharing display device and the screen sharing source device to obtain the second peripheral operation data and sending it to the screen sharing device includes: The first peripheral operation data is converted according to the conversion relationship between the synchronously carried hardware parameters and the hardware parameters of the screen transmission source device to obtain the second peripheral operation data, which is then sent to the screen transmission device.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the cross-device control method as described in any one of claims 1-9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-device control method as described in any one of claims 1-9.

Images