Cross-device window dragging method and device and storage medium

Abstract

The invention provides a cross-device window dragging method and device and a storage medium. The method is applied to a first electronic device of a Windows system and a second electronic device of an Android system, and when a mouse moving by dragging a first window displayed in the first electronic device passes through the second electronic device, the first electronic device sends a window passing instruction to the second electronic device; the second electronic equipment can start the first floating window which does not display the first window, so that a user visually perceives that the first window is traversed. After the mouse is released, the first electronic equipment notifies the second electronic equipment user of releasing the pressing on the left key of the mouse, so that the second electronic equipment can release the first floating window and start the second floating window serving as the expansion screen of the first electronic equipment, and the first window can be displayed in the display area of the second floating window; therefore, cross-device dragging of the window is realized in a keyboard and mouse sharing scene.

Description

Technical Field

[0001] This application relates to the field of electronic devices, and more particularly to a method, device, and storage medium for dragging windows across devices. Background Technology

[0002] Keyboard and mouse sharing refers to the ability of multiple electronic devices that establish a keyboard and mouse sharing connection to share a single set of input / output (I / O) devices, such as a keyboard and mouse. In other words, in a keyboard and mouse sharing scenario, a user can use the same mouse to operate multiple electronic devices.

[0003] However, with the diversification of user needs, simply using the same mouse to navigate between different electronic devices and operate them is no longer sufficient to meet user requirements. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a method, device, and storage medium for cross-device window dragging, aiming to enable cross-device window dragging in keyboard and mouse sharing scenarios, particularly dragging windows from Windows devices to Android devices, thereby better meeting user needs and improving user experience.

[0005] In a first aspect, this application provides a cross-device drag-and-drop method for windows, applicable to a system including a first electronic device and a second electronic device, wherein the first electronic device operates on a Windows system and the second electronic device operates on an Android system. The method includes: a first electronic device displaying a first window; the first electronic device dragging the first window within its screen in response to a user's dragging operation of the mouse cursor; when the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sending a window crossing instruction for the first window to a second electronic device and hiding the first window displayed on the first electronic device; wherein, the mouse corresponding to the mouse cursor is connected to either the first or second electronic device; the second electronic device displaying a first floating window on its screen in response to the window crossing instruction, the first floating window not displaying the first window; during the process of the mouse cursor dragging the first floating window within the screen of the second electronic device, the first electronic device sending a first instruction to the second electronic device in response to a user's release operation of the left mouse button; the second electronic device releasing the first floating window in response to the first instruction and displaying a second floating window on its screen, the second floating window covering the first floating window; and the second electronic device displaying a first collaborative screen in the display area of ​​the second floating window upon receiving a first collaborative screen sent by the first electronic device, the first collaborative screen including all the display content of the first window.

[0006] The first electronic device is, for example, a PC with a Windows system as described in the following embodiments, and the second electronic device is, for example, a tablet with an Android system as described in the following embodiments.

[0007] The first window is, for example, window 20a as described in the following embodiment.

[0008] The first floating window is, for example, the floating window 40a described in the following embodiments. The second floating window is, for example, the floating window 50a described in the following embodiments.

[0009] The first collaborative screen, for example, is the window 20b and taskbar 30b displayed in the display area of ​​the floating window 50a, as described in the following embodiment.

[0010] In the keyboard and mouse sharing scenario, various instructions sent by the first electronic device to the second electronic device, and various messages sent by the second electronic device to the first electronic device, are sent based on the keyboard and mouse sharing connection established between the first electronic device and the second electronic device.

[0011] The first collaborative screen, and the second collaborative screen described below, are transmitted based on the extended connection (screen collaborative connection) established between the first electronic device and the second electronic device.

[0012] In this aspect, the first electronic device and the second electronic device establish a keyboard and mouse sharing connection, that is, they are in a keyboard and mouse sharing scenario.

[0013] For details on the specific implementation process of dragging the first window from the first electronic device to the second electronic device, please refer to the descriptions of steps 101 to 139 and steps 201 to 227 in the following embodiments, which will not be repeated here.

[0014] This enables cross-device drag-and-drop of windows in keyboard and mouse sharing scenarios, especially dragging windows from Windows devices to Android devices.

[0015] According to the first aspect, after displaying the first collaborative screen in the display area of ​​the second floating window, the method further includes: the first electronic device displaying the second window; the first electronic device dragging the second window within the screen of the first electronic device in response to a user's dragging operation of the mouse cursor on the second window; when the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sending a second collaborative screen for the second window to the second electronic device, the second collaborative screen including a first part of the second window and displaying the second part of the second window on the screen of the first electronic device; wherein the first part is the part that moves out of the screen edge of the first electronic device, and the second part is the part that does not move out of the screen edge of the first electronic device; when the second electronic device receives the second collaborative screen sent by the first electronic device, it displays the second collaborative screen in the display area of ​​the second floating window; wherein the window state of the second window in the second collaborative screen is a restored state, and the second collaborative screen obscures the first collaborative screen; wherein when the window state is restored, the title bar of the first window includes a maximize control but does not include a restore control.

[0016] The second window displayed on the first electronic device, for example Figure 18 The window 70a corresponding to document 1 shown in (1) is displayed. The second collaborative screen displayed in the second floating window is, for example... Figure 18 Neutralize (3) Figure 18 The window 70b shown in (4) is shown in the middle.

[0017] The generation process for the second collaborative screen also requires the use of the compositing module. Since the first collaborative screen, which includes the taskbar of the first electronic device, is already displayed in the second floating window, the second collaborative screen can be generated simply by using the second window.

[0018] For the process of dragging the second window from the first electronic device to the second floating window already displayed on the second electronic device, please refer to the following instructions. Figure 18 The implementation details are omitted here.

[0019] According to the first aspect, or any implementation of the first aspect above, when the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sends a window crossing instruction for the first window to the second electronic device and hides the first window displayed on the first electronic device, including: when the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sends a window crossing instruction for the first window to the second electronic device; after sending the window crossing instruction to the second electronic device, when the first electronic device receives a window crossing confirmation message from the second electronic device for the window crossing instruction, it hides the first window displayed on the first electronic device.

[0020] For information on mouse cursor movement, crossing the first window, and hiding the first window, please refer to [link to documentation / reference]. Figure 12 , Figure 13 , Figure 14A , Figure 14B The steps 110 to 113, 207, 114 to 117 in the following embodiments will not be repeated here.

[0021] According to the first aspect, or any implementation of the first aspect above, after the second electronic device displays a first floating window on the screen of the second electronic device in response to the window crossing instruction, the method further includes: the second electronic device generating a window crossing confirmation message for the window crossing instruction and sending the window crossing message to the first electronic device.

[0022] The specific process of the second electronic device making a window crossing confirmation message and sending the window crossing message to the first electronic device can be found in steps 113, 201 to 207 in the following embodiments, and will not be repeated here.

[0023] According to the first aspect, or any implementation of the first aspect above, the first instruction includes a release window instruction and a screen collaboration instruction; the release window instruction is used to instruct the second electronic device to release the first floating window; the screen collaboration instruction is used to instruct the second electronic device to establish a screen collaboration connection with the first electronic device in extended mode; wherein, after the screen collaboration connection is successfully established, the second floating window is displayed on the screen of the second electronic device.

[0024] The screen collaboration indicator mentioned in this aspect is, for example, the extended mode indicator described in the following embodiments.

[0025] Regarding the operations performed by the second electronic device in accordance with the release window instruction and the screen collaboration instruction, please refer to steps 129, 217 to 221, 134, and 222 to 227 in the following embodiments, which will not be repeated here.

[0026] According to the first aspect, or any implementation of the first aspect above, after the second electronic device releases the first floating window in response to the first instruction and displays the second floating window on the screen of the second electronic device, the method further includes: the second electronic device generating a screen collaboration connection success message for the screen collaboration instruction and sending the screen collaboration connection success message to the first electronic device.

[0027] The screen collaboration indicator mentioned in this aspect is, for example, the extended mode indicator described in the following embodiments. The screen collaboration connection success message is, for example, the extended connection success message described in the following embodiments.

[0028] For details regarding the implementation of the extended connection success message (screen collaboration connection success message) issued by the second electronic device in response to the extended mode indication (screen collaboration indication) after releasing the first floating window, please refer to steps 134 and 222 to 227 in the following embodiments, which will not be repeated here.

[0029] According to the first aspect, or any implementation of the first aspect above, the first collaborative screen further includes the taskbar of the first electronic device; the method further includes: in response to the screen collaboration connection success message, the first electronic device sets the window state of the first window to a maximized state so that the first window displays all the display content included; the first electronic device synthesizes and draws the first collaborative screen based on the first window with the window state maximized and the taskbar of the first electronic device; and sends the obtained first collaborative screen to the second electronic device.

[0030] The taskbar of the first electronic device is, for example, taskbar 30a as described in the following embodiments and figures. For specific implementation details regarding the composite drawing of the first collaborative screen, please refer to step 138 in the following embodiments, which will not be repeated here.

[0031] According to the first aspect, or any implementation of the first aspect above, when the window is in the maximized state, the title bar of the first window includes a restore control but does not include a maximize control.

[0032] Regarding the style of the title bar of the first window when its window state is maximized, for example... Figure 3E The title bar of the window (window 20b) corresponding to document 2 displayed on the Android device.

[0033] According to the first aspect, or any implementation of the first aspect above, the first electronic device hides a first window displayed on the first electronic device, including: the first electronic device determining the width and height of the first window based on the handle of the first window; the first electronic device determining the current X and Y coordinates of the mouse cursor based on the mouse movement information; the first electronic device determining the hiding coordinates of the first window based on the current X and Y coordinates of the mouse cursor, the width of the first window, and the X coordinate range corresponding to the screen of the first electronic device; wherein the X coordinate range includes the X-axis coordinate of the left edge of the screen of the first electronic device and the X-axis coordinate of the right edge of the screen of the first electronic device, and the X-axis coordinate of the left edge of the screen is smaller than the X-axis coordinate of the right edge of the screen; the first electronic device hides the first window at the location of the hiding coordinates.

[0034] For information on how to determine the hidden coordinates, please refer to [link / reference needed]. Figure 12 The description of the illustrated embodiment will not be repeated here.

[0035] According to the first aspect, or any implementation of the first aspect above, the first electronic device determines the hidden coordinates of the first window based on the current X and Y coordinates of the mouse cursor, the width of the first window, and the X coordinate range corresponding to the screen of the first electronic device. This includes: when the X coordinate of the mouse cursor is not greater than the X-axis coordinate of the left edge of the screen, the first electronic device determines the difference between the X-axis coordinate of the left edge of the screen and the width of the first window; the first electronic device uses the difference as the X coordinate of the hidden coordinates and the Y coordinate of the mouse cursor as the Y coordinate of the hidden coordinates.

[0036] For information on the hidden coordinates corresponding to the first window crossing the left edge of the Windows device screen, please refer to [link / reference]. Figure 14B The description of the illustrated embodiment will not be repeated here.

[0037] According to the first aspect, or any implementation of the first aspect above, the first electronic device determines the hidden coordinates of the first window based on the current X and Y coordinates of the mouse cursor, the width of the first window, and the X coordinate range corresponding to the screen of the first electronic device, including: when the X coordinate of the mouse cursor is not less than the X-axis coordinate of the right edge of the screen, the first electronic device uses the X-axis coordinate of the right edge of the screen as the X coordinate of the hidden coordinates and uses the Y coordinate of the mouse cursor as the Y coordinate of the hidden coordinates.

[0038] For information on the hidden coordinates corresponding to the first window crossing the right edge of the Windows device screen, please refer to [link / reference]. Figure 14A The description of the illustrated embodiment will not be repeated here.

[0039] According to the first aspect, or any implementation of the first aspect above, the first electronic device hides the first window after the location of the hidden coordinates, and the method further includes: in response to a drag operation that moves the first window from the second electronic device back to the first electronic device, displaying the first window from the hidden coordinates.

[0040] According to the first aspect, or any implementation of the first aspect above, the first electronic device displays a first window, including: the first electronic device displays a first window in a maximized state; wherein, when the window is in a maximized state, the title bar of the first window includes a restore control but does not include a maximize control.

[0041] That is, before dragging the first window, it fills the entire display area of ​​the Windows device screen. In this scenario, the title bar of the first window looks like this: Figure 3E The title bar of the window (window 20b) corresponding to document 2 displayed on the Android device.

[0042] According to the first aspect, or any implementation of the first aspect above, the first electronic device responds to a user's drag operation of the mouse cursor on the first window, dragging the first window within the screen of the first electronic device, including: the first electronic device responds to a user's drag operation of the mouse cursor on the first window, switching the window state of the first window from a maximized state to a restored state; wherein, when the window state is restored, the title bar of the first window includes a maximize control but does not include a restore control; the first electronic device drags the first window in the restored state within the screen of the first electronic device.

[0043] That is, for the first window that is maximized, it is automatically switched to the restored state first, and then dragged and moved.

[0044] According to the first aspect, or any implementation of the first aspect above, the first electronic device displays a first window, including: the first electronic device displays a first window in a restored state; wherein, when the window state is restored, the title bar of the first window includes a maximize control but does not include a restore control. That is, the style of the title bar of the first window is as follows: Figure 2A The title bar of the window (window 20a) corresponding to document 2 displayed on the Windows device.

[0045] According to the first aspect, or any implementation of the first aspect above, the first electronic device responds to a user's drag operation of the mouse cursor on the first window, dragging the first window to move within the screen of the first electronic device, including: the first electronic device responds to a user's drag operation of the mouse cursor on the first window, and the dragged window, whose window state is restored, moves within the screen of the first electronic device. That is, direct dragging, such as... Figure 2A The window (window 20a) corresponding to document 2 displayed on the Windows device is shown in the image.

[0046] According to the first aspect, or any implementation of the first aspect above, during the process of dragging the first window within the screen of the first electronic device, the method further includes: during the process of dragging the first window within the screen of the first electronic device, the first electronic device determines, based on the mouse movement information, whether the mouse cursor corresponding to the mouse moves out of the screen of the first electronic device from the left edge or the right edge of the screen of the first electronic device; when it is determined that the mouse cursor corresponding to the mouse moves out of the screen of the first electronic device from the left edge or the right edge of the screen of the first electronic device, the first electronic device determines that the mouse cursor has moved out of the edge of the screen of the first electronic device.

[0047] For details on the implementation of this aspect, please refer to the specific descriptions of steps 109 and 110 in the following embodiments, which will not be repeated here.

[0048] According to the first aspect, or any implementation of the first aspect above, the first electronic device responds to a user's drag operation of the mouse cursor on the first window, and drags the first window to move within the screen of the first electronic device, including: when the first electronic device hears a window movement message of the first window and hears a left mouse button press message, it determines that the user has performed a drag operation on the first window by operating the mouse cursor; the first electronic device responds to the drag operation and drags the first window to move within the screen of the first electronic device.

[0049] For details regarding the user's dragging operation of the mouse cursor on the first window, and the specific operations performed by the first electronic device in response to the dragging operation, please refer to the description of steps 101 to 107 in the following embodiments, which will not be repeated here.

[0050] According to the first aspect, or any implementation of the first aspect above, the first electronic device, in response to the user's release operation on the left mouse button, sends a first instruction to the second electronic device, including: when the first electronic device hears the release message on the left mouse button and hears the message that the first window stops moving, it determines that the user has performed a release operation on the left mouse button; the first electronic device, in response to the release operation, sends a first instruction to the second electronic device.

[0051] The first indication may include the release window indication mentioned in step 125 and the extended mode indication (screen collaboration indication) in step 131 in the following embodiments.

[0052] For details regarding the user's release of the left mouse button and the first electronic device's response to the release operation by sending a first instruction to the second electronic device, please refer to the description of steps 123 to 134 in the following embodiments, which will not be repeated here.

[0053] According to the first aspect, or any implementation of the first aspect above, the second electronic device includes a screen computer and a foldable screen phone.

[0054] Therefore, by selecting an electronic device with a relatively large screen as the second electronic device, the second floating window displayed on the second electronic device can be relatively large, thereby enabling the first window dragged from the first electronic device to the second floating window of the second electronic device to be tiled, making it easier for the user to view the content in the window.

[0055] Secondly, this application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, which, when executed by the processor, cause the electronic device to perform the methods of the first aspect or any possible implementation thereof.

[0056] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0057] Thirdly, this application provides a computer-readable medium for storing a computer program including instructions for performing the methods in the first aspect or any possible implementation thereof.

[0058] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0059] Fourthly, this application provides a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0060] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0061] Fifthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

[0062] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0063] Sixthly, this application provides a system comprising a first electronic device and a second electronic device, wherein the first electronic device operates on a Windows system and the second electronic device operates on an Android system;

[0064] The first electronic device and the second electronic device may execute the method in the first aspect or any possible implementation of the first aspect to enable cross-device dragging of windows.

[0065] The sixth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the sixth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0066] Figures 1A to 1D This is an illustrative diagram of a document drag-and-drop scenario in a keyboard and mouse sharing environment.

[0067] Figures 2A to 2C This is an illustrative diagram of a window dragging scenario in a keyboard and mouse sharing environment;

[0068] Figures 3A-3E This is an illustrative diagram illustrating window dragging in a keyboard and mouse sharing scenario.

[0069] Figures 4A to 4C This is an illustration of the sidebar style corresponding to different states of the floating window on the Android device after an extended connection is established between a Windows device and an Android device, as shown in the example.

[0070] Figure 5 This is an example illustration of the operation of the taskbar corresponding to control 50a-4 on the floating window 50a;

[0071] Figure 6A and Figure 6B This is a schematic diagram illustrating the scaling down and scaling up of the floating window 50a as an example;

[0072] Figure 7 This is an illustrative diagram showing the change of the floating window 50a after an Android device is rotated from landscape mode to portrait mode.

[0073] Figure 8A and Figure 8B This is a schematic diagram of the software architecture of a Windows device as an example.

[0074] Figure 9A and Figure 9B This is a schematic diagram of the software architecture of an exemplary Android device.

[0075] Figure 10A timing diagram illustrating the interaction of functional modules involved in implementing the cross-device drag-and-drop method for windows provided in this application embodiment on both Windows and Android devices, as exemplarily shown.

[0076] Figure 11 As an example shown Figure 10 A timing diagram of the interaction of functional modules involved in implementing the cross-device drag-and-drop method for windows provided in the embodiments of this application on a Windows device;

[0077] Figure 12 This is an illustrative diagram showing the relationship between a composited canvas and the screen of a Windows device.

[0078] Figure 13 A flowchart illustrating the calculation of the hidden position of window 20a as an example;

[0079] Figure 14A and Figure 14B This is a schematic diagram illustrating the hidden position of window 20a as an example.

[0080] Figure 15 For example Figure 10 A timing diagram of the interaction of functional modules involved in implementing the cross-device drag-and-drop method for windows provided in the embodiments of this application on an Android device;

[0081] Figure 16A This is another timing diagram showing the interaction of functional modules involved in implementing the cross-device drag-and-drop method for windows provided in the embodiments of this application on an exemplary Android device;

[0082] Figure 16B This is an illustrative diagram showing the determination of aspect ratios for extended screens of different sizes.

[0083] Figure 17 This is an illustrative diagram showing the interface changes of the Windows device and the Android device during the first dragging of window 20a to the Android device when the collaboration function is not enabled, based on the cross-device dragging method of the window provided in the embodiments of this application.

[0084] Figure 18 This is an illustrative diagram showing the interface changes of the Windows device and the Android device during the process of dragging window 70a to the Android device when the collaboration function is activated and the floating window 50a is displayed, based on the cross-device dragging method of the window provided in the embodiments of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0087] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0088] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0089] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0090] With the widespread use of electronic devices, more and more users own multiple (at least two) electronic devices. To facilitate users' simultaneous use of multiple electronic devices for work and entertainment, keyboard and mouse sharing functionality has become increasingly popular. Based on this function, electronic devices such as mobile phones and tablets that do not have or are not connected to input / output (I / O) devices such as keyboards and mice can share the computer's keyboard, mouse, or touchpad.

[0091] Specifically, in this application, electronic devices that do not have or are not connected to I / O devices refer to Android devices (electronic devices with the Android operating system), and computers refer to Windows devices (electronic devices with the Windows operating system).

[0092] In other words, Android devices and Windows devices with keyboard and mouse sharing capabilities can establish a connection, allowing the Windows mouse to move freely between the two devices. This enables users to control applications, view and edit files on different electronic devices using the same mouse.

[0093] Regarding the establishment of a keyboard and mouse sharing connection, one possible implementation is to place the Android device that needs to establish the keyboard and mouse sharing connection next to the Windows device, and then move the mouse pointer of the Windows device to the side edge of the Windows device's screen and tap the side edge of the Windows device's screen twice in succession to establish the keyboard and mouse sharing connection.

[0094] For example, in some other possible implementations, the Android device that needs to establish a keyboard and mouse sharing connection can be placed next to the Windows device, and then the Windows device can access the PC Manager, and the keyboard and mouse sharing entry under the Smart Interconnection directory in the PC Manager can be used to establish a keyboard and mouse sharing connection between the Windows device and the Android device placed next to it.

[0095] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the sole limitation of this embodiment. For specific implementation details regarding establishing a keyboard and mouse sharing connection between an Android device and a Windows device, please refer to existing keyboard and mouse sharing documentation, which will not be repeated here.

[0096] Furthermore, it's worth noting that after establishing a keyboard and mouse sharing connection between the Android and Windows devices—that is, when the Android and Windows devices are in a keyboard and mouse sharing scenario—the Windows device's mouse can not only move freely between the Android and Windows devices, but in some possible scenarios, users can also control the Windows device's mouse to drag and drop images and documents between the two electronic devices. For example, a user can use the Windows device's mouse to drag and drop documents from the Windows device to the Android device.

[0097] In addition, it should be noted that after completing the drag-and-drop transfer of the document, the document will exist not only on the Windows device but also on the Android device, allowing users to view the document on both Windows and Android devices.

[0098] To better understand the above-mentioned drag-and-drop document transfer using a mouse on a Windows device, the following will combine... Figures 1A to 1D Please provide a detailed explanation.

[0099] See Figure 1A For example, this embodiment uses a Windows device as the PC, and the Android devices that establish a keyboard and mouse sharing connection with the PC are a tablet and a foldable phone. Figure 1A As shown, when a user uses the PC mouse to manipulate Document 2 placed on the PC desktop, for example, moving the mouse cursor to Document 2, pressing the left mouse button, and dragging Document 2, in a keyboard and mouse sharing scenario, as the user presses the left mouse button and drags the Document 2 icon, the Document 2 icon will move along with the mouse cursor, such as from... Figure 1A The indicated position was moved to Figure 1B The location shown.

[0100] For example, if the user continues to hold down the left mouse button and drag the document 2 icon, when the mouse cursor moves to the side edge of the PC screen, such as... Figure 1C The right edge of the screen. Through a keyboard and mouse sharing connection between the PC and tablet, document 2 will be transferred to the tablet, and its icon will appear on the tablet's screen, as shown below. Figure 1D As shown. At the same time, the icon for document 2 is still displayed on the PC screen.

[0101] In other words, currently in keyboard and mouse sharing scenarios, users can use the same mouse to drag and drop images and documents between different electronic devices. If a user wants to drag and drop an already opened document, they need to first close the document window using the close control in the document's title bar, and then use the aforementioned method... Figures 1A to 1D Only the drag-and-drop method shown can be used to drag the document to other electronic devices. That is, the currently open window cannot be dragged from a Windows device to an Android device; cross-device dragging of windows is not possible.

[0102] In view of this, this application provides a method for cross-device dragging of windows. Based on the cross-device dragging method provided in this application, it is possible to drag windows across devices in keyboard and mouse sharing scenarios, especially to drag windows from Windows devices to Android devices.

[0103] Regarding the cross-device dragging of windows described in this application, the changes displayed on the screens of Windows devices and Android devices are as follows: Figures 2A to 2C .

[0104] See Figure 2AFor example, let's take a Windows device as the PC and an Android device that establishes a keyboard and mouse sharing connection with the PC as the tablet. Figure 2A In the keyboard and mouse sharing scenario shown, when the user operates the PC's mouse, opens document 2 on the PC, and moves the mouse cursor ( Figure 2A Move the mouse cursor (shown as an arrow) to the title bar of the window corresponding to Document 2, and press and hold the left mouse button while dragging it to the right edge of the PC screen. The window of Document 2 will move to the right edge of the PC screen along with the mouse cursor.

[0105] See Figure 2B For example, when the window of document 2 moves to the right edge of the PC screen as the mouse cursor moves, if the user continues to hold down the left mouse button and move the cursor to the right, based on the cross-device dragging method of the window provided in this application, the window of document 2 on the PC will be dragged to the screen of the tablet located on the right side of the PC and with which it has established a keyboard and mouse sharing connection, as shown below. Figure 2C As shown. That is, based on the cross-device drag-and-drop method for windows provided in this application, the window of document 2 displayed on the PC will be displayed as a floating window on the tablet screen, and at this time the window of document 2 will no longer be displayed on the PC screen.

[0106] See also Figure 2C For example, after the window of document 2 moves from the PC to the tablet, the mouse connected to the PC will also move to the tablet. That is, the mouse cursor will move from the PC to the tablet. In one possible implementation, after the mouse cursor moves from the PC to the tablet, the displayed mouse cursor can be as follows: Figure 2C The dot pattern shown in the image. Furthermore, upon first traversal, the mouse cursor can be on state 1 of the floating window displayed on the tablet.

[0107] Regarding the switching of mouse cursor styles in different positions, setting the initial position, and... Figure 3A The functions of status bar 1 and status bar 2 shown in the figure are detailed in the following embodiments, and will not be repeated here.

[0108] Therefore, when a keyboard and mouse sharing connection is established between a Windows device and an Android device, i.e., in a keyboard and mouse sharing scenario, and the user uses a Windows device, such as the PC mentioned in the above embodiment, to open multiple applications and needs to use these applications simultaneously, such as viewing the content in the corresponding windows of these applications, the cross-device dragging method of the window provided in this application can monitor the dragging of the windows corresponding to the applications / documents on the PC, and combine this with mouse crossing control in keyboard and mouse sharing. When the mouse crosses over, the position of the window moving off the PC screen is calculated, and the relevant information of the window is transmitted to the Android device, such as the tablet mentioned in the above embodiment. When the window is released on the tablet (the user releases the left mouse button), a floating window will be launched on the tablet as an extended screen of the PC, and the window of document 2 mentioned in the above embodiment will be displayed in the floating window. Furthermore, the window of document 2 will no longer be displayed on the PC, thus realizing the experience of dragging and dropping windows from the PC to the tablet for display.

[0109] Furthermore, it should be noted that the cross-device drag-and-drop window method provided in this application is intended to enable Windows devices in keyboard and mouse sharing scenarios to achieve dual-screen or even multi-screen (three or more) office experiences via Android devices. That is, the window dragged by the mouse also needs to be displayed as a window. Therefore, the Android device mentioned in this application can be an Android device with a relatively large screen, such as an Android tablet or an Android foldable phone. This allows the currently used window to be tiled, making it easier for the user to view the content within the window.

[0110] Furthermore, it should be noted that in order to enable the tablet to act as an extended screen for the PC, and thus allow windows to be dragged and dropped from the PC onto the tablet in a keyboard and mouse sharing scenario, the cross-device drag-and-drop method for windows provided in this application also requires the use of screen collaboration functionality (hereinafter referred to as: collaboration functionality).

[0111] Specifically, in the various embodiments of this application, the collaborative function includes: using distributed technology to collaboratively connect devices from different systems, thereby using the Sink end (receiving end) as an extended screen of the Source end (sending end), thus realizing a multi-screen office experience.

[0112] Based on the collaboration feature, users can achieve a multi-screen (at least 2 screens) experience. For example, multiple files (documents, images, etc.) on the source side, such as file A and file B, can be displayed side-by-side, allowing users to view files A and B simultaneously.

[0113] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0114] Furthermore, it should be noted that the collaboration function has two modes: mirror mode and extended mode. Mirror mode means that the content displayed in the floating window on the tablet is exactly the same as that on the PC screen. Extended mode means that the floating window on the tablet displays additional content not shown on the PC screen.

[0115] Therefore, in order to achieve the effect of dragging and dropping the window corresponding to document 2 on the PC to display on the tablet, this application adopts the extended mode in the collaborative mode.

[0116] To better understand the process from dragging a window from a Windows device to the Android device connecting with the Windows device, activating the extended mode of the collaborative function, and finally displaying the window dragged from the Windows device on the Android device's screen, the following detailed explanation of the changes on the screens of the Windows device and the Android device during this process, with reference to the accompanying diagrams.

[0117] Using the Windows device mentioned in the above embodiment as a PC, the window to be dragged as the window of document 2 displayed on the PC, and the Android device to be dragged as a tablet as an example. For ease of explanation, in this embodiment, the arrow-shaped cursor corresponding to the mouse on the PC is called cursor 10a, the dot-shaped cursor corresponding to the mouse on the tablet is called cursor 10b, the window corresponding to document 2 on the PC is called window 20a, the window corresponding to document 2 on the tablet is called window 20b, the taskbar displayed at the bottom of the PC screen is called taskbar 30a, the floating window displayed before the collaboration function is activated when the mouse is moved to the tablet is called floating window 40a, the floating window that replaces floating window 40a after the collaboration function is activated (equivalent to the PC's extended screen) is called floating window 50a, the taskbar 30a displayed in floating window 50a is referred to as taskbar 30b, the status bar at the top of both floating window 40a and floating window 50a is called status bar 1, and the status bar at the bottom is called status bar 2.

[0118] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0119] Furthermore, it should be noted that in the keyboard and mouse sharing scenarios described in the embodiments of this application, the mouse that can be shared between Windows devices and Android devices is... Figures 1A to 1DThe example shown is a mouse connected to a PC. The mouse will not be shown in subsequent figures.

[0120] For example, when a user holds down the left mouse button and drags window 20a to the right edge of the PC screen (e.g., ... Figure 2B As shown, if you do not release the left mouse button and continue dragging window 20a to the right, cursor 10a will gradually move closer to the right edge of the PC screen. To achieve the effect of window 20a moving from the PC to the tablet, as cursor 10a moves closer to the right edge of the PC screen, part of window 20a will be moved off the PC screen; that is, only the portion before the cursor 10a's position will be displayed on the PC screen. For example... Figure 3A As shown. Because, Figure 3A In the middle, the cursor 10a has not yet moved to the right edge of the PC screen, that is, the mouse has not yet crossed the edge, so the cursor 10b will not be displayed on the tablet, nor will window 20a.

[0121] See Figure 3B For example, when the user presses the left mouse button and continues dragging window 20a, and cursor 10a moves to the right edge of the PC screen, based on the keyboard and mouse sharing connection established between the PC and the tablet, the PC's mouse cursor will move from the PC to the tablet; that is, cursor 10a will move to the tablet and change to the style of cursor 10b. At the same time, a floating window 40a, approximately one-third the size of the cursor, will appear on the tablet, while window 20a on the PC will be hidden. Figure 3C As shown.

[0122] It's important to note that before the user releases the left mouse button to activate the collaboration feature and before the floating window 50a, acting as an extended screen for the PC, appears on the tablet, the user can at any time move the mouse back to the PC. At this point, the dragged window 20a will also return to the PC screen. Therefore, before activating the collaboration feature, to visually suggest that window 20a has moved from the PC to the tablet, the floating window 40a is displayed first. That is, the floating window 40a is not an extended screen for the PC; it's simply a floating window drawn and displayed on the tablet. After the floating window 50a, acting as an extended screen for the PC, is activated, it will overlay the floating window 40a, and the floating window 40a will be destroyed.

[0123] Therefore, in order to prevent users from perceiving that floating window 40a has been replaced by floating window 50a, in some possible cases, floating window 40a and floating window 50a can be drawn to the same size, and the selection controls in status bar 1 and status bar 2 can have the same function.

[0124] This embodiment only uses the example of setting functional controls in status bar 1 for illustration, but in actual applications, functional controls can also be set in status bar 2 according to business needs.

[0125] like Figure 3D and Figure 3E As shown, control 40a-1 in floating window 40a has the same function as control 50a-1 in floating window 50a; both are used to close the floating window currently displayed on the tablet screen. Regarding control 50a-1, since the collaboration function is already enabled, clicking this control will disconnect the collaboration between the tablet and the PC, meaning the tablet will no longer act as an extended screen for the PC.

[0126] See also Figure 3D and Figure 3E Controls 40a-2 in floating window 40a and 50a-2 in floating window 50a have the same function: maximizing the currently displayed floating window on the tablet screen. Controls 40a-3 in floating window 40a and 50a-3 in floating window 50a have the same function: minimizing the currently displayed floating window on the tablet screen. Controls 40a-4 in floating window 40a and 50a-4 in floating window 50a have the same function: displaying a hidden taskbar at the top edge of the current floating window.

[0127] See also Figure 3D For example, if a user holds down the left mouse button and drags the floating window 40a until it is fully displayed on the tablet screen, the PC will send a release instruction for window 40a to the tablet based on the keyboard and mouse sharing connection. Upon receiving this instruction, the tablet will release window 40a. This release of window 40a can be understood as the tablet injecting an "up" event into the mouse. Specific implementation details of this operation can be found in the description of step 221 in the following embodiments, and will not be repeated here.

[0128] When the mouse is released and the floating window 40a is displayed on the tablet, the PC will activate the collaboration function and establish a collaboration connection with the tablet in extended mode. After the collaboration connection is established, the tablet will create a floating window 50a, decompress the collaboration screen transmitted by the PC through the collaboration connection, and then display the collaboration screen in the display area of ​​the floating window 50a.

[0129] The display area of ​​the floating window 50a is the area of ​​the floating window 50a with status bar 1 and status bar 2 removed.

[0130] Furthermore, since the floating window 50a serves as an extended screen for the PC, the PC's taskbar 30a will be displayed within the floating window 50a. Therefore, the collaborative screen needs to include both window 20b and taskbar 30b.

[0131] Furthermore, to provide a better user experience, one possible implementation is to set the collaborative screen to fill the entire display area of ​​the floating window 50a. This way, the user visually perceives window 20a as being dragged entirely onto the tablet. Therefore, when generating the collaborative screen, the compositing (drawing) rendering module in the PC specifically draws window 20b based on window 20a in maximized mode. In other words, window 20b is essentially window 20a in maximized mode.

[0132] For example, in this embodiment, the title bar of the non-maximized window 20a includes a maximize control for maximizing the window 20a, such as... Figure 2A As shown. For window 20b drawn based on window 20a in maximized mode, since it is essentially window 20a in maximized mode, therefore, in various embodiments of this application, the functional controls displayed in the title bar of window 20b displayed in the floating window 50a dragged onto the tablet will not include the maximize control; instead, a restore control will be displayed, such as... Figure 3C As shown.

[0133] In this way, after using floating window 50a to cover floating window 40a, the user can directly see window 20b, such as... Figure 3E As shown.

[0134] See also Figure 3D For example, while waiting for the PC and tablet to establish a collaborative connection, and while the floating window 50a covers the floating window 40a, the floating window 40a can also display dynamically loading icons, such as... Figure 3D The icons 40a-5 shown are used to indicate to the user that window 20a is currently loading.

[0135] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0136] See Figure 3E For example, the cross-device drag-and-drop window solution provided in this embodiment does not use the entire tablet screen as an extended screen for the PC. Instead, a floating window 50a is first displayed on the tablet, and the display area within the floating window 50a is used as an extended screen for the PC. Therefore, in order not to obscure the content of window 20a within the floating window 50a, the sidebar that would normally appear at the edge of the extended screen in the extended mode of the collaboration function (usually the left / right edge of the screen for tablets, and the top / bottom edge of the screen for foldable phones) will be hidden. That is, the sidebar is not displayed in the floating window 50a.

[0137] For example, in some possible implementations, a hotspot can be set on the right edge of the floating window 50a. When the user moves the cursor 10b to this area, presses the left mouse button and slides to the left, or clicks / doubles the left mouse button, the hidden sidebar is then displayed on the right edge of the floating window 50a.

[0138] For example, in some other possible implementations, an operable control can also be placed on the right edge of the floating window 50a, such as... Figure 3E The control 50a-5 is shown in the image. When the user moves the cursor 10b to the control 50a-5 and clicks the left mouse button, the hidden sidebar will be displayed on the right edge of the floating window 50a.

[0139] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the sole limitation of this embodiment. For ease of explanation, the hiding and showing of the sidebar is illustrated in this embodiment. Figure 3E The form of the control shown is illustrated in 50a-5.

[0140] Based on the above definition of cursor styles, when the cursor is on a PC, its style is that of cursor 10a; when the cursor is on a tablet, its style is that of cursor 10b. Since the floating window 50a serves as an extension of the PC's display screen, when the cursor moves to the display area within the floating window 50a, i.e. Figure 4A When the area displaying window 20b and taskbar 30b is shown, it is equivalent to operating on the PC, so the cursor will switch from the style of cursor 10b to the style of cursor 10a.

[0141] See Figure 4A In step (1), when the cursor 10a moves to control 50a-5, and the user clicks the left mouse button, control 50a-5 will disappear and be replaced by... Figure 4A The sidebar 50a-6 is shown in (2).

[0142] Understandably, the sidebar 50a-6 will display function options related to the extended mode, limited by factors such as... Figure 4A The size of the middle (2) area is not shown in sidebars 50a-6. To better understand the functional options included in the sidebar, the following is a combination of... Figure 4B Please provide an explanation.

[0143] See Figure 4BFor example, as an optional case, the sidebar 50a-6 may include function options for returning to the desktop, function options for saving the content in the extended screen, function options for undoing the last operation, function options for moving the current window to the other end for display, function options for switching collaboration modes, function options for port collaboration, etc., which will not be listed here, and this embodiment does not limit it.

[0144] Understandably, in one possible implementation, the sidebar 50a-6 can be set to disappear when any function option in the sidebar is selected. In another possible implementation, the sidebar 50a-6 can be set to disappear automatically after a set display period, at which point the control 50a-5 can be redisplayed in the floating window 50.

[0145] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0146] Furthermore, regarding the function option for switching collaboration modes, if this function option is selected while the system is in extended mode, the collaboration mode will switch from extended mode to mirror mode. Conversely, if this function option is selected while the system is in mirror mode, the collaboration mode will switch from mirror mode to extended mode. Because of the difference between mirror mode and extended mode, the function options displayed in the sidebar will also change. The specific selection can be determined based on these two collaboration modes, and this embodiment does not impose any restrictions on this.

[0147] See also Figure 3E For example, if a user moves the cursor 10b to control 50a-2 and clicks the left mouse button, the floating window 50a will be maximized and displayed on the tablet screen, as shown. Figure 4C As shown. In this case, status bars 1 and 2 of the floating window 50a will not be displayed, but in maximized mode, the sidebar corresponding to the current collaboration mode will be displayed on the left or right side of the tablet screen (the left side is used as an example in this embodiment), such as... Figure 4C The sidebar 50a-7 is shown in the image. Taking the current collaboration mode as the extended mode as an example, sidebar 50a-7 can display the same functional options as sidebar 50a-6.

[0148] Since window 20b in the collaboration view is drawn based on the maximized window 20a, the collaboration view received from the PC will also be enlarged as a whole after the floating window 50a is maximized. Therefore, when the floating window 50a is maximized, window 20b and taskbar 30b will still occupy the entire display area of ​​the floating window 50a, such as... Figure 4C As shown.

[0149] Furthermore, operations on the floating window 50a can also be performed through the function options in the taskbar corresponding to control 50a-4. For example, when the cursor 10b moves over control 50a-4, clicking the left mouse button will bring up a hidden taskbar in the area where control 50a-4 is located. Figure 5 The taskbar 50a-8 shown in (1) is shown.

[0150] See Figure 5 In (1), for example, in some possible implementations, the taskbar 50a-8 may include the same functional controls as controls 50a-1, 50a-2, and 50a-3, as well as the functional options for switching collaborative modes shown in sidebar 50a-6 or sidebar 50a-7.

[0151] For example, when a user clicks the same functional control as control 50a-2 in the taskbar 50a-8, the floating window 50a will be maximized, that is, the tablet displays the interface content shown in Figure 4 (2).

[0152] For example, when a user clicks the same functional control as control 50a-3 in taskbar 50a-8, the floating window 50a will be minimized. In one possible implementation, the minimized floating window will... Figure 5 The style of capsule 60a shown in Figure (2) is displayed. When the user clicks on capsule 60a, the floating window 50a will be re-displayed in the tablet interface, as shown in Figure 4 (1).

[0153] For example, when a user clicks the same functional control as control 50a-1 in the taskbar 50a-8, the tablet will disconnect from the PC and close the floating window 50a. That is, the tablet will revert to the style when window 20 was dragged to the front of the tablet, as shown below. Figure 3B The flat plate shown.

[0154] It should be noted that when the collaborative connection between the tablet and the PC is lost, in some possible implementations, the PC can automatically display window 20a. In other possible implementations, the user can click on a specified location or control to redisplay window 20a on the PC screen.

[0155] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0156] For example, when a user clicks the function option in the taskbar 50a-8 for switching collaboration modes, the collaboration mode will switch, and the content displayed in the floating window 50a will change accordingly. For instance, when switching to mirror mode, the floating window 50a will display the PC's current desktop. When switching back to extended mode, windows dragged into the floating window 50a will be displayed, such as window 20a mentioned above.

[0157] As described above, clicking control 50a-4 will display taskbar 50a-8. Control 40a-4 in the floating window 40a has the same function as control 50a-4. Therefore, to prevent the cursor 10b on the tablet interface from clicking control 40a-4 when the mouse just moves from the PC to the tablet, the cursor 10b appearing on the tablet can be displayed in the status bar 1 of the floating window 40a, and positioned away from control 40a-4. Figure 3C As shown.

[0158] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0159] In addition, it should be noted that in this embodiment, the user can also zoom in or out of the floating window 50a displayed on the tablet by manipulating the mouse.

[0160] For example, when the user moves the cursor to the edge of the floating window 50a, that is, after the display area of ​​the floating window 50a is removed, the cursor style will switch back to cursor 10b.

[0161] See Figure 6A In example (1), when the cursor 10b moves to the lower right corner of the floating window 50a, and the user presses the left mouse button and drags the floating window 50a in the direction of the arrow, the floating window 50a will shrink as a whole. When dragged to... Figure 6A After the user releases the left mouse button at the position shown in (2), the floating window 50a will shrink to the position shown in (2). Figure 6A The size of (2) is determined and it remains in that position.

[0162] See Figure 6B In example (1), when the cursor 10b moves to the lower right corner of the floating window 50a, and the user presses the left mouse button and drags the floating window 50a in the direction of the arrow, the floating window 50a will be enlarged as a whole. When dragged to... Figure 6B After the user releases the left mouse button at the position shown in (2), the floating window 50a will enlarge to Figure 6B The size of (2) is determined and it remains in that position.

[0163] Furthermore, it should be noted that, in order to ensure the user experience, based on the window dragging method provided in this embodiment, when the tablet displaying the floating window 50a rotates from landscape mode to portrait mode, such as rotating from 7 (1) in the direction of the arrow... Figure 7 After (2), the floating window 50a will also rotate as a whole, and the size of the floating window 50a will be adjusted according to the horizontal distance after the vertical screen.

[0164] Furthermore, in some possible implementations, the floating window 50a will be centered from both the left and right sides when rotating from landscape to portrait or vice versa. However, when the user drags it, the floating window 50a will remain at the position the user dragged.

[0165] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0166] To better understand the cross-device dragging method for windows provided in this application, the following is combined with... Figure 8A , Figure 8B , Figure 9A and Figure 9B The software architectures of Windows devices and Android devices that implement cross-device drag-and-drop window functionality are described separately.

[0167] It should be noted that in practical applications, both Windows and Android devices can employ layered architectures, event-driven architectures, microkernel architectures, microservice architectures, or cloud architectures. Furthermore, both Windows and Android devices can be divided into user mode and kernel mode.

[0168] See Figure 8A This example illustrates a software architecture for a Windows device.

[0169] like Figure 8AAs shown, from a user-mode perspective, the components, from bottom to top, can include NTDLL, service processes, system processes, environment subsystems, data transmission modules, and user processes. Specifically, in this embodiment, cross-device dragging of windows mainly relies on the keyboard and mouse sharing service and collaboration service within the service processes, as well as the data transmission mode. The keyboard and mouse sharing service (process) is the service process that implements the aforementioned keyboard and mouse sharing function; the collaboration service (process) is the service process that implements the aforementioned collaboration function, responsible for establishing the collaborative connection between the Windows device and the Android device; the data transmission module is responsible for signaling and data transmission. In one implementation, when both the Windows device and the Android device are electronic devices manufactured by Honor, the data transmission module, for example, is MagicLink.

[0170] For information on the functions of other processes, systems, NTDLLs, etc. in user mode, please refer to the existing Windows system architecture description; it will not be elaborated here.

[0171] Furthermore, it should be noted that in some possible implementations, the user process may include the process corresponding to an application like the PC Manager mentioned above. Through the PC Manager's corresponding process (such as the business management process), various business functions and the scheduling of various virtualization services can be achieved. For example, scheduling of keyboard and mouse sharing services and collaboration services within the service process.

[0172] Furthermore, it should be noted that, since this embodiment improves upon the existing keyboard and mouse sharing scenario by enabling cross-device dragging of windows, the improved keyboard and mouse sharing service includes at least the following: Figure 8B The four functional modules are shown.

[0173] See Figure 8B For example, the keyboard and mouse sharing service in a Windows device includes a window management module, a system message module, a window settings module, and a protocol management module.

[0174] The window management module is responsible for managing dragged windows, managing window messages, and identifying their positions; the window settings module is responsible for calculating the positions of windows when they are moved, hidden, or shown; the system message module is responsible for sending messages when the mouse starts or stops dragging windows; and the protocol management module is responsible for assembling and unpacking data related to window dragging commands and window parameters.

[0175] It's important to note that in Windows programming, once a Windows application starts executing, the system creates a message queue for that application. This message queue stores information about the windows created by that application. For example, when a user presses the mouse button on a window, a WM_RBUTTONDOWN message is generated. The system automatically places this message into the message queue of the application to which the current window belongs, waiting for the application to terminate. Windows sequentially places generated messages into the message queue, and the application continuously reads messages from this queue through a message loop and responds accordingly. When the mouse moves or drags the window, the system generates a system message for window movement, which is the message managed by the aforementioned system message module.

[0176] See also Figure 8A For example, from a kernel-level perspective, the structure, from bottom to top, can include a hardware abstraction layer, a kernel and device driver layer, a window management and image rendering module, and the Windows executable. Specifically, in the technical solution provided in this embodiment, cross-device window dragging not only relies on keyboard and mouse sharing functionality but also on collaborative functionality. Both of these functions depend on WiFi or Bluetooth, therefore requiring WiFi and Bluetooth-related functional modules, such as WiFi drivers and Bluetooth drivers.

[0177] Furthermore, for Windows devices lacking I / O capabilities, external keyboards, mice, etc., need to be connected via USB, Bluetooth, or WiFi. Therefore, in addition to WiFi and Bluetooth-related modules, USB-related modules, such as USB drivers, are also required.

[0178] For a description of the Windows Executor in kernel mode, other functional modules involved in the kernel and device driver layers, window management and drawing modules, hardware abstraction layer, etc., please refer to the introduction of the existing Windows system architecture, which will not be repeated here.

[0179] That concludes the introduction to the software architecture of Windows devices. As you can understand, Figure 8A and Figure 8B The layers in the software Windows device shown, and the functional modules contained in each layer, do not constitute a specific limitation on the Windows device. In some possible implementations, the Windows device may include more or fewer layers than shown, and each layer may include more or fewer components; this application does not impose any limitations.

[0180] See Figure 9A This example illustrates a software architecture for a Windows device.

[0181] like Figure 9A As shown, the layered architecture of Android devices divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some implementations, the Android system is divided into five layers, from top to bottom: the Application Layer (APP layer), the Framework Layer (Application Framework, FWK layer), the Android Runtime and system libraries, the Hardware Abstraction Layer (HAL layer), and the Kernel Layer (Linux Kernel layer).

[0182] The APP layer can include a series of application packages. For example... Figure 9A As shown, the application package may include applications such as camera, games, video, and live streaming, which will not be listed here, and this application does not impose any restrictions on them.

[0183] The FWK layer provides application programming interfaces (APIs) and programming frameworks for the APP layer applications. In some implementations, these APIs and frameworks can be described as functions. For example... Figure 9A As shown, the FWK layer may include functions such as content providers, keyboard and mouse sharing services, collaboration services, and data transfer modules, which will not be listed here, and this application does not impose any restrictions on them.

[0184] The functions of the collaboration services and data transfer modules in Android devices are the same as those in Windows devices, and will not be repeated here.

[0185] The keyboard and mouse sharing service on Android devices differs from that on Windows devices. Its main functional modules include, for example... Figure 9B As shown, there are modules such as window management, window settings, protocol management, and virtual data injection. The window management, window settings, and protocol management modules function identically to those in Windows devices, and will not be described further here.

[0186] Because the user did not release the left mouse button and then press it again when the mouse moved from a Windows device to an Android device, the cursor 10b displayed on the status bar 1 of the aforementioned floating window 40a (as shown in the image) appears to be missing. Figure 3BAs shown, there is no corresponding "down" event on the tablet side. Therefore, when the user releases the mouse and the PC sends a window release instruction to the tablet, the tablet cannot detect the "up" event, thus failing to release the floating window 40a. To enable the tablet to release the floating window 40a, in this embodiment, the keyboard and mouse sharing service on the tablet actively registers a "down" event through a virtual data injection module when the mouse cursor moves across the screen. This way, when the user subsequently releases the left mouse button, an "up" event will be detected.

[0187] See also Figure 9A For example, the Android Runtime includes core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0188] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0189] The application layer and the framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0190] See also Figure 9A For example, a system library may include multiple functional modules. Examples include: a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), etc.

[0191] The Surface Manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing.

[0192] Understandably, the 2D graphics engine mentioned above is a 2D drawing engine.

[0193] See also Figure 9AFor example, the HAL layer is an interface layer located between the operating system kernel (kernel layer) and the hardware circuitry. Its purpose is to isolate the FWK layer from the kernel so that Android does not become overly dependent on the kernel, thereby allowing FWK development to be carried out without considering drivers.

[0194] See also Figure 9A For example, the HAL layer may include various interfaces, such as audio and video interfaces, USB interfaces, Bluetooth interfaces, WiFi interfaces, etc., which will not be listed here, and this application does not limit them.

[0195] See also Figure 9A For example, the kernel layer in the Android system is the layer between hardware and software. The kernel layer may include various processes / threads, power management modules, and various drivers, such as WiFi drivers, Bluetooth drivers, and kernel drivers (Uinput drivers) used to simulate user input events.

[0196] That concludes the introduction to the software architecture of Android devices. As you can understand, Figure 9A and Figure 9B The layers in the illustrated Android device and the functional modules contained in each layer do not constitute a specific limitation on the Android device. In some possible implementations, the Android device may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0197] To better understand the specific implementation of the cross-device drag-and-drop method for windows provided in this application, the following description, in conjunction with the accompanying drawings, details the interaction logic of the keyboard and mouse sharing service (including functional modules), collaboration service, and data transmission module in Windows devices, and the keyboard and mouse sharing service (including functional modules), collaboration service, and data transmission module in Android devices.

[0198] See Figure 10 To implement the cross-device drag-and-drop method for the provided window, the system message module, window management module, window settings module, protocol management module, collaboration service, and data transmission module in the Windows device need to cooperate with each other, according to... Figure 10 The interactive processing is performed in the order shown, from steps 101 to 139. The data transmission module, protocol management module, window settings module, virtual data injection module, and collaboration service in the Android device need to follow... Figure 10The interactive processing is performed in the order shown in steps 201 to 227. For the sequential logic between operations processed by the Windows device and those processed by the Android device, please refer to [link to relevant documentation]. Figure 10 .

[0199] To facilitate the differentiation of functional modules in Windows devices and Android devices, in the following embodiments, the system message module, window management module, window settings module, protocol management module, collaboration service, and data transmission module in Windows devices are respectively represented as Windows system message module, Windows window management module, Windows window settings module, Windows protocol management module, Windows collaboration service, and Windows data transmission module; and the window management module, window settings module, protocol management module, collaboration service, data transmission module, and virtual data injection module in Android devices are respectively represented as Android window management module, Android window settings module, Android protocol management module, Android collaboration service, Android data transmission module, and Android virtual data injection module.

[0200] against Figure 10 The specific details of steps 101 to 139 implemented by the Windows device shown in the figure are as follows: Figure 11 A detailed explanation is needed. It should be noted that... Figure 11 Only a portion of the text is shown here; for more details, please refer to the corresponding text sections.

[0201] See Figure 11 To implement the cross-device drag-and-drop method for the provided window, steps 101 to 139, executed by the system message module, window management module, window settings module, protocol management module, collaboration service, and data transmission module in the Windows device, specifically include:

[0202] 101. The Windows Window Management Module sends a request to the Windows System Messaging Module to register a window movement message fence, so that the Windows Messaging System can monitor the movement of windows displayed on the Windows device.

[0203] 102. The Windows window management module sends a request to the Windows system message module in advance to register the mouse message fence, so that the Windows message system can monitor the mouse used in the keyboard and mouse sharing scenario.

[0204] Specifically, in this embodiment, the cross-device dragging of the window relies on the keyboard and mouse sharing function, that is, the Android device to which the window is dragged from the Windows device has established a keyboard and mouse sharing connection with the Windows device. Therefore, in a feasible implementation, the operations of steps 101 and 102 can be triggered after the keyboard and mouse sharing connection between the Windows device and the Android device is established.

[0205] For example, when a user drags a window displayed on a Windows device using the mouse in a keyboard and mouse sharing scenario (taking window 20a in the above embodiment as an example), the window movement message fence and mouse message fence registered in steps 101 and 102 above will be triggered. In this case, the Windows system message module will execute steps 103 and 104.

[0206] 103. The Windows system message module obtains the window movement message corresponding to window 20a and sends the window movement message to the Windows window management module.

[0207] 104. The Windows system message module obtains a message indicating that the mouse is pressed in a keyboard and mouse sharing scenario, determines that the left mouse button is currently pressed, and sends the left mouse button press message to the Windows device's window management module.

[0208] 105. After receiving both the window movement message and the left mouse button press message from the Windows system message module, the Windows window management module determines that window 20a is currently being dragged and moved.

[0209] It should be noted that in some possible implementations, the dragged window 20a may be displayed in maximized mode on the Windows device, meaning that window 20a fills the entire display area of ​​the Windows device. In this case, upon confirmation that window 20a has been dragged, the Windows device will respond to the drag operation by automatically restoring window 20a from maximized mode to a smaller window mode, and then moving it as the mouse drags.

[0210] Furthermore, it should be noted that in some other possible implementations, the dragged window 20a may currently be displayed in a small window mode, such as... Figure 2A The style shown is correct. In this case, when confirming that window 20a is dragged and moved, window 20a will maintain its current style and can be moved directly with the mouse.

[0211] Furthermore, it should be noted that since the Windows device and Android device are currently in a keyboard and mouse sharing scenario, when a user drags window 20a with the mouse, there may be instances where window 20a displayed on the Windows device needs to be dragged and moved to the Android device for display. Therefore, when the Windows window management module determines that window 20a is currently being dragged and moved, it will send the window parameters of window 20a to the Windows window settings module for further processing, i.e., execute step 106.

[0212] Furthermore, it should be noted that in one possible scenario, the user might use the arrow keys and keyboard shortcuts to move window 20a on the Windows device screen. In this case, since the user is not operating the mouse, only the window movement message fence registered in step 101 is triggered. Therefore, the Windows system message module only executes step 103, which sends the window movement message to the Windows window management module for processing. Since there is only a window movement message and no left mouse button press message, the Windows window management module determines that window 20a is not currently being dragged or moved. In this case, the Windows window management module will not execute step 106.

[0213] In another possible scenario, the user might have simply clicked on a control within window 20a by pressing the left mouse button, or clicked on other icons or documents on the desktop. In this case, since window 20a has not moved, the Windows system message module only executes step 104, which sends the left mouse button press message to the Windows window management module for processing. Because there is only a left mouse button press message and no corresponding window movement message for window 20a, the Windows window management module determines that window 20a has not been dragged or moved. In this case, the Windows window management module will not execute step 106.

[0214] In another possible scenario, the user might have pressed the right mouse button. In this case, since window 20a hasn't moved, the Windows system message module only executes step 104, which sends the right-click message to the Windows window management module for processing. Because there's only a right-click message and no left-click message, nor a corresponding window movement message for window 20a, the Windows window management module determines that window 20a hasn't been dragged or moved. In this case, the Windows window management module will not execute step 106.

[0215] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0216] 106. The Windows window management module sends the relevant window parameters of window 20a to the Windows window settings module.

[0217] Among these parameters, the relevant window parameters for window 20a include, for example, the handle of window 20a. Understandably, the window handle includes the coordinates of the top-left corner (vertex) of the window, as well as the window's width and height data. Therefore, by obtaining the handle of window 20a, the size information of window 20a (width data xWidth, height data yHeight) can be determined.

[0218] 107. The Windows window settings module calculates the window data (such as width data xWidth and height data yHeight) of window 20a based on the window parameters of window 20a sent by the Windows window management module.

[0219] As described in the above embodiments, in the cross-device dragging method for windows provided in this application, the crossing of window 20a occurs when the mouse crosses over. Therefore, after the Windows system message module monitors the window movement message and the left mouse button press message of window 20a, if the user continues to press the left mouse button to drag window 20a, such as moving window 20a towards the side edge of the screen, such as the right edge in the above embodiments, as the cursor displayed on the Windows device moves, such as the cursor 10a in the above embodiments, the Windows system message module will continuously obtain the mouse movement position information and send the mouse movement position information to the Windows window management module, that is, execute step 108.

[0220] It should be noted that in this embodiment and the following embodiments, the mouse movement position information refers to the cursor corresponding to the mouse, such as the movement position information of cursor 10a in the above embodiment.

[0221] 108. The Windows System Messages module sends mouse movement position messages to the Windows Window Management module.

[0222] 109. The Windows window management module performs edge detection based on the mouse movement position information to determine whether the mouse has been dragged to the side edge of the Windows device screen (such as the right edge in the above embodiment).

[0223] Understandably, the screen size of a Windows device is known. Therefore, given that the screen size and the current mouse movement position (coordinates) of cursor 10a are known, it is possible to determine whether the mouse has been dragged to the side edge of the Windows device screen.

[0224] Specifically, in a keyboard and mouse sharing scenario, the Windows window management module triggers a keyboard and mouse crossing when it determines that the mouse has been dragged to the edge of the screen (left or right edge). Simultaneously, in the cross-device drag-and-drop window solution provided in this application, the Windows window management module notifies the Windows window settings module that a drag crossing has occurred, i.e., executes step 110. This enables the Windows window settings module to generate a window crossing instruction (command) based on the window data, i.e., executes step 111.

[0225] 110. When the Windows window management module determines that the mouse has reached the edge of the screen based on the mouse movement position information, it notifies the Windows window settings module that a drag-and-drop has occurred.

[0226] 111. The Windows window settings module generates a window crossing instruction corresponding to window 20a based on the window data, and sends the window crossing instruction of window 20a to the Windows protocol management module.

[0227] 112. The Windows protocol management module packages the data of the window crossing instruction according to the predetermined protocol.

[0228] 113. The Windows protocol management module sends the obtained window crossing instruction data packet to the Android data transmission module via the Windows data transmission module.

[0229] Understandably, both the Windows and Android protocol management modules are primarily responsible for assembling and unpacking related instructions (commands), window parameters, confirmation messages, etc., generated during window dragging. In other words, all data to be sent to the other end needs to be packaged by the current device's protocol management module before being handed over to the other end's data transmission module.

[0230] Correspondingly, data packets from the peer also need to be unpacked by the protocol management module of the current device before they can be sent to the corresponding functional modules for further processing.

[0231] The protocols followed by the Windows protocol management module in assembling and unpacking packets, and by the Android protocol management module in assembling and unpacking packets, are pre-agreed upon by the Android and Windows devices. This application does not limit the specific methods of agreement or the predetermined protocols.

[0232] See Figure 11 It is known that subsequent processing of window 20a dragging in Windows devices can only continue after the Windows data transmission module receives the window crossing confirmation data packet from the Android data transmission module. The Android data transmission module sending the window crossing confirmation data packet to the Windows data transmission module requires... Figure 10 The processing steps 201 to 207 are shown in the diagram. For specific implementation details of steps 201 to 207, please refer to [link to documentation]. Figure 15 ,and Figure 15 The corresponding implementation details will not be elaborated here.

[0233] 114. After receiving the window traversal confirmation data packet from the Android data transmission module, the Windows data transmission module sends the window traversal confirmation data packet to the Windows protocol management module for unpacking.

[0234] 115. The Windows protocol management module unpacks the window crossing confirmation data packet according to the agreed protocol, and then obtains the window crossing confirmation message.

[0235] 116. The Windows Protocol Management module sends the unpacked window traversal confirmation message to the Windows window settings.

[0236] 117. After receiving the window crossing confirmation message for window 20a from the Android device, the Windows window settings module hides window 20a displayed on the Windows device screen.

[0237] Specifically, window 20a is hidden at a specified coordinate, so that when window 20a is dragged from the Android device back to the Windows device, window 20a will appear from that hidden coordinate.

[0238] Alternatively, after the collaborative connection between the Windows device and the Android device is broken, window 20a can automatically appear from the hidden coordinates. Or, after the collaborative connection between the Windows device and the Android device is broken, when the user clicks the icon of document 2 displayed on the Windows device, window 20a appears from the hidden coordinates in response to that action.

[0239] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0240] Furthermore, it should be noted that the process of displaying window 20a on the PC can be divided into compositing rendering, caching, and display. The compositing rendering of the image corresponding to window 20a (hereinafter referred to as window 20a) is, for example, in... Figure 8A This is accomplished in the window management and image rendering module shown. After obtaining the composited and rendered window 20a, it is cached in the cache corresponding to window 20a, and then retrieved from the cache by the display driver for display.

[0241] Specifically, the canvas for composite rendering is typically larger than the display area of ​​a Windows device screen. For example... Figure 12 As shown, the composite drawing canvas can have a visible area (the display interface of the Windows device screen) and an invisible area.

[0242] In scenarios where window 20a needs to be displayed on a Windows device screen, after the display driver retrieves window 20a from the cache, it specifically sends window 20a to the display. Figure 12 The visible area of ​​the canvas, i.e. the display interface of the Windows device screen, is used for display.

[0243] In scenarios where window 20a does not need to be displayed on the Windows device screen, such as the scenario where window 20a is hidden as described in this embodiment, window 20a is still actually being composited and drawn. The only difference is that after the display driver retrieves window 20a from the cache, it sends it to the display. Figure 12 The invisible area of ​​the canvas.

[0244] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0245] Furthermore, in some possible implementations, to provide a better user experience, the hidden coordinates are determined, and then the proportional relationship between the y-axis coordinate of the hidden coordinates and the height (yHeight) of the Windows device is determined, thereby allowing the image to be viewed from the side edge of the Android device, such as... Figure 3C The left edge, as shown, extends out of the floating window 40a at a position corresponding to the proportion.

[0246] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the only limitation of this embodiment.

[0247] To better understand how the hidden coordinates of window 20a are determined, the following is combined with... Figure 13 , Figure 14A and Figure 14B will be described.

[0248] Refer to Figure 13 , the determination of the hidden coordinates is mainly implemented by the Windows window setting module, and can be specifically implemented through steps S1 to S5.

[0249] S1, the Windows window setting module determines the width data xWidth and height data yHeight of window 20a according to the window parameters.

[0250] Specifically, the Windows window setting module determines according to the handle of window 20a provided by the Windows window management module. Regarding the implementation details of determining the width and height of the window according to the handle of the window, existing literature can be referred to and will not be described here.

[0251] S2, the Windows window setting module determines the current coordinates (x0, y0) of cursor 10a according to the mouse movement position information.

[0252] S3, the Windows window determines whether x0 ≤ x_min / x0 ≥ x_max.

[0253] Exemplarily, when the top left vertex of the Windows device screen is the origin coordinate (0, 0), x_min is 0. x_max is the coordinate corresponding to the X axis in the coordinates of the bottom right vertex of the screen. For example, for a 1920 * 1080 screen, x_max is 1920.

[0254] Based on this, if x0 ≤ x_min, it indicates that the mouse crosses from the left edge of the Windows device screen; if x0 ≥ x_max, it indicates that the mouse crosses from the right edge of the Windows device screen; if x_min < x0 < x_max, it indicates that the mouse is still in the screen of the Windows device, has not reached the left edge, and has not reached the right edge, that is, no crossing will occur.

[0255] When it is determined that no crossing occurs, the Windows window setting module can wait for the window parameters and mouse movement position information obtained at the next moment, and then re - execute step S3 according to the newly obtained information.

[0256] S4, the Windows window determines that the dragged window 20a crosses from the left edge / right edge of the screen.

[0257] S5, the Windows window determines the hidden coordinates (x, y) of window 20a in the Windows device according to the crossing position and the current coordinates of cursor 10a.

[0258] Understandably, in this embodiment, the keyboard, mouse, and window traverses from the left or right edge of the Windows device screen. Therefore, the Y-axis coordinate of the hidden coordinates can be directly the current Y-axis coordinate of cursor 10a, such as y0 determined in step S2. The X-axis coordinate needs to be determined based on the traversal position and the width data xWidth.

[0259] See Figure 14A For example, in a scenario where the screen resolution of a Windows device is 1920*1080 and the screen resolution of window 20a is 200*100, when the user drags window 20a to the right edge of the Windows device screen with the mouse, the Windows window settings module will... Figure 13 The processing flow determines that when x0 ≥ 1920, it can be determined that window 20a and the mouse will cross the right edge of the Windows device screen. For Figure 14A In the scene shown, the hidden coordinates along the X-axis are x = Xright = x_max = 1920, and the hidden coordinates along the Y-axis are y = Yright = y0. That is, the hidden coordinates (x, y) are (1920, y0).

[0260] See Figure 14B For example, in a scenario where the screen resolution of a Windows device is 1920*1080 and the screen resolution of window 20a is 200*100, when the user drags window 20a to the left edge of the Windows device screen with the mouse, the Windows window settings module will... Figure 13 The processing flow determines that when x0≤0, it can be determined that window 20a and the mouse will cross the left edge of the Windows device screen. For Figure 14B In the scenario shown, this embodiment hides the window 20a only when it is completely moved off the Windows device screen. Therefore, the coordinates for hiding the window along the X-axis are x = Xleft = -xWidth, and the coordinates along the Y-axis are y = Yleft = y0. That is, the hiding coordinates (x, y) are (-xWidth, y0).

[0261] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0262] As described in the above embodiments, when window 20a traverses, the first floating window 40a that is activated does not display completely in the center of the Android device screen, but rather on the edge of the screen closer to the Windows device, such as the aforementioned one-third-sized floating window 40a. Figure 3C(As shown). Therefore, even if a crossing occurs, the user can still press the left mouse button to continue moving until the user releases the mouse (i.e., releases the left mouse button).

[0263] For example, in some possible scenarios, the user might drag window 20a back to the Windows device before releasing the mouse. In other possible scenarios, for those crossing the right edge, it might continue to move to the right so that the floating window 40a can be fully displayed on the Android device screen.

[0264] Therefore, if the left mouse button is pressed and moved, the Windows system message module will continue to obtain the mouse movement position information and send the mouse movement position information to the Windows window management module for processing, i.e., execute step 118.

[0265] 118. The Windows system message module obtains the mouse movement position information and sends it to the Windows window management module.

[0266] 119. The Windows window management module determines the window movement position information based on the mouse movement position information and sends the window movement position information to the Windows window settings module.

[0267] Understandably, during window dragging, the mouse cursor rests on the window, causing it to move. Therefore, given the mouse movement position information, the Windows window management module can directly determine the window movement position information corresponding to window 20a.

[0268] 120. The Windows window settings module generates a window movement instruction based on the window movement position information sent by the Windows window management module, and sends the window movement instruction to the Windows protocol management module.

[0269] 121. The Windows protocol management module assembles the window movement indication data into packets according to the agreed protocol, thereby obtaining the window movement indication data packet.

[0270] 122. The Windows protocol management module sends the obtained window movement instruction data packet to the Android data transmission module via the Windows data transmission module.

[0271] In this way, the Android window device module can change the position of the floating window 40a based on the window's movement position information.

[0272] Since users can release the mouse at any time after the floating window 40a appears on the Android device screen, the operations in steps 123 to 126 can also occur during the execution of steps 118 to 122.

[0273] See also Figure 11 For example, when the user releases the mouse, the Windows system message module will monitor the left mouse button release message and the window stop moving message, and send the left mouse button release message and the window stop moving message to the Windows window management module for further processing, namely, executing steps 123 and 124.

[0274] 123, The Windows system message module sends a left mouse button lift message to the Windows window management module.

[0275] 124. The Windows System Messages module sends a "stop moving window" message to the Windows Window Management module.

[0276] 125. After receiving the left mouse button release message and window stop moving message sent by the Windows system message module, the Windows window management module will generate a release window instruction and send the release window instruction to the Windows window device settings module.

[0277] 126. After receiving the window release instruction sent by the Windows window management module, the Windows window device settings module sends the window release instruction to the Windows protocol management module for packet processing.

[0278] 127. The Windows protocol management module assembles the window release instruction data into packets according to the agreed protocol, thereby obtaining the window release instruction data packet.

[0279] 128. The Windows Protocol Management Module sends the received window release instruction data packet to the Windows Data Transmission Module, which then sends it outwards.

[0280] 129. The Windows data transmission module sends the window release instruction data packet provided by the Windows protocol management module to the Android data transmission module so that other functional modules in the Android device can perform subsequent processing based on the window release instruction data packet.

[0281] As described in the above embodiments, the cross-device dragging of the window provided in this application still requires the assistance of the collaboration function to drag the window 20a displayed on the Windows device screen to the Android device screen. Furthermore, as described in the above embodiments, when the user releases the mouse, i.e., when the loading icon 40a-5 appears on the floating window 40a, the Windows device and the Android device will automatically enable the collaboration function without the user's awareness or the need for the user to enable it, and establish a collaboration connection between them in the extended mode supported by the collaboration function (hereinafter referred to as extended connection). Therefore, the Windows window management module will execute the operation of enabling the collaboration function, i.e., execute step 130, while generating the window release instruction.

[0282] 130. When generating the window release instruction, the Windows Window Management module notifies the Windows Collaboration Service to enable the collaboration function.

[0283] 131. The Windows Collaboration Service sends an Extended Mode Instruction to the Windows Protocol Management Module so that the Windows Collaboration Management Module can package the Extended Mode Instruction.

[0284] 132. The Windows protocol management module assembles the data indicated by the extended mode according to the agreed protocol, and then obtains the extended mode indication data packet.

[0285] 133. The Windows Protocol Management Module sends the obtained Extended Mode Indication Data Packet to the Windows Data Transmission Module, which then sends it outwards.

[0286] 134. The Windows data transmission module sends the extended mode instruction data packet provided by the Windows protocol management module to the Android data transmission module so that other functional modules in the Android device can perform subsequent processing according to the extended mode instruction data packet, thereby completing the extended connection between the two.

[0287] In some embodiments, such as Figure 10 As shown, from receiving the extended mode indication data packet to sending the extended connection success data packet to the PC, the Android device needs at least [time period missing]. Figure 10 Steps 222 to 227, as shown in the diagram, are necessary to establish an extended connection between the Windows device and the Android device. For detailed implementation information on steps 222 to 227, please refer to [link to documentation / reference]. Figure 15 ,and Figure 15 The corresponding implementation details will not be elaborated here.

[0288] 135. After receiving the extended connection success data packet from the Android data transmission module, the Windows data transmission module sends the extended connection success data packet to the Windows protocol management module for unpacking.

[0289] 136. The Windows protocol management module unpacks the extended connection success data packet according to the agreed protocol and obtains the extended connection success message returned by the Android device.

[0290] 137. The Windows Protocol Management module sends an extended connection success message to the Windows Window Settings module.

[0291] 138. After confirming the successful extended connection between the Windows device and the Android device, the Windows window settings module will generate a collaborative screen that needs to be displayed in the display area of ​​the floating window 50a and send it to the Android device for display.

[0292] The generation of the collaborative screen is similar to that of window 20a, except that the compositing and rendering process is based on the maximized mode of window 20a and taskbar 30a when generating the collaborative screen. Therefore, the generated collaborative screen ultimately includes the aforementioned window 20b and taskbar 30b.

[0293] For example, the composite rendering of collaborative scenes is still in Figure 8A This is completed in the window management and image rendering modules shown. After obtaining the composited and rendered collaborative screen, it is cached in the corresponding cache of the collaborative screen. Then, the Windows protocol management module assembles the packets according to the agreed protocol, and the Windows data transmission module sends the assembled collaborative screen packets to the Android data transmission module.

[0294] Accordingly, after the Android data transmission module receives the packet of the collaborative screen, it will forward it to the Android protocol management module for unpacking, thereby obtaining the collaborative screen, and finally displaying the obtained collaborative screen in the display area of ​​the floating window 50a.

[0295] For details regarding the specific content included in the collaborative screen, please refer to the above. Figure 3E The description of the corresponding embodiments will not be repeated here.

[0296] After completing step 138, the Windows settings module will notify the Windows window management module that the dragging of window 20a is complete, and then proceed to step 139.

[0297] This completes the process of dragging window 20a from the Windows device screen to the Android device screen. The corresponding screen changes are shown in the image below. Figure 2A , Figure 2B , Figures 3A to 3D .

[0298] This concludes the introduction to the operations performed by various functional modules in the Windows device during the cross-device dragging process. For any parts not described in this embodiment, you can refer to the operations performed by various functional modules in the Android device for understanding, and they will not be elaborated further here.

[0299] against Figure 10 The specific details of steps 201 to 227 implemented by the Android device, as shown in the figure, are as follows: Figure 15 A detailed explanation is needed. It should be noted that... Figure 15 Only a portion of the text is shown here; for more details, please refer to the corresponding text sections.

[0300] See Figure 15 To implement the cross-device drag-and-drop method for the window provided in the application, steps 201 to 227 of the data transmission module, protocol management module, window settings module, virtual data injection module, window settings module, and collaborative service execution in the Android device specifically include:

[0301] 201. After receiving the window crossing instruction data packet sent by the Windows data transmission module, the Android data transmission module sends the window crossing instruction data packet to the Android protocol management module for unpacking.

[0302] 202. The Android protocol management module unpacks the window crossing instruction data packet according to the agreement, and then obtains the window crossing instruction.

[0303] 203. The Android protocol management module sends the obtained window crossing instruction to the Android window management module for processing.

[0304] 204. The Android window management module starts the floating window 40a according to the window crossing instruction.

[0305] In this way, before the extended connection between the Windows device and the Android device is successfully established, the floating window 40a can be used instead of the floating window 50a, so that the user visually feels that the window 20a on the Windows device has indeed been dragged to the Android device, and is just waiting to load the content of the window 20a.

[0306] 205. The Android window management module generates a window crossing confirmation message and sends the window crossing confirmation message to the Android protocol management module for packet assembly.

[0307] 206. The Android protocol management module assembles the window crossing confirmation data into packets according to the agreed protocol, thereby obtaining the window crossing data confirmation data packet.

[0308] 207. The Android protocol management module sends the obtained window traversal data confirmation data packet to the Windows data transmission module via the Android data transmission module.

[0309] In this way, after receiving the window traversal data confirmation packet sent by the Android device, the Windows device can execute... Figure 11 The steps 114 and subsequent steps are shown in the diagram.

[0310] See also Figure 15 For example, after launching the floating window 40a, the Android window management module will continue to execute steps 208 to 211. That is, steps 208 to 211 can be performed synchronously with steps 205 to 208, without any order.

[0311] 208. After starting the floating window 40a, the Android window management module determines the display position of the floating window 40a on the Android device screen and sends the determined display position information to the Android window settings module to instruct the Android window settings module to set the display position of the floating window 40a.

[0312] 209. The Android window settings module sets the display position of the floating window 40a according to the display position information provided by the Android window settings module, so that the floating window 40a can be displayed at the specified position on the Android screen.

[0313] It should be understood that if the mouse does not have a corresponding down event, when the mouse moves to the Android device and the user releases the mouse, an up event will occur, which the Windows device will not be aware of. Before the mouse moves to the Android device, the user's left-click operation (down event) is registered on the Windows device. When the mouse moves from the Windows device to the Android device, there is no corresponding down event on the Android device. Therefore, to ensure that the Windows device is aware when the mouse is released after moving to the Android device, the Android window settings module, after setting the display position of the floating window 40a, sends the cursor position information of the current cursor 10b on the Android device to the Android virtual data injection module. This allows the Android virtual data injection module to inject a down event for the mouse on the Android device. That is, steps 210 and 211 are executed, so the Windows device will be aware that the mouse has been released.

[0314] 210. The Android window settings module informs the Android virtual data injection module of the mouse position on the Android device, that is, the position information of the cursor 10b on the Android device.

[0315] 211. The Android virtual data injection module injects a down event at the mouse position.

[0316] pass Figures 3A to 3E As described in the illustrated embodiment, when the mouse crosses over window 20a and the floating window 40a is first displayed on the Android device screen, it is not displayed completely on the Android device screen. Instead, approximately one-third of the floating window 40a is displayed near the side edge of the screen on the Windows device, such as the left edge in the above embodiment. Therefore, in order to allow the floating window 40a to still be dragged after the mouse crosses over to the Android device, a down event corresponding to the mouse cursor 10b needs to be injected into the status bar 1 of the floating window 40a, and a drag listener needs to be added to the status bar 1. For details on the implementation of this operation, please refer to [link to relevant documentation]. Figure 16A The operations performed by the window settings module and HwFreeFormCaptionView in the illustrated embodiment will not be described in detail here.

[0317] Furthermore, as described above regarding the operations performed by the functional modules in the Windows device, after receiving the window traversal confirmation data packet from the Android device, the Windows device generates a window movement indicator data packet as the user continues to press the left mouse button and move the mouse, and sends this window movement indicator data packet to the Android device. Upon receiving this window movement indicator data packet, the Android device executes steps 212 to 216.

[0318] 212. After receiving the window movement indication data packet sent by the Windows data transmission module, the Android data transmission module sends the window movement indication data packet to the Android protocol management module for unpacking.

[0319] 213. According to the agreement, the Android protocol management module unpacks the window movement instruction data packet to obtain the window movement instruction.

[0320] 214. The Android protocol management module sends the received window movement instruction to the Android window management module for processing.

[0321] 215. The Android window management module extracts the window movement position information from the window movement indicator and sends the extracted window movement position information to the Android window settings module.

[0322] 216. The Android window settings module moves the position of the floating window 40a displayed on the Android screen according to the window movement position information sent by the Android window management module.

[0323] For example, based on the window's movement position information, the floating window 40a can be moved from... Figure 3C Move to the position shown in the image. Figure 3D The location shown.

[0324] Understandably, before the user releases the mouse, as the mouse is pressed and moved, the Windows device will continuously send the corresponding window movement indication data packets to the Android device after each movement, and the Android device will continuously move the display position of the floating window 40a according to the window movement position information obtained by unpacking.

[0325] As described above, after the Android virtual data injection module injects a down event into the mouse, the Windows device can detect when the user releases the mouse. The aforementioned functional modules within the Windows device then process the data and ultimately send a release window indication data packet and an extended mode indication data packet to the Android data transmission module via the Windows data transmission module. See also... Figure 10 and Figure 15 It can be seen that when the Android device receives the release window instruction data packet, it will execute steps 217 to 221, and after receiving the extended mode instruction data packet, it will execute steps 222 to 227.

[0326] 217. After receiving the release window instruction data packet sent by the Windows data transmission module, the Android data transmission module sends the release window instruction data packet to the Android protocol management module for unpacking.

[0327] 218. According to the agreement, the Android protocol management module unpacks the release window instruction data packet to obtain the release window instruction.

[0328] 219. The Android protocol management module sends the received release window instruction to the Android window management module for processing.

[0329] 220. The Android window management module sends a window release instruction to the Android window settings module so that the Android window settings module can release the floating window 40a.

[0330] 221, Android window settings module releases floating window 40a.

[0331] At this point, control 40a-5 can be displayed in the floating window 40a.

[0332] It should be noted that since a mouse down event is injected in step 211, when the mouse is released and a release window instruction is received from the Windows device, the Android device can inject a mouse up event, thereby releasing the floating window 40a.

[0333] 222. After receiving the extended mode indication data packet sent by the Windows data transmission module, the Android data transmission module sends the extended mode indication data packet to the Android protocol management module for unpacking.

[0334] 223. According to the convention, the Android protocol management module unpacks the extended mode indication data packet to obtain the extended mode indication.

[0335] 224. The Android Protocol Management module sends the received extended mode instruction to the Android Collaboration Service for processing.

[0336] Because the time interval between the release window indicator and the extended mode indicator is usually small, the Android device is already performing operations such as unpacking and transmitting the extended mode indicator data packet during the release of the floating window 40a. Therefore, after the Android window settings module releases the floating window 40a, the Android Collaboration Service will quickly establish an extended connection with the Windows device and then generate an extended connection success message.

[0337] 225. The Android Collaboration Service establishes an extended connection with the Windows device, and after completing the extended connection, sends an extended connection success message to the Android Protocol Management Module for packet assembly.

[0338] 226. The Android protocol management module assembles the data from the successful extended connection according to the agreed protocol, and then obtains the extended connection success data packet.

[0339] 227. The Android protocol management module sends the obtained extended connection success data packet to the Android data transmission module, which in turn sends the extended connection success data packet to the Windows data transmission module.

[0340] This completes the process of dragging window 20a from the Windows device screen to the Android device screen. The corresponding screen changes are shown in the image below. Figure 2A , Figure 2B , Figures 3A to 3D Regarding from Figure 3D The floating window 40a shown is replaced with Figure 3E The details of the floating window 50a shown below are combined with... Figure 16A Please provide a detailed explanation.

[0341] As described in the above embodiments, the floating window 50a displayed on the Android device screen can be zoomed in and out by dragging, and its size also changes when the screen is rotated. The implementation of these functions relies on certain functional modules (methods / functions) of the FWK layer in the Android system, such as ActivityStarter, DisplayContent, MultiWindowsManager, FreeFormCaptionView, and MouseManager.

[0342] The ActivityStarter is a functional module used to launch Activity components, such as floating windows 40a and 50a. Unlike the native Android ActivityStarter, the ActivityStarter in this application can also set the collaboration mode corresponding to floating window 50a. For distinction, the ActivityStarter implemented in this application will be referred to as HwActivityStarterEx. It should be understood that this is only for ease of explanation, and the name of the functional module implementing this functionality is not limited.

[0343] In this application, the DisplayContent, or display content manager, is responsible for preventing the floating window 40a from scaling. This ensures that once the floating window 50a is activated, it can completely cover the floating window 40a without the user notifying it, and that the floating window 50a will not be unable to fully cover the floating window 40a if the floating window 40a is enlarged.

[0344] MultiWindowsManager is a multi-window manager. Unlike the native Android MultiWindowsManager, the MultiWindowsManager in this application ensures the floating window 50a remains functional in landscape mode even when the Android device's screen rotates, maintaining the same effect in portrait mode. For clarity, the MultiWindowsManager implemented in this application will be referred to as HwMultiWindowsManager. It should be understood that this is merely for illustrative purposes, and the name of the functional module implementing this feature is not limited.

[0345] In this context, FreeFormCaptionView can be understood as the taskbar in free window mode, specifically the functional module responsible for managing status bars 1 and 2 within floating windows 40a and 50a. Since the user can drag floating windows 40a and 50a by manipulating the mouse when the cursor 10b is placed in the status bar 1 area in the cross-device dragging scheme provided in this application, unlike the native Android FreeFormCaptionView, the FreeFormCaptionView in this application can be used to set drag listeners for this taskbar. For distinction, the FreeFormCaptionView implemented in this application will be referred to as HwFreeFormCaptionView. It should be understood that this is merely for illustrative purposes, and the name of the functional module implementing this function is not limited.

[0346] MouseManager, or mouse management, is used to manage mouse events such as down, up, and drag events. In this application, it is mainly used to manage mouse drag listening on status bar 1, and to cancel the drag listening after releasing the mouse and triggering an up event. Since the Android device uses the keyboard and mouse of the Windows device in the keyboard and mouse sharing mode, it also manages the keyboard and mouse of the Windows device. For distinction, the MouseManager implemented in this application will be referred to as HnPcMouseManager. It should be understood that this is only for ease of explanation, and the name of the functional module implementing this function is not limited.

[0347] For ease of description, Figure 16A For Android devices, only the Android window management module and Android window settings module in the collaboration service and keyboard and mouse service are shown, as well as the HwActivityStarterEx, DisplayContent, HwMultiWindowsManager, HwFreeFormCaptionView, and HnPcMouseManager listed above in the FWK layer. Specific functional modules are not shown for Windows devices.

[0348] In this context, the window management module in an Android device can be represented as DragWindowManager, and the window settings module can be represented as PcScreenCoverActivity. It should be understood that this is for illustrative purposes only, and the names of the functional modules implementing this functionality are not limited.

[0349] See Figure 16AFor example, in this embodiment, the extended connection between the Windows device and the Android device is achieved by triggering the collaborative function based on the keyboard and mouse sharing connection between the Windows device and the Android device. To ensure that the display area of ​​the floating window 50a can display the screen content of the window 20a recorded and projected by the Windows device without black borders after the extended connection between the Windows device and the Android device is established, the aspect ratio of the area displaying window 20a in the floating window 50a is agreed upon when the keyboard and mouse sharing connection is established. Thus, the Windows device can subsequently record and project window 20a according to the agreed aspect ratio, and the Android device can also scale the display proportionally according to the same aspect ratio, thereby allowing window 20a to be displayed without black borders in the display area of ​​the floating window 50a. The interface effect is more like window 20a being directly dragged and dropped onto the Android device for display.

[0350] Regarding the aspect ratio convention, for example, when a Windows device and an Android device establish a keyboard and mouse sharing connection based on a keyboard and mouse sharing service, the Windows device determines the aspect ratio based on its current screen width and height and sends it to the Android device. This way, the Android device knows the maximum width and height of the window 20a that needs to be displayed in the floating window should a window crossover occur subsequently.

[0351] See Figure 16B For example, in a Windows device with a 16:9 aspect ratio, the aspect ratio is also 16:9, as agreed upon with Android devices. However, for different extended screen styles on Android devices, this aspect ratio represents the proportion of different areas. For example, for the floating window 50a, the 16:9 aspect ratio indicates the aspect ratio of the remaining area after removing status bars 1 and 2, that is, the ratio of the width of the floating window 50a to (the height of the floating window 50a - the height of status bar 1 - the height of status bar 2) is 16:9.

[0352] See also Figure 16B For example, in a scenario where the extended screen is the entire screen of an Android device, the 16:9 aspect ratio indicator is the ratio of the area after removing the sidebars 50a-7 and the taskbar 30b.

[0353] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0354] See also Figure 16AFor example, after a keyboard and mouse sharing connection is established between a Windows device and an Android device, and the aspect ratio is agreed upon, if the user presses the left mouse button and drags the window, the mouse cursor will cross over the window, as described above. Figures 10-13 , Figure 14A , Figure 14B and Figure 15 The processing logic of the illustrated embodiment can determine the hidden coordinates of the window being dragged through. To ensure that the window appearing first on the Android device, such as the floating window 40a in the above embodiment, and the window 20a on the Windows device can exhibit a parallel crossing effect, the Y-axis height of the upper edge of the floating window 40a should be the same as the Y-axis coordinate of the hidden coordinates. Taking the Y-axis coordinate of the hidden coordinates as y0 as an example... Figure 14A and Figure 14B As shown.

[0355] See also Figure 16A For example, after y0b is obtained by the Android window management module, the Android window management module will set the launch bounds of the floating window 40a according to y0 and the agreed aspect ratio, and set the window mode of the floating window 40a to the floating window mode.

[0356] It should be understood that floating window 40a is a virtual window that replaces floating window 50a on the Android device screen before floating window 50a is displayed. As described above, it is not a full-screen extension of floating window 50a, but rather a floating window mode. Therefore, the window mode of floating window 40, which replaces floating window 50a, also needs to be set to floating window mode.

[0357] Furthermore, since the dimensions of the floating window 40a and the floating window 50a are generally the same, the area in the floating window 40a that satisfies the agreed aspect ratio is the same as that in the floating window 50a. The following embodiments will all use this as an example. Figure 16B The area shown in the image represents region 1.

[0358] Since the heights of status bars 1 and 2 are usually fixed, while their widths are the same as the width of area 1, the bounds of area 1 can be determined first when setting the bounds of the floating window 40a.

[0359] In addition, it should be noted that in some possible cases, the height of status bar 1 and the height of status bar 2 can be the same.

[0360] Furthermore, since floating window 40a is a virtual window that replaces floating window 50a, in this embodiment, we take the example that floating window 40a and floating window 50a have the same size. In this case, the heights of status bars 1 and 2 in both floating window 40a and floating window 50a shown in the accompanying drawings are the same.

[0361] For example, in some possible implementations, such as when it is agreed that region 1 occupies 60% (or other percentage) of the Android device screen, the width / height of the Android device screen can be obtained first, and then the width / height of the Android device screen can be multiplied by 60% to obtain the width / height corresponding to region 1. Then, based on the agreed aspect ratio and the determined width / height of region 1, the height / width corresponding to region 1 can be obtained.

[0362] For example, after determining the width of region 1, the X coordinate of the left edge of region 1 can be used as the X coordinate of the top left corner of the floating window 40a, and the X coordinate of the right edge of region 1 can be used as the X coordinate of the bottom right corner of the floating window 40a.

[0363] Furthermore, as described above, y0 is the Y-axis coordinate of the top edge (top) of the floating window 40a. Therefore, the Y-axis coordinate of the top left corner vertex of the floating window 40a can be determined as y0.

[0364] Furthermore, given that the heights of status bar 1 and status bar 2 are fixed, and the height of area 1 is determined, and given that top = y0, then the bottom edge of the floating window 40a is y0 + the height of status bar 1 + the height of area 1 + the height of status bar 2.

[0365] This allows you to set the bounds (left, top, right, bottom) of the floating window 40a.

[0366] For example, Figure 16A The Android window management module, following the above method, sets the bounds and window mode of the floating window 40a, which can be used as... Figure 15 One specific implementation of step 208 in the illustrated embodiment.

[0367] See also Figure 16A For example, after completing the above two steps (setting the bounds of floating window 40a and setting the window mode to floating window mode), the Android window management module will notify the Android window setting module to launch the Activity of floating window 40a, that is, create the Activity of floating window 40a.

[0368] Understandably, when the Android window management module notifies the Android window settings module to launch the Activity of floating window 40a, it will inform the Android window settings module of the parameters set in the two steps mentioned above. Thus, when the Android window settings module calls the StartActivity method to notify HwActivityStarterEx to draw the Activity of floating window 40a, it will inform HwActivityStarterEx of the bounds of floating window 40a and the floating window mode as input parameters.

[0369] See also Figure 16A For example, HwActivityStarterEx will draw the Activity of the floating window 40a according to the parameters passed by the Android window settings module, and notify HwFreeFormCaptionView to set a drag listener for the Caption in the floating window 40a, specifically the status bar 1 mentioned above. In this way, when the cursor 10b falls on the status bar 1, the user can drag the floating window 40a by controlling the mouse.

[0370] See also Figure 16A For example, after HwFreeFormCaptionView sets up drag-and-drop listeners for status bar 1, it will notify HnPcMouseManager. In this way, HnPcMouseManager can manage the drag-and-drop listeners for status bar 1, such as listening for whether an event occurs where status bar 1 is dragged.

[0371] In addition, it should be noted that HnPcMouseManager will also cancel the drag listening for status bar 1 after the mouse is released and the up event occurs.

[0372] See also Figure 16A For example, when the Activity for the floating window 40a is drawn in HwActivityStarterEx, i.e. when the floating window 40a is displayed on the Android device screen, DisplayContent will also set the floating window 40a, specifically by disabling scaling of the floating window 40a.

[0373] See also Figure 16A For example, the Android window settings module will also avoid the hotspot of the status bar 1 of the floating window 40a when determining the floating window 40a, for example... Figure 3D The area containing controls 40a-1 to 40a-4 shown in the figure is used to set the initial position of the mouse, that is, the position of cursor 10b after the mouse passes through it.

[0374] Understandably, since the mouse cursor just passed through and the floating window 40a just appeared, that is... Figure 3C As shown, even if the user doesn't release the mouse, the floating window 40a can still be dragged to the right, for example, to... Figure 3D The mouse is released only at the indicated position. Therefore, the initial position set by the Android window management module for cursor 10b must fall within the area of ​​status bar 1 and avoid the various functional controls displayed on status bar 1.

[0375] Furthermore, it should be noted that in some possible implementation scenarios, the area along the side edge of the Android device screen is often designated as a hotspot, meaning that mouse operations along the side edge, such as swiping, will bring up other windows or the taskbar. Therefore, the initial position set by the Android window management module for cursor 10b needs to avoid the hotspot of the Android device screen's side edge. Since this case involves displaying the floating window 40a across the left edge of the Android device screen, the initial position set by the Android window management module for cursor 10b needs to avoid the hotspot of the left edge of the Android device screen.

[0376] For example, given that the top of the floating window 40a is known and the height of the status bar 1 is known, in one possible implementation, the X-axis coordinate of the initial mouse position can be between the left and right coordinates determined above, and avoid the coordinates of controls 40a-1 to 40a-4, denoted as x1, and the corresponding y1 can be (top + height of status bar 1 / 2).

[0377] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0378] For example, Figure 16A The Android window device module sets the initial mouse position (i.e., displays cursor 10b in status bar 1) and injects the corresponding down event into the mouse. HwFreeFormCaptionView and HnPcMouseManager, following the same method, set and cancel drag listeners for status bar 1. This can be used as... Figure 15 One specific implementation of step 211 in the illustrated embodiment.

[0379] See also Figure 16A For example, after setting the initial mouse position, the Android window settings module injects a down event into the mouse. See [link to documentation] for details. Figure 15 Steps 209 to 211 are not described here.

[0380] See also Figure 16A For example, after completing the drawing and setting of the floating window 40a, the Android window settings module will call onResume() to inform the Android window management module that the floating window 40a has been successfully started and is available.

[0381] See also Figure 16A For example, if the Android window management module receives mouse movement information from the Windows device after completing the above operations, it can then cooperate with other functional modules to... Figure 15 The operations shown in steps 212 to 216 enable the movement of the floating window 40a; the specific implementation details will not be elaborated here.

[0382] See also Figure 16A For example, if the user releases the mouse (i.e., the Android device detects a mouse up event), HnPcMouseManager will cancel its drag-and-drop listener for status bar 1. Meanwhile, based on... Figure 10 and Figure 11 As shown, after the user releases the mouse, the Windows device sends a release window instruction to the Android device, and at the same time executes the extended mode to start the collaboration function and establish an extended connection with the Android device.

[0383] See also Figure 16A For example, during the process of establishing an extended connection between an Android device and a Windows device, HwActivityStarterEx will first set the bounds of the floating window 50a when drawing the Activity of the floating window 50a.

[0384] As described above, floating window 50a needs to cover floating window 40a. Therefore, when setting the bounds of floating window 50a, simply find the bounds of the currently displayed floating window 40a, and then set the bounds of floating window 50a according to the bounds of floating window 40a.

[0385] For example, in one possible scenario, the bounds of floating window 50a can be the same as the bounds of floating window 40a. This allows for seamless overlay of floating window 40a, so the user will not perceive that floating window 40a has been replaced.

[0386] In addition, HwActivityStarterEx will also need to set up window modules for floating window 50a when drawing the Activity of floating window 50a.

[0387] Specifically, if floating window 40a exists in the current Android device interface, then the window mode of floating window 50as will also be set to floating window mode. If floating window 40a does not exist, then the collaboration mode needs to be changed to mirror mode, that is, the window mode of floating window 50a needs to be set to full-screen mode.

[0388] See also Figure 16A For example, after completing the above settings and drawing the Activity of floating window 50a according to the above requirements, that is, after starting floating window 50a, HwActivityStarterEx can call finish() to close floating window 40a.

[0389] For information on releasing the mouse and establishing extended connections, please refer to [link / reference needed]. Figure 11 Steps 125 to 137 shown, and Figure 15 Steps 217 to 227 are described in detail here.

[0390] See also Figure 16A For example, after the floating window 50a is displayed, if the Android device screen rotates, HwMultiWindowsManager will perform operations to ensure that the floating window 50a is displayed correctly in landscape mode, regardless of whether the Android device screen is in portrait or landscape orientation. For information on how the position and size of the floating window 50a change when the Android device screen rotates, please refer to [link to relevant documentation]. Figure 7 As shown.

[0391] Understandably, if the Android device screen rotates while displaying the floating window 40a, HwMultiWindowsManager can ensure that the floating window 40a is displayed in landscape mode, regardless of whether the Android device screen is in landscape or portrait mode.

[0392] This concludes the introduction to the operations performed by various functional modules on the Android device during cross-device dragging. For any parts not described in this embodiment, please refer to the operations performed by various functional modules on the Windows device for further understanding.

[0393] Therefore, when a Windows device and an Android device establish a keyboard and mouse sharing connection (i.e., are in a keyboard and mouse sharing scenario but the collaboration function is not activated), and the user uses a Windows device, such as the PC mentioned in the above embodiment, to open multiple applications and needs to use these applications simultaneously, such as viewing the content in the windows corresponding to these applications, the cross-device dragging method of the window provided in this application can monitor the dragging of the windows corresponding to the applications / documents on the PC, and, in conjunction with mouse crossing control in keyboard and mouse sharing, calculate the position of the window moving off the PC screen when the mouse crosses over, and transmit the relevant information of the window to the Android device, such as the tablet mentioned in the above embodiment. When the window is released on the tablet (the user releases the left mouse button), the PC and the tablet are triggered to establish a collaborative connection, setting the tablet as an extended screen of the PC. Then, the hidden window on the PC, such as the document 2 window mentioned in the above embodiment, is moved to the extended screen, i.e., the tablet, for display, thereby realizing the experience of dual-screen office.

[0394] Based on the cross-device drag-and-drop method for windows provided in this application, during the window dragging process, the screen changes of Windows devices and Android devices can be as follows: Figure 17 As shown in (1) to (5).

[0395] Furthermore, it should be noted that in another possible implementation, the user might drag multiple windows displayed on the Windows device sequentially to the Android device for display. In this scenario, if the floating window 50a described in the above embodiment is not displayed on the Android device, when the user first drags a window from the Windows device, such as window 20a corresponding to document 2 in the above embodiment, the processing flow of the above embodiment can be followed. During the initial dragging of the window from the Windows device to the Android device, the screen changes on both the Windows device and the Android device can be as follows: Figure 17 As shown in (1) to (5).

[0396] For example, after window 20a is dragged and displayed in the floating window 50a on the Android device, if the user opens document 1 on the Windows device, the window corresponding to document 1 is as follows: Figure 18 The window 70a shown in (1) is shown in the middle.

[0397] See Figure 18 Zhong (1) and Figure 18In example (2), when the user drags window 70a within the screen of the Windows device using mouse 10a and gradually drags it to the right edge of the Windows device screen, cursor 10a will cross over. Since the collaborative connection between the Windows device and the Android device is not broken, the floating window 50a, which serves as an extension screen for the Windows device, remains displayed on the Android device screen. Therefore, when cursor 10a and window 70a cross over, there is no need to restart floating window 40a, nor is it necessary to re-establish the collaborative connection and floating window 50a. Thus, the portion of window 70a that has moved out of the Windows device (hereinafter referred to as the first portion) and cursor 10a will directly cross over into floating window 50a, while the portion of window 70a that has not moved out of the Windows device screen (hereinafter referred to as the second portion) will be displayed on the Windows device, as shown below. Figure 18 As shown in (3). For ease of distinction, in this embodiment, the window 70a that crosses into the floating window 50a is referred to as window 70b.

[0398] For example, when window 70a and cursor 10a directly cross over to floating window 50a, as cursor 10a drags window 70b, window 70b will continue to move with cursor 10a. In this way, the first part of window 70a displayed in floating window 50a will become increasingly larger, while the second part of window 70a displayed on the Windows device will become increasingly smaller, until window 70a is completely dragged into floating window 50a for display.

[0399] Furthermore, it should be noted that for windows dragged into floating window 50a after window 20a, the window style displayed in its restored state (small window) on the Windows device before the crossing will be maintained, and it will cover window 20b after the crossing. That is, windows that cross over after window 20a, such as window 70a, will partially obscure window 20b. Figure 18 As shown in (4).

[0400] Furthermore, it should be noted that regardless of whether the window 20b or 70b is dragged and dropped into the floating window 50a, the user can edit the content displayed in the window 50a using the keyboard and mouse in a shared keyboard and mouse environment. Moreover, any editing of the content in the floating window 50a will be synchronized to the corresponding document on the Windows device. For example, if the user adds text to window 20a and saves the edited window 20a, that text will be simultaneously added to document 2 on the Windows device.

[0401] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0402] Furthermore, it is understood that, in order to achieve the aforementioned functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0403] Furthermore, it should be noted that in practical application scenarios, the cross-device drag-and-drop window methods provided in the above embodiments, implemented by electronic devices, can also be executed by a chip system included in the electronic device. This chip system may include a processor. The chip system can be coupled to a memory, enabling it to call a computer program stored in the memory during runtime to implement the steps executed by the electronic device. The processor in the chip system can be an application processor or a non-application processor.

[0404] In addition, this application embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the above-described related method steps to implement the cross-device drag-and-drop method for windows in the above embodiment.

[0405] In addition, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the above-mentioned related steps to implement the cross-device drag-and-drop method for windows in the above embodiments.

[0406] In addition, embodiments of this application also provide a chip (which may also be a component or module), the chip may include one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the processing circuits communicate with each other through internal connection paths, the processing circuits execute the above-mentioned related method steps to implement the cross-device dragging method of the window in the above embodiments, so as to control the receiving pin to receive signals and control the sending pin to send signals.

[0407] Furthermore, embodiments of this application also provide a system comprising the aforementioned Windows device and an Android device that establishes a mouse sharing connection with the Windows device. The Windows device and the Android device can use the cross-device drag-and-drop method provided in the above embodiments to drag and drop windows from the Windows device to the Android device for display.

[0408] Furthermore, as can be seen from the above description, the electronic devices, computer-readable storage media, computer program products, or chips provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0409] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for dragging a window across devices, characterized in that: Applied to a system including a first electronic device and a second electronic device, the operating system of the first electronic device is a Windows system, and the operating system of the second electronic device is an Android system; the method includes: The first electronic device displays a first window; The first electronic device drags the first window to move within the screen of the first electronic device in response to a user operating a mouse cursor to drag the first window; When the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sends a window crossing instruction for the first window to the second electronic device, and hides the first window displayed on the first electronic device; wherein the mouse corresponding to the mouse cursor is connected to the first electronic device or the second electronic device; The second electronic device displays the first floating window on the screen of the second electronic device in response to the window crossing instruction, and the first window is not displayed in the first floating window; In the process of the mouse cursor dragging the first floating window to move within the screen of the second electronic device, the first electronic device sends a first instruction to the second electronic device in response to the user releasing the left button of the mouse; In response to the first instruction, the second electronic device releases the first floating window and displays a second floating window on the screen of the second electronic device, wherein the second floating window covers the first floating window; When the second electronic device receives the first collaborative screen sent by the first electronic device, the second electronic device displays the first collaborative screen in the display area of ​​the second floating window, where the first collaborative screen includes all display contents of the first window.

2. The method according to claim 1, characterized in that: After displaying the first collaborative screen in the display area of ​​the second floating window, the method further includes: The first electronic device displays a second window; The first electronic device drags the second window to move within the screen of the first electronic device in response to a user operating a mouse cursor to drag the second window; When the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sends a second collaborative screen for the second window to the second electronic device, the second collaborative screen includes a first part of the second window, and displays the second part of the second window on the screen of the first electronic device; wherein the first part is the part that moves out of the screen edge of the first electronic device, and the second part is the part that does not move out of the screen edge of the first electronic device; When the second electronic device receives the second collaborative screen sent by the first electronic device, the second collaborative screen is displayed in the display area of ​​the second floating window; wherein the window state of the second window in the second collaborative screen is a restored state, and the second collaborative screen covers the first collaborative screen; Wherein, when the window state is the restored state, the title bar of the first window includes a maximize control but does not include a restore control.

3. The method according to claim 1, characterized in that When the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sends a window crossing instruction for the first window to the second electronic device, and hides the first window displayed on the first electronic device, including: When the mouse cursor moves out of the screen edge of the first electronic device, the first electronic device sends a window crossing instruction for the first window to the second electronic device; After sending the window crossing instruction to the second electronic device, the first electronic device hides the first window displayed on the first electronic device when receiving a window crossing confirmation message made by the second electronic device in response to the window crossing instruction.

4. The method according to claim 3, characterized in that After the second electronic device displays the first floating window on the screen of the second electronic device in response to the window crossing instruction, the method further includes: The second electronic device generates the window crossing confirmation message for the window crossing indication, and sends the window crossing message to the first electronic device.

5. The method according to claim 1, characterized in that The first indication includes a release window indication and a screen coordination indication; The release window instruction is used to instruct the second electronic device to release the first floating window; The screen cooperation instruction is used to instruct the second electronic device to establish a screen cooperation connection with the first electronic device in which the cooperation mode is an extended mode; Wherein, the second floating window is displayed on the screen of the second electronic device after the screen cooperative connection is successfully established.

6. The method according to claim 1, characterized in that After the second electronic device releases the first floating window in response to the first instruction and displays a second floating window on the screen of the second electronic device, the method further includes: The second electronic device generates a screen coordination connection success message for the screen coordination indication, and sends the screen coordination connection success message to the first electronic device.

7. The method according to claim 6, characterized in that The first collaborative screen also includes a task bar of the first electronic device; The method further comprises: In response to the screen cooperative connection success message, the first electronic device sets the window state of the first window to a maximized state so that the first window displays all display contents included therein; The first electronic device synthesizes and draws the first collaborative screen based on the first window in the maximized state and the task bar of the first electronic device; and sends the obtained first collaborative screen to the second electronic device.

8. The method according to claim 7, characterized in that When the window state is the maximized state, the title bar of the first window includes a restore control but does not include a maximize control.

9. The method according to any one of claims 1 to 8, characterized in that: Hiding, by the first electronic device, the first window displayed on the first electronic device includes: The first electronic device determines the width and height of the first window according to the handle of the first window; The first electronic device determines the current X coordinate and Y coordinate of the mouse cursor corresponding to the mouse according to the movement information of the mouse; The first electronic device determines, according to the current X-coordinate and Y-coordinate of the mouse cursor, the width of the first window, and the X-coordinate interval corresponding to the screen of the first electronic device, the hidden coordinates of hiding the first window; wherein the X-coordinate interval includes the X-axis coordinate of the left edge of the screen of the first electronic device and the X-axis coordinate of the right edge of the screen of the first electronic device, and the X-axis coordinate of the left edge of the screen is smaller than the X-axis coordinate of the right edge of the screen; The first electronic device hides the first window at the location where the hiding coordinates are located.

10. The method according to claim 9, characterized in that The first electronic device determines, according to the current X coordinate and Y coordinate of the mouse cursor, the width of the first window, and the X coordinate interval corresponding to the screen of the first electronic device, the hiding coordinates for hiding the first window, including: When the X-coordinate of the mouse cursor is not greater than the X-axis coordinate of the left edge of the screen, the first electronic device determines a difference between the X-axis coordinate of the left edge of the screen and the width of the first window; The first electronic device uses the difference as the X coordinate of the hidden coordinate and uses the Y coordinate of the mouse cursor as the Y coordinate of the hidden coordinate.

11. The method according to claim 9, characterized in that The first electronic device determines, according to the current X coordinate and Y coordinate of the mouse cursor, the width of the first window, and the X coordinate interval corresponding to the screen of the first electronic device, the hiding coordinates for hiding the first window, including: When the X coordinate of the mouse cursor is not less than the X axis coordinate of the right edge of the screen, the first electronic device uses the X axis coordinate of the right edge of the screen as the X coordinate of the hidden coordinate and uses the Y coordinate of the mouse cursor as the Y coordinate of the hidden coordinate.

12. The method according to claim 9, characterized in that After the first electronic device hides the first window at the location where the hiding coordinates are located, the method further includes: In response to a drag operation of traversing the first window from the second electronic device back to the first electronic device, the first window is displayed from the hidden coordinates.

13. The method according to any one of claims 1 to 12, characterized in that: The first electronic device displays a first window, including: The first electronic device displays a first window in a maximized state; wherein, when the window state is the maximized state, a title bar of the first window includes a restore control but does not include a maximize control.

14. The method according to claim 13, characterized in that The first electronic device, in response to a user operating a mouse cursor to drag the first window, drags the first window to move within a screen of the first electronic device, including: The first electronic device switches the window state of the first window from the maximized state to the restored state in response to a user operating a mouse cursor to drag the first window; wherein, when the window state is the restored state, the title bar of the first window includes a maximize control but does not include a restore control; The first electronic device drags the first window whose window state is the restored state to move within the screen of the first electronic device.

15. The method according to any one of claims 1 to 12, characterized in that: The first electronic device displays a first window, including: The first electronic device displays a first window whose window state is a restored state; wherein, when the window state is the restored state, a title bar of the first window includes a maximize control but does not include a restore control.

16. The method according to claim 15, characterized in that The first electronic device, in response to a user operating a mouse cursor to drag the first window, drags the first window to move within a screen of the first electronic device, including: In response to the user operating the mouse cursor to drag the first window, the first electronic device drags the first window whose window state is the restored state to move within the screen of the first electronic device.

17. The method according to any one of claims 1 to 16, characterized in that: In the process of dragging the first window to move within the screen of the first electronic device, the method further includes: In the process of dragging the first window to move within the screen of the first electronic device, the first electronic device determines, based on the movement information of the mouse, whether a mouse cursor corresponding to the mouse moves out of the screen of the first electronic device from a left edge of the screen of the first electronic device or a right edge of the screen; When it is determined that the mouse cursor corresponding to the mouse moves out of the screen of the first electronic device from the left edge of the screen of the first electronic device or the right edge of the screen, the first electronic device determines that the mouse cursor moves out of the edge of the screen of the first electronic device.

18. The method according to any one of claims 1 to 17, characterized in that The first electronic device, in response to a user operating a mouse cursor to drag the first window, drags the first window to move within a screen of the first electronic device, including: When the first electronic device monitors the window move message of the first window and the left button of the mouse is pressed, the first electronic device determines that the user operates the mouse cursor to perform the drag operation on the first window; In response to the drag operation, the first electronic device drags the first window to move within the screen of the first electronic device.

19. The method according to any one of claims 1 to 18, characterized in that The first electronic device sends a first indication to the second electronic device in response to a user releasing the left button of the mouse, including: When the first electronic device monitors a message of releasing the left button of the mouse and a message of stopping the movement of the first window, it determines that the user has performed the release operation on the left button of the mouse; The first electronic device sends a first indication to the second electronic device in response to the release operation.

20. The method according to any one of claims 1 to 19, characterized in that The second electronic device includes a screen computer and a foldable screen mobile phone.

21. A window cross-device dragging system, characterized in that: include: A first electronic device and a second electronic device, a keyboard and mouse sharing connection is established between the first electronic device and the second electronic device, the operating system of the first electronic device is a Windows system, and the operating system of the second electronic device is an Android system; The first electronic device and the second electronic device are used to execute the cross-device dragging method of a window as described in any one of claims 1 to 20.

22. An electronic device, characterized in that: The electronic device comprises: a memory and a processor, wherein the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the cross-device dragging method of a window as described in any one of claims 1 to 20.

23. A computer-readable storage medium, characterized in that: It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the cross-device dragging method of a window as claimed in any one of claims 1 to 20.

Images