Cross-device picture dragging display method, device and system

Abstract

The invention provides a cross-device picture dragging display method, device and system, the method can be applied to a first electronic device installed with a first operating system, the first electronic device is connected with a second electronic device installed with a second operating system, and the first operating system is different from the second operating system. The method comprises the following steps: displaying a first window of a first application on a display screen of first electronic equipment, wherein the first application is an application of a global wide area network web type; and when the first picture is dragged to the first window from the display screen of the second electronic equipment, the first electronic equipment displays the first picture in the first window in response to a dragging release operation of a user on the first picture. According to the scheme, when a user drags the picture to the window of the web type application in a cross-device mode every time, the picture can be normally displayed, and the problem that the picture is not displayed or the picture dragged in a cross-device mode last time is displayed is solved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method, device and system for displaying cross-device image dragging. Background Art

[0002] In a multi-device collaboration scenario, a user can drag an object from a first electronic device to a second electronic device, or drag an object from a second electronic device to a first electronic device, where the object may include documents, pictures, application icons, videos, audio, etc. Figure 1 As shown, the user drags a picture in a mobile phone to an application window of a personal computer (PC).

[0003] Currently, based on Figure 1 The picture dragging scenario shown has the following problems:

[0004] First, when you drag a picture to the application window on the PC for the first time, the application window does not display the dragged picture. Figure 2 As shown, when the user drags Picture 1 from the mobile phone to the PC for the first time, the application window does not display Picture 1.

[0005] Second, when you drag the image to the PC application window for the second time, the application window displays the image you dragged for the first time, such as Figure 3 As shown, the user drags the picture 2 in the mobile phone to the application window on the PC for the second time, and the application window displays the picture 1 dragged for the first time. Summary of the invention

[0006] The embodiments of the present application provide a method, device and system for displaying a picture dragged across devices, so that each time a picture is dragged across devices to a window of a web-type application, the picture can be displayed normally.

[0007] In a first aspect, an embodiment of the present application provides a method for displaying a cross-device image drag, which is applied to a first electronic device installed with a first operating system, the first electronic device is connected to a second electronic device installed with a second operating system, the first operating system is different from the second operating system, the method comprising: displaying a first window of a first application and a second window of a second application on a display screen of the first electronic device, the first application being a global wide area network web-type application, and the second application being a non-web-type application; when a first image is dragged from the display screen of the second electronic device to the first window, in response to a user's drag-and-release operation on the first image, the first electronic device displays the first image in the first window; when a second image is dragged from the display screen of the second electronic device to the second window, in response to the user's drag-and-release operation on the second image, the first electronic device displays the second image in the second window.

[0008] For example, Figure 1 As shown, the first electronic device is a PC, the operating system of the PC may be a Windows system, the second electronic device is a mobile phone, the operating system of the mobile phone may be an Android system. The first application may be a web-type application such as a super note, a browser, etc., and the second application may be a non-web-type application such as a document word, a slide PPT, etc.

[0009] In the above scheme, similar to the effect of dragging and displaying pictures across devices of non-web type applications, when the user drags the picture across devices to the window of the web type application, the picture can be displayed normally, and there will be no problem of not displaying the picture or displaying the picture of the previous cross-device drag. Displaying the picture of the previous cross-device drag can also be described as displaying the picture in a dislocated manner. For example, after dragging the picture across devices for the second time, the window displays the picture of the first cross-device drag. For another example, after dragging the picture across devices for the third time, the window displays the picture of the second cross-device drag.

[0010] In an optional embodiment of the first aspect, before the first electronic device displays the first picture in the first window, the method also includes: the first application determines whether to block the drag event of the Google kernel of the first electronic device by querying a preset object; the first electronic device displays the first picture in the first window, including: when it is determined to block the drag event of the Google kernel, the first application obtains the first picture from the picture path indicated by the super keyboard and mouse service of the first electronic device, and does not obtain the picture from the Google kernel; the first application displays the first picture in the first window.

[0011] In this embodiment, the preset object may be a file (eg, a first file described below) or a mark bit (eg, a first mark bit described below).

[0012] It should be noted that shielding the drag event of the Google kernel of the first electronic device refers to shielding the drag event of the Google kernel once, that is, the first application will not obtain the image from the Google kernel of the first electronic device in this drag event.

[0013] In the above scheme, the web-type application can determine whether to block the drag event of the Google kernel by querying the existence of the preset object or the value of the preset object. If it is determined to block the drag event of the Google kernel, the web-type application obtains the cross-device dragged picture from the super keyboard and mouse service of the first electronic device instead of obtaining the picture from the Google kernel. Since the picture path indicated by the super keyboard and mouse service of the first electronic device is the picture path of the picture dragged across devices this time, if the picture is successfully transmitted, there will be no problem of not being able to obtain the picture or obtaining the picture of the previous cross-device drag, so the picture can be displayed normally in the window area of ​​the web-type application.

[0014] In an optional embodiment of the first aspect, the preset object is a first file; the first application determines whether to block the drag event of the Google kernel of the first electronic device by querying the preset object, including: the first application queries whether there is a first file; when the first file is queried, the first application blocks the drag event of the Google kernel of the first electronic device.

[0015] In the above solution, the first application determines whether it needs to block the Google kernel once by querying whether the first file exists. If the Google kernel is blocked once, the first application will not get the image from the Google kernel this time.

[0016] In an optional embodiment of the first aspect, before the first application obtains the first picture from the picture path indicated by the super keyboard and mouse service, the above method also includes: the first application queries whether there is a second file; the second file is a file created by the super keyboard and mouse service, used to indicate whether the first picture is transmitted successfully and the picture path of the first picture that is transmitted successfully; when the second file is queried, the first application obtains the picture path from the second file.

[0017] In the above scheme, when the first application queries the first file, the first application can further query the second file. If the first image is successfully transmitted, the first application can obtain and display the first image through the image path in the second file.

[0018] In an optional embodiment of the first aspect, the first application queries whether there is a second file, including: in response to the user's drag and release operation on the first image, the first application queries whether there is a first file; when the first file is queried, the first application queries whether there is a second file according to a preset query frequency.

[0019] In this embodiment, the preset query frequency may be once every 2 seconds. It should be understood that the cross-device transmission of the first image takes time. After the image transmission is completed (may be successful or fail), the super keyboard and mouse service of the first electronic device creates the second file. Before this, the first application cannot query the second file, so the first application needs to query multiple times.

[0020] In some embodiments, when the first file is not found, the first application obtains the image from the Google kernel of the first electronic device.

[0021] In the above solution, when the first application queries the first file, the first application can query the second file according to the preset query frequency to obtain the image path of the first image that is successfully transmitted.

[0022] In an optional embodiment of the first aspect, the above method also includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service of the first electronic device creates a first file; the first file is used to indicate that the super keyboard and mouse service is dragging the picture across devices.

[0023] In the above solution, when the super keyboard and mouse service of the first electronic device detects the dragging of an image across devices, it creates a first file to indicate that the dragging is a cross-device dragging.

[0024] In an optional embodiment of the first aspect, the above method also includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service queries whether there is a third file; the third file is a file created by the super keyboard and mouse service when the third picture was dragged across devices the last time, and is used to indicate whether the third picture is transmitted successfully and the picture path of the successfully transmitted third picture; if the third file is queried, the super keyboard and mouse service deletes the third file.

[0025] In the above scheme, after the super keyboard and mouse service of the first electronic device creates the first file for this cross-device dragging, it needs to delete the third file used to indicate the image transmission status of the previous cross-device dragging, so as to avoid the situation where, after creating the second file used to indicate the image transmission status of this cross-device dragging, the first application cannot determine whether to obtain the image display from the third file or from the image path indicated by the second file, that is, to avoid file confusion.

[0026] In an optional embodiment of the first aspect, the above method also includes: in response to the user's drag and release operation on the first picture, the super keyboard and mouse service notifies the super keyboard and mouse service of the second electronic device to start transmitting the first picture; after the transmission of the first picture is completed, the super keyboard and mouse service deletes the first file and creates a second file.

[0027] In the above scheme, if the first file is not deleted after the first image is transferred, the first application will think that the super keyboard and mouse service of the first electronic device has been dragging the image. Therefore, the first file is deleted to ensure that the local image dragging process on the first electronic device can run normally. The second file is created to indicate the image position of the first image dragged across devices by the first application.

[0028] In an optional embodiment of the first aspect, the preset object is a first mark bit; the first application determines whether to block the drag event of the Google kernel of the first electronic device by querying the preset object, including: the first application queries the first mark bit; when the first mark bit is a first value, the first application blocks the drag event of the Google kernel; or when the first mark bit is a second value, the first application does not block the drag event of the Google kernel; when the drag event of the Google kernel is not blocked, the first application obtains a picture from the Google kernel.

[0029] Exemplarily, the first flag bit may be recorded as flag, the first value is true, and the second value is false.

[0030] In the above solution, the first application queries the value of the first flag to determine whether the Google kernel needs to be blocked once. If the Google kernel is blocked once, the first application will not get the image from the Google kernel this time.

[0031] In an optional embodiment of the first aspect, before the first application obtains the first image from the image path indicated by the super keyboard and mouse service, the above method also includes: the first application queries a second flag bit, and the second flag bit is used to indicate whether the first image is transmitted successfully; when the second flag bit indicates that the first image is transmitted successfully, the first application obtains the first image from the super keyboard and mouse service.

[0032] In this embodiment, the first application obtains the image path of the first image that is successfully transmitted from the third notification message sent by the super keyboard and mouse service.

[0033] In the above scheme, when the first application determines that the first mark bit is the first value, the first application can determine whether the first picture dragged this time is successfully transmitted by querying the value of the second mark bit. If the transmission is successful, the picture path can be obtained from the message sent by the super keyboard and mouse service of the first electronic device (i.e., the third notification message).

[0034] In an optional embodiment of the first aspect, the above method also includes: after the first application is started, the first application subscribes to the super keyboard and mouse service for related events of cross-device dragging, so that when the related events are triggered, the first application obtains data of the related events from the super keyboard and mouse service; the related events of cross-device dragging include at least one of the following: the picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device; the picture is dragged from the display screen of the second electronic device to the first window.

[0035] In this embodiment, the first application obtains data of related events from the super keyboard and mouse service of the first electronic device, including but not limited to the identifier of the related event, the coordinate data of the cursor, the thumbnail of the picture dragged across devices, the picture path (if the picture is transferred successfully), etc.

[0036] In the above scheme, the first application registers events with the super keyboard and mouse service of the first electronic device, so that when related events of cross-device dragging are triggered, the first application responds to related events and makes corresponding decisions, such as determining whether to block the Google kernel.

[0037] In an optional embodiment of the first aspect, the above method also includes: after the first application is started, the first application initializes a first flag bit and a second flag bit; the first flag bit is used to indicate whether to block the drag event of the Google kernel of the first electronic device, and the second flag bit is used to indicate whether the image dragged across devices is transmitted successfully.

[0038] Exemplarily, the initial first flag bit may be recorded as flag=false, and the initial second flag bit may be recorded as dataflag=false.

[0039] In the above scheme, the first application creates two flags to respectively record whether the drag event of the Google kernel is currently blocked and the current transmission status of the dragged image across devices. The first application determines the subsequent execution action by querying these two flags.

[0040] In an optional embodiment of the first aspect, the above method also includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service of the first electronic device sends a first notification message to the first application; the first application modifies the first flag bit based on the first notification message, and the modified first flag bit is used to indicate that the drag event of the Google kernel of the first electronic device is blocked.

[0041] In this embodiment, the first notification message is used to indicate that an event of dragging a picture across devices occurs at the edge of the screen of the first electronic device.

[0042] In the above solution, the first application modifies the first flag bit based on the first notification message to record that the drag event of the Google kernel needs to be shielded once, that is, the image is not obtained from the Google kernel this time.

[0043] In an optional embodiment of the first aspect, the above method also includes: in response to the user's drag and release operation on the first image, the super keyboard and mouse service of the first electronic device sends a second notification message to the first application; the first application queries the first flag bit in response to the second notification message; when the first flag bit indicates that the drag event of the Google kernel of the first electronic device is blocked, the first application starts a timer to receive a third notification message from the super keyboard and mouse service within the duration of the timer, and the third notification message is used to indicate whether the first image is transmitted successfully.

[0044] In this embodiment, the second notification message is used to indicate that the first image dragged across devices is dragged to the window of the first application and released.

[0045] In the above scheme, after receiving the second notification message, the first application queries the first flag to determine whether the image released by dragging is from the first electronic device or from another device (such as the second electronic device). After determining that it is a picture dragged across devices, a timer is started to wait for the transmission of the picture dragged across devices.

[0046] In an optional embodiment of the first aspect, the above method also includes: before the timer expires, the first application receives a third notification message; if the third notification message is used to indicate that the first image transmission failed, the first application modifies the first mark bit; or, if the third notification message is used to indicate that the first image transmission is successful, the first application modifies the first mark bit and the second mark bit.

[0047] In this embodiment, the modified first flag is used to indicate that the drag event of the Google kernel of the first electronic device is not blocked, which can be recorded as flag = false. The modified second flag is restored to the initial value (ie, the second value), which can be recorded as dataflag = false.

[0048] In the above scheme, before the timer times out, the third notification message received by the first application may indicate two situations. Based on different situations, the first application modifies at least one of the two mark bits to update the mark bit value based on the picture transmission situation.

[0049] In an optional embodiment of the first aspect, the method further includes: when the timer times out and the first application does not receive the third notification message, the first application modifies the first mark bit.

[0050] In the above solution, the first application modifies the first flag bit from the first value (true) to the second value (false), and no longer receives the third notification message of the super keyboard and mouse service from the first electronic device.

[0051] In an optional embodiment of the first aspect, the method further includes: after the first electronic device is turned on, starting a super keyboard and mouse service of the first electronic device.

[0052] In this embodiment, the super keyboard and mouse service of the first electronic device may be used to detect, for example, a cross-device drag event, a drag release event, and the like.

[0053] In an optional embodiment of the first aspect, the first electronic device or the second electronic device is connected to an input device, and the above method also includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the first electronic device displays a thumbnail of the first picture, and a cursor is displayed on the thumbnail of the first picture; in response to the drag operation applied to the input device, during the process of the first picture being dragged from the edge of the display screen of the first electronic device to the first window, the first electronic device displays an animation of the thumbnail of the first picture moving with the cursor on the display screen of the first electronic device.

[0054] In this embodiment, when the second electronic device detects that the first image reaches the edge of the display screen of the second electronic device, the second electronic device sends drag data to the first electronic device, and the drag data includes, for example, a thumbnail of the first image, cursor coordinate data, etc.

[0055] In a second aspect, an embodiment of the present application provides an electronic device, which is the first electronic device as described in any one of the first aspects, and the first electronic device includes: a memory and at least one processor; the memory is used to store computer program code, and the computer program code includes instructions; when the at least one processor executes the instructions, the first electronic device executes the method as described in any one of the first aspects.

[0056] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by an electronic device, the method described in any one of the first aspects is implemented.

[0057] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, an electronic device executes any method as described in the first aspect.

[0058] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute any method as described in the first aspect.

[0059] In a sixth aspect, an embodiment of the present application provides a cross-device drag system, including: a first electronic device and a second electronic device, the first electronic device being connected to the second electronic device; the first electronic device executing the method as described in any one of the first aspects.

[0060] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding optional embodiments are similar and will not be repeated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of dragging an image across devices provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of dragging an image across devices provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of dragging an image across devices provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a first electronic device and a second electronic device in a cross-device dragging system provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of a second electronic device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of a flow chart of a method for displaying a cross-device image by dragging it provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of a flow chart of a method for displaying a cross-device image by dragging it provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of a flow chart of a method for displaying a cross-device image by dragging it provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of a flow chart of a method for displaying a cross-device image by dragging it provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of a first electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0072] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0073] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0074] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, such as image data) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0075] Drag (or drag and drop) is mainly used for data transmission. Users can use input devices such as stylus, mouse, touchpad, etc. to drag the object to be dragged (such as files, pictures, etc.) to the specified location. Traditional drag scenarios include: drag on the PC and drag on touch terminals such as mobile phones. For drag on the PC, users can move the object to be dragged by clicking the left mouse button (or other buttons on the mouse) and dragging, and move the object from one position to another on the PC display by releasing the left mouse button (or other buttons on the mouse). For drag on touch terminals such as mobile phones, users can press and drag the selected object on the touch screen with their fingers or stylus to move the object until the finger or stylus leaves the touch screen, and move the object from one position to another on the mobile phone touch screen.

[0076] In a multi-device collaboration scenario, dragging is a function that users often use, and dragging in a multi-device collaboration scenario involves dragging an object across multiple devices. The following embodiment is described by taking dragging a picture between two devices as an example.

[0077] Currently, when dragging an image between two devices, the following phenomena occur:

[0078] In one example, Figure 2The PC shown in the figure establishes a connection with the mobile phone (e.g., establishing a Bluetooth connection, or a WiFi P2P (a protocol released by the WiFi Alliance, also known as WiFi Direct) connection), the PC is connected to the wireless mouse (e.g., a Bluetooth connection), and when the user opens an application on the PC, the PC displays the application window (e.g., Figure 2 The user can move the wireless mouse so that the cursor passes from the right edge of the PC display screen to the mobile phone display screen, and then the mobile phone display screen displays picture 1. The user can select picture 1 on the mobile phone by clicking the left button of the wireless mouse and dragging it, and drag it from the mobile phone display screen to the PC display screen, and release the left button of the mouse to drag picture 1 to the designated position 21 in the application window area of ​​the PC. However, after releasing the left button of the mouse, picture 1 is not displayed at the designated position 21 of the application window. This phenomenon mainly occurs in global wide area network (web) type applications, and when a picture is dragged across devices to a web type application window for the first time, there is a problem that the dragged picture is not displayed.

[0079] Web-type applications are applications accessed through the web. They are typical client / server network applications. The interaction between the client and the server is based on the hypertext transfer protocol (HTTP). Web-type applications include but are not limited to browsers, super notes and other applications.

[0080] In one example, when a user executes Figure 2 After dragging the picture shown in Figure 3 As shown, the user moves the wireless mouse again so that the cursor passes from the right side of the PC display screen to the mobile phone display screen. The user clicks the left button of the wireless mouse and does not release it while dragging to select the picture 2 on the mobile phone, drag it from the mobile phone display screen to the PC display screen, and at the designated position 31 of the application window area on the PC, the user releases the left button of the mouse to drag the picture 2 to the designated position 31. However, after releasing the left button of the mouse, the picture 2 dragged for the second time is not displayed at the designated position 31, but the picture 1 dragged most recently is displayed. This phenomenon also occurs in web-type applications.

[0081] Similarly, using a similar operation method, when the user drags picture 3 from the mobile phone to a specified position in the PC application window area for the third time and releases it, the third dragged picture 3 is not displayed at the specified position, but the second dragged picture 2 is displayed. That is, when dragging a picture to a web-type application window across devices other than for the first time, there is a problem of misaligned display of the dragged picture.

[0082] In response to the above-mentioned problems, an embodiment of the present application provides a method for displaying dragged images across devices. Users can use input devices such as wired or wireless mice, touchpads, styluses, touch screens, etc. to drag images from one device to a web application window of another device. The dragged image and the image that is finally displayed are the same image. Problems such as the image not being displayed the first time the image is dragged and the dragged image being displayed out of position will not occur, thereby improving the user experience of dragging images.

[0083] In order to facilitate understanding of the solution, the system architecture of the technical solution provided by this application is first described below.

[0084] For example, Figure 4 A schematic diagram of a system architecture provided in an embodiment of the present application. Figure 4 As shown, the system architecture at least includes: a first electronic device 101 and a second electronic device 102. The first electronic device 101 and the second electronic device 102 can establish a connection via a wired or wireless method. Based on the established connection, the first electronic device 101 and the second electronic device 102 can be used in a collaborative manner, for example, allowing the user to use input devices such as a mouse, a touchpad, a stylus, a touch screen, etc. to drag objects such as application windows, video components, floating windows, widgets, application icons, UI controls, pictures, etc. from the first electronic device 101 to the collaboratively used second electronic device 102, or from the second electronic device 102 to the collaboratively used first electronic device 101. The first electronic device 101 and the second electronic device 102 can share the above objects.

[0085] In one example, the first electronic device 101 and the second electronic device 102 establish a connection wirelessly, and the wireless communication protocol adopted may be a wireless fidelity (Wi-Fi) protocol, a Bluetooth protocol, a ZigBee protocol, a near field communication (NFC) protocol, various cellular network protocols, etc., and this embodiment of the application does not impose any limitation on this.

[0086] The first electronic device 101 and the second electronic device 102 may be electronic devices such as mobile phones, tablet computers, handheld computers, PCs, cellular phones, personal digital assistants (PDAs), vehicle-mounted devices, wearable devices, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in smart homes, etc. The embodiments of the present application do not specifically limit the device forms of the first electronic device 101 and the second electronic device 102. In some embodiments, the device forms of the first electronic device 101 and the second electronic device 102 may be the same, such as both the first electronic device 101 and the second electronic device 102 are mobile phones. In other embodiments, the device forms of the first electronic device 101 and the second electronic device 102 may be different, for example, Figure 1 The first electronic device 101 shown may be a PC, and the second electronic device 102 may be a mobile phone.

[0087] The first electronic device 101 and the second electronic device 102 can be touch screen devices or non-touch screen devices. In the embodiment of the present application, the first electronic device 101 and the second electronic device 102 are both electronic devices that can run an operating system, install applications, and have a display (or display screen). The operating system of the first electronic device 101 and the second electronic device 102 can be an Android system, an iOS system, a Windows system, a Mac system, a Linux system, etc., and the embodiment of the present application does not impose any limitation on this. The operating systems of the first electronic device 101 and the second electronic device 102 are different. In an example, the first electronic device 101 and the second electronic device 102 may include a memory, a processor and a display, respectively, wherein the memory is used to store the operating system, and the processor is used to run the operating system stored in the memory. Exemplarily, Figure 4 The operating system of the first electronic device 101 is a Windows system, and the operating system of the second electronic device 102 is an Android system.

[0088] In the embodiment of the present application, when the first electronic device 101 is connected to the second electronic device 102, the user can use the input device of the first electronic device 101, such as a mouse, touchpad, stylus or touch screen, to move the cursor to the display screen of the second electronic device 102, and drag the picture displayed on the display screen of the second electronic device 102 to the window of the web-type application on the display screen of the first electronic device 101. For example, Figure 1As shown, taking the first electronic device 101 as a PC and the second electronic device 102 as a mobile phone as an example, the PC opens a web-type application, and the user can drag the picture displayed on the mobile phone into the window of the web-type application of the PC.

[0089] It should be noted that the above is explained by taking the example of a user dragging a picture displayed on the second electronic device 102 from the first electronic device 101 to a first electronic device 101. In some embodiments, when the second electronic device 102 is connected to multiple first electronic devices 101, the user can drag the picture displayed on the second electronic device 102 to one or more devices of the multiple first electronic devices 101 by dragging. Among them, the device forms of different first electronic devices 101 can be the same or different. The connection method between the second electronic device 102 and different first electronic devices 101 can be the same or different, and this embodiment does not specifically limit this.

[0090] In the embodiment of the present application, the first electronic device 101 is taken as a PC as an example. Figure 4 As shown, the first electronic device 101 may include an application layer, a windows system and a hardware layer. The application layer may include web-type applications, such as Figure 4 The super note application shown, the super note application obtains the drag release event and the dragged picture through the Google kernel. The application layer may also include a super keyboard and mouse service, which can obtain the drag event, drag release event, etc. across devices on the first electronic device 101. The windows system may include a Bluetooth driver and a wireless fidelity (WiFi) driver, the Bluetooth driver is used to drive the Bluetooth communication module of the hardware layer to work, and the WiFi driver is used to drive the WiFi P2P module of the hardware layer to work. In one example, the super keyboard and mouse service of the first electronic device 101 calls the Bluetooth driver to trigger the Bluetooth communication module of the first electronic device 101 to interact with the Bluetooth communication module of the second electronic device 102 for data, such as transmitting drag data, etc. In another example, in one example, the super keyboard and mouse service of the first electronic device 101 calls the WiFi driver to trigger the WiFi P2P module of the first electronic device 101 to interact with the WiFi P2P module of the second electronic device 102 for data, such as transmitting drag data, etc.

[0091] In the embodiment of the present application, taking the second electronic device 102 as a mobile phone as an example, the software system of the second electronic device 102 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, taking the software system of the layered architecture as an Android system as an example, the software structure of the second electronic device 102 is exemplified.

[0092] like Figure 4 As shown, the layered architecture divides the software system of the second electronic device 102 into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers, namely, application layer (applications), application framework layer (application framework), Android runtime (Android runtime) and system library, and kernel layer (kernel).

[0093] The application layer may include a series of application packages. The application layer includes applications such as photo albums, folders, navigation, browsers, emails, calendars, alarm clocks, etc. In the embodiment of the present application, the application layer also includes a super keyboard and mouse service. Similar to the super keyboard and mouse service of the first electronic device 101, the super keyboard and mouse service of the second electronic device 102 can be informed of any drag event, drag release event, etc. on the second electronic device 102, and can trigger the Bluetooth communication module / WiFi P2P module of the second electronic device 102 to interact with the Bluetooth communication module / WiFi P2P module of the first electronic device 101 by calling the Bluetooth / WiFi driver of the second electronic device 102 to perform data, such as transmitting drag data, etc.

[0094] The application framework layer provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer may include, for example, input management service (IMS), display policy service, power management service (PMS), display management service (DMS), activity manager, window manager, content provider, view system, telephony manager, resource manager, notification manager, etc.

[0095] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0096] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0097] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0098] The system library can include multiple functional modules. For example: surface manager, media library, three-dimensional (3D) graphics processing library (for example: OpenGL ES), two-dimensional (2D) graphics engine (for example: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0099] The kernel layer is a layer between hardware and software. The kernel layer includes, for example, display drivers, camera drivers, audio drivers, sensor drivers, Bluetooth drivers, WiFi drivers, etc., and the present application embodiment does not impose any restrictions on this.

[0100] In the embodiment of the present application, the hardware layer of the second electronic device 102 at least includes a Bluetooth communication module, a WiFi P2P module, a display screen, etc.

[0101] It should be noted that Figure 4 The layers in the hierarchical structure of the first electronic device 101 and the second electronic device 102 and the modules or components included in each layer do not constitute a specific limitation on the first electronic device 101 and the second electronic device 102. In other embodiments, the first electronic device 101 and the second electronic device 102 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the embodiments of the present application.

[0102] See attached Figure 5In the embodiment of the present application, the second electronic device 102 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0103] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the second electronic device 102. In other embodiments of the present application, the second electronic device 102 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0104] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, a display processing unit (displayprocess unit, DPU), and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. In some embodiments, the second electronic device 102 may also include one or more processors 110. Among them, the controller may be a nerve center and command center of the second electronic device 102. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. The memory may save instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the second electronic device 102 system.

[0105] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. Among them, the USB interface 130 is an interface that complies with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the second electronic device 102, and can also be used to transmit data between the second electronic device 102 and a peripheral device. It can also be used to connect headphones and play audio through the headphones.

[0106] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the second electronic device 102. In other embodiments of the present application, the second electronic device 102 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0107] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the second electronic device 102. While the charging management module 140 is charging the battery 142, it may also power the second electronic device 102 through the power management module 141.

[0108] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0109] The wireless communication function of the second electronic device 102 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the second electronic device 102 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 can be reused as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0110] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the second electronic device 102. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier, etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0111] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0112] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc., applied to the second electronic device 102. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0113] In some embodiments, the antenna 1 of the second electronic device 102 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the second electronic device 102 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include GSM, GPRS, CDMA, WCDMA, TD-SCDMA, LTE, GNSS, WLAN, NFC, FM, and / or IR technology. The above-mentioned GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0114] The second electronic device 102 can realize the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute instructions to generate or change display information.

[0115] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the second electronic device 102 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0116] The second electronic device 102 can implement a shooting function through an ISP, one or more cameras 193, a video codec, a GPU, one or more display screens 194, and an application processor.

[0117] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, the second electronic device 102 can realize intelligent cognition and other applications, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0118] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the second electronic device 102. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, data files such as music, photos, and videos are stored in the external memory card.

[0119] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can enable the second electronic device 102 to perform various functional applications and data processing by running the above instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the second electronic device 102. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. In some embodiments, the processor 110 can enable the second electronic device 102 to perform various functional applications and data processing by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor 110.

[0120] The second electronic device 102 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

[0121] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0122] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The second electronic device 102 can listen to music or listen to a hands-free call through the speaker 170A.

[0123] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the second electronic device 102 receives a call or voice message, the receiver 170B can be placed close to the ear to receive the voice.

[0124] The microphone 170C, also called a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 170C with his mouth, and the sound signal is input into the microphone 170C. The second electronic device 102 can be provided with at least one microphone 170C.

[0125] The earphone jack 170D is used to connect a wired earphone.

[0126] The sensor 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0127] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The second electronic device 102 may receive key input and generate key signal input related to user settings and function control of the second electronic device 102.

[0128] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the second electronic device 102 by inserting or removing the SIM card from the SIM card interface 195. The second electronic device 102 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0129] The following is combined with Figure 4 ,right Figure 2 and Figure 3 The scheme of display effect of dragging pictures across devices is described and analyzed in detail. Figure 6The present invention provides a flowchart of a method for displaying a cross-device image by dragging and dropping. The method can be applied to, for example, Figure 4 The first electronic device 101 shown in the figure, the method involves the internal execution process between the super keyboard and mouse service, the Google kernel and the web type application in the first electronic device 101.

[0130] For ease of understanding, the following describes the solution by taking the first electronic device 101 as a PC, the second electronic device 102 as a mobile phone, and the web-type application installed on the PC as a super note application as an example.

[0131] When the PC is connected to the mobile phone, the user opens the Super Note on the PC. The user drags Picture 1 from the mobile phone to the window area of ​​the Super Note on the PC for the first time. Figure 6 As shown, the method includes:

[0132] S601a. ​​When a drag-and-release operation is detected in the window area of ​​the PC-side super note, the PC-side super keyboard and mouse service notifies the mobile-side super keyboard and mouse service to start transmitting picture 1.

[0133] S602. After image 1 is successfully transmitted, the PC-side super keyboard and mouse service notifies the PC-side Google kernel to cache image 1.

[0134] S603. Google kernel cache image 1 on PC.

[0135] In some embodiments, while executing S601a, the following may also be executed:

[0136] S601b. When a drag-and-release operation is detected in the window area of ​​the PC-side super note, the PC-side Google kernel notifies the super note of a drag-and-release event, wherein the drag-and-release event is used to indicate that a drag-and-release operation has occurred in the window area of ​​the super note.

[0137] In this embodiment, the picture 1 is the picture that the user dragged from the mobile phone for the first time. After the PC-side Google kernel detects the drag-and-release operation, since the PC-side Google kernel has not cached any pictures, that is, there is no picture in the cache of the PC-side Google kernel, when the PC-side Google kernel triggers to notify the PC-side Super Note of the drag-and-release event, the dragged picture 1 is not transmitted to the PC-side Super Note. When the picture 1 is successfully transmitted from the mobile phone to the PC, the PC-side Super Keyboard and Mouse Service notifies the PC-side Google kernel to cache the picture 1. After the PC-side Google kernel caches the picture 1, it will not trigger the transmission of the picture 1 to the PC-side Super Note again. This results in the phenomenon that the PC-side Super Note window does not display the picture when the user drags the picture from the mobile phone to the window area of ​​the PC-side Super Note for the first time.

[0138] Continue to refer Figure 6In some embodiments, the user drags the picture 2 from the mobile phone to the window area of ​​the PC super note for the second time, and the method further includes:

[0139] S604a. When a drag-and-release operation is detected again in the window area of ​​the PC-side super note, the PC-side super keyboard and mouse service notifies the mobile-side super keyboard and mouse service to start transmitting picture 2.

[0140] S605. After image 2 is successfully transmitted, the PC-side super keyboard and mouse service notifies the PC-side Google kernel to cache image 2.

[0141] S606. The Google kernel on the PC caches the image 2.

[0142] In some embodiments, while executing S604a, the following may also be executed:

[0143] S604b. When a drag-and-release operation is detected again in the window area of ​​the PC-side super note, the PC-side Google kernel notifies the PC-side super note of the drag-and-release event and transmits the cached image 1 to the PC-side super note.

[0144] S607. The PC-side Super Note displays Picture 1 in the window area.

[0145] Image 2 in this embodiment is the image dragged by the user from the mobile phone for the second time. Assuming that Image 1, which was dragged most recently (i.e., the first drag) before the second drag, has been successfully transmitted, after the Google kernel on the PC detects the drag-and-release operation again, since the Google kernel on the PC has cached Image 1, when the Google kernel on the PC triggers again to notify the Super Note on the PC of the drag-and-release event, Image 1 in the cache will be transmitted to the Super Note on the PC at the same time. In this way, when the user drags the image from the mobile phone to the window area of ​​the Super Note on the PC for the second time, the window of the Super Note on the PC will display Image 1, which results in the dragged image being displayed out of place.

[0146] Similar to the first image transfer, when Image 2 is successfully transferred from the mobile phone to the PC, the PC Super Keyboard and Mouse Service will notify the PC Google Kernel to cache Image 2. After the PC Google Kernel caches Image 2, it will not trigger the transfer of Image 2 to the PC Super Note again.

[0147] It should be noted that the PC-side Google kernel only caches the most recently dragged picture. For example, when picture 2 is successfully transferred, the PC-side Google kernel will store the second picture after deleting picture 1, that is, the PC-side Google kernel stores pictures in a replacement manner.

[0148] It should also be noted that, regardless of whether it is the first or second time to drag the image, after the drag and release operation, the PC-side super keyboard and mouse service notifies the mobile-side super keyboard and mouse service to start transmitting the dragged image. After the image is successfully transmitted, the dragged image will be stored in a temporary path (or designated path) on the PC side. The PC-side Google kernel can cache the image under the temporary path to the PC-side Google kernel by calling a preset function (such as the DoDragDrop function).

[0149] based on Figure 6 In the display method of dragging and dropping images across devices shown, when a user drags an image from a mobile phone to a PC-side super note for the first time, the PC-side Google kernel will directly obtain the image file from the cache and send the cached image file to the PC-side super note. However, since there is no image file in the cache at this time (the image is still being transferred, or has just been successfully transferred and has not yet been cached in the Google kernel), the PC-side super note has no image to display. When the user drags an image from a mobile phone to a PC-side super note for the second time, the PC-side Google kernel will directly obtain the image file of the first drag from the cache and send the image file to the PC-side super note, causing the PC-side super note to display the image of the first drag.

[0150] In order to solve the problem that the image is not displayed for the first drag and the dragged image is displayed misaligned, the embodiment of the present application improves the display method of cross-device image dragging, and the improvement scheme includes:

[0151] Solution 1: The PC-side Super Keyboard and Mouse Service creates a first file to indicate that the PC-side Super Keyboard and Mouse Service is dragging images across devices. In other words, the existence of the first file instructs the PC-side Super Note to block a drag event of the PC-side Google kernel. The PC-side Super Keyboard and Mouse Service creates a second file to indicate whether the image dragged across devices on the PC-side Super Note is successfully transferred and the image path.

[0152] Solution 2: PC-side Super Note registers PC-side Super Keyboard and Mouse Service to obtain drag events, drag release events, image paths, etc. from the PC-side Super Keyboard and Mouse Service. PC-side Super Note sets the first mark bit and the second mark bit. Based on the first mark bit, PC-side Super Note can determine whether to obtain the cross-device dragged image from the cache of the PC-side Google kernel or from the path indicated by the PC-side Super Keyboard and Mouse Service. Based on the second mark bit, PC-side Super Note can know whether the cross-device dragged image is successfully transferred to the PC.

[0153] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0154] For example, Figure 7 The present invention provides a flowchart of a method for displaying a cross-device image by dragging and dropping. The method can be applied to, for example, Figure 4 For ease of understanding, the following example illustrates a solution using the first electronic device 101 as a PC, the second electronic device 102 as a mobile phone, and the web application installed on the PC as a super note application. When the PC and the mobile phone are connected, the user opens the super note on the PC. Figure 7 As shown, the method includes:

[0155] S701. When a drag event is detected, the PC-side super keyboard and mouse service creates a first file.

[0156] In this step, the drag event refers to the event of dragging the first image from the mobile phone to the edge of the PC screen. The purpose of creating the first file is to indicate that the PC-side super keyboard and mouse service is dragging images across devices. In this way, the PC-side super note can query whether the first file exists (see S708 below for details) to know whether the PC-side super keyboard and mouse service is dragging images across devices. If the first file exists, it means that the current scene is a scene of dragging images across devices with the keyboard and mouse, that is, the PC-side super keyboard and mouse service is dragging images across devices. If the first file does not exist, it means that the current scene is a scene of non-keyboard and mouse dragging images across devices, such as dragging images between the application window on the PC and the desktop.

[0157] Exemplarily, the first file may be a file in XML format, the file name may be DragToSuperNote, and the storage path may be C:\Program Files\HONOR\PCManager\DragToSuperNote.xml.

[0158] In this embodiment, the first picture can be the picture that the user drags from the mobile phone for the first time after opening the super note application on the PC, or it can be the picture that the user drags from the mobile phone for the i-th time (i is a positive integer greater than or equal to 2). This embodiment does not make any specific limitations on this.

[0159] S702. The PC-side super keyboard and mouse service determines whether there is a third file.

[0160] In this step, the third file is a file created by the PC-side super keyboard and mouse service when the image is dragged across devices the most recently. For example, before S701, a cross-device drag of an image is performed, such as dragging the third image, and the PC-side super keyboard and mouse service creates a third file for the third image dragged across devices, and the third file is used to indicate whether the third image dragged across devices is successfully transmitted.

[0161] If the PC-side super keyboard and mouse service determines that there is a third file, execute S703; if the PC-side super keyboard and mouse service determines that there is no third file, jump to S704.

[0162] S703. The PC-side super keyboard and mouse service deletes the third file.

[0163] In this embodiment, S703 is an optional step.

[0164] In some embodiments, the first picture dragged by the user in S701 is the picture that the user drags from the mobile phone to the PC-side super note for the first time after opening the super note on the PC. It can be understood that since the PC-side super keyboard and mouse service has not yet obtained the dragged picture from the mobile phone-side keyboard and mouse service, the PC-side super keyboard and mouse service will not query the third file and will not delete the third file.

[0165] In some embodiments, the first picture dragged by the user in S701 is the picture dragged from the mobile phone to the PC-side super note for the i-th time after the user opens the super note on the PC, where i is a positive integer greater than or equal to 2. Taking the first picture as the picture that the user drags from the mobile phone to the PC-side super note for the second time as an example, assuming that the third picture is the picture that the user drags from the mobile phone to the PC-side super note for the first time, after the third picture is transferred, the PC-side super keyboard and mouse service will create a third file, and the third file is used to indicate whether the third picture is successfully transferred and the picture path of the third picture that is successfully transferred. It is understandable that if the PC-side super keyboard and mouse service does not delete the third file created when the picture is dragged for the first time before the PC-side super note obtains the second cross-device drag picture, it will cause file confusion. Therefore, the PC-side super keyboard and mouse service needs to query whether there is a file indicating the picture path that has been created historically after detecting a new cross-device drag event. If so, the file needs to be deleted in advance.

[0166] S704. When a drag-release event is detected, the PC-side super keyboard and mouse service notifies the mobile-side super keyboard and mouse service to start transmitting the first dragged picture.

[0167] In this embodiment, the drag-and-release event refers to the event of dragging the first image from the edge of the PC screen and releasing the first image in the window area of ​​the super note. After the first image is transferred, you can execute:

[0168] S705. The PC-side super keyboard and mouse service deletes the first file.

[0169] The purpose of deleting the first file by the PC-side Super Keyboard and Mouse Service is to not affect the image dragging process of the PC-side Super Note itself (for example, by dragging the image on the PC Windows desktop to the Super Note, the desktop image can be obtained from the PC-side Google Kernel). It can be understood that if the first file is not deleted after the first image dragged from the mobile phone is successfully transferred, since the PC-side Super Note can query the first file, it will be considered that the PC-side Super Keyboard and Mouse Service has been dragging the file. Then, for example, when the user drags the image on the PC Windows desktop to the PC-side Super Note, the PC-side Super Note will still block the drag event of the PC-side Google Kernel, resulting in the inability to drag the image on the Windows desktop to the PC-side Super Note. Therefore, after transferring the image across devices, the first file needs to be deleted to ensure the normal operation of the subsequent local image dragging on the PC.

[0170] S706. The PC-side super keyboard and mouse service creates a second file.

[0171] In this step, the second file is used to indicate whether the first picture dragged from the mobile phone is successfully transferred, that is, the purpose of the PC-side super keyboard and mouse service to create the second file is to indicate whether the first picture dragged across devices is successfully transferred. If the first picture is successfully transferred, the second file is also used to indicate the image path of the first picture, so that the PC-side super note can obtain and display the first picture dragged across devices based on the image path in the second file.

[0172] Exemplarily, the second file may be a file in XML format, the file name may be DragToSuperNoteFileList, and the storage path may be: C:\Program Files\HONOR\PCManager\DragToSuperNoteFileList.xml. Optionally, the second file may include the current protocol version, data marked as prepared for "Super Note", the time when the dragged image transfer is completed, the image path, etc.

[0173] In some embodiments, while executing S704, the following may also be executed:

[0174] S707. When a drag-and-release event is detected, the PC-side Google kernel notifies the PC-side Super Note of the drag-and-release event.

[0175] In this step, the Google kernel on the PC can detect the drag and drop event.

[0176] S708. The PC-side super note queries whether there is a first file.

[0177] If the PC-side super note finds the first file, execute S709a; or, if the PC-side super note does not find the first file, execute S709b.

[0178] S709a. The PC-side Super Note blocks the drag events detected by the PC Google kernel.

[0179] S709b. The PC-side Super Note executes the process of dragging pictures in Windows. For the process of dragging pictures in Windows, please refer to Figure 6 The illustrated embodiment mainly includes: when the PC-side super note receives a drag-and-release event notified by the PC-side Google kernel, it can obtain a picture from the PC-side Google kernel and display the picture in the window area of ​​the PC-side super note.

[0180] It should be noted that it takes a certain amount of time for the PC-side super keyboard and mouse service to notify the mobile-side super keyboard and mouse service to transfer the dragged first picture. The first file will be deleted only after the first picture is successfully transferred (S705). Therefore, before executing S705, the PC-side super note has completed the query for the first file (S708), and generally the situation where the first file cannot be found will not occur.

[0181] Based on S707 to S709a / S709b, it can be known that after the PC-side super note learns the drag release event, the PC-side super note learns whether the drag is a cross-device image drag of the PC-side super keyboard and mouse service by querying whether there is a first file. If the PC-side super note finds the first file, the PC-side super note will block the drag event detected by the PC-side Google kernel, that is, the PC-side super note will not obtain the image from the PC-side Google kernel when the drag is released. Alternatively, if the PC-side super note does not find the first file, the PC-side super note will not block the drag event detected by the PC-side Google kernel, that is, the PC-side super note will obtain and display the image from the PC-side Google kernel when the drag is released.

[0182] It should be noted that in S709a, the PC-side super note blocks the drag event detected by the PC Google kernel, which means that the PC-side super note blocks the drag event detected by the PC Google kernel once, that is, only blocks the Google kernel for this drag release event, and does not obtain the image from the Google kernel. It is understandable that after this drag release event, if a new cross-device drag image is known, the PC-side super note needs to re-query whether there is a first file to determine whether the Google kernel needs to be blocked this time.

[0183] After S709a, you can also execute:

[0184] S710. The PC-side super note queries whether there is a second file according to a preset query frequency.

[0185] For example, the query frequency may be once every 2 seconds, that is, the PC-side Super Note queries whether there is a second file under the specified path every 2 seconds. If the PC-side Super Note queries the second file within a preset time (such as within 30 seconds), the following can be executed:

[0186] S711. The PC-side Super Note determines whether the first dragged image is transferred successfully.

[0187] In one example, the second file includes a flag indicating whether the first image is successfully transferred by dragging it, and the PC-side Super Note obtains the flag from the second file to determine whether the first image is successfully transferred.

[0188] If the PC-side SuperNote determines that the first dragged image has been successfully transferred, you can execute:

[0189] S712. The PC-side super note obtains and displays the first dragged picture.

[0190] In one example, the PC-side super note obtains the image path from the second file, obtains the dragged first image under the image path, and displays it.

[0191] The method for displaying cross-device image dragging shown in this embodiment is that the PC-side super keyboard and mouse service creates a first file so that the PC-side super note can determine whether it is necessary to block the drag event of the PC-side Google kernel by querying whether the first file exists. After the PC-side super note queries the first file, the drag event of the Google kernel will be blocked once, that is, the PC-side super note will not obtain the dragged image from the Google kernel. In addition, the PC-side super keyboard and mouse service also creates a second file so that the PC-side super note can query the second file to know whether the image dragged across devices is successfully transmitted, and the image path after the successful transmission. The PC-side super note can obtain and display the dragged image based on the image path indicated by the second file.

[0192] compared to Figure 6 In the scheme shown, since the PC-side Super Keyboard and Mouse Service creates the first file and the second file, when the PC-side Super Note queries the first file and the second file indicates that the image dragged across devices is successfully transferred and the image path, the PC-side Super Note will not obtain the image from the PC-side Google kernel, but obtain the image transferred across devices based on the image path indicated by the second file. Therefore, Figure 7 The solution shown will not cause the problem of images not being displayed when dragged across devices. In addition, since the PC-side Super Keyboard and Mouse Service will create a new first file and second file based on this drag each time an image is dragged across devices, the PC-side local will not retain the first file and second file created by the last drag, which can prevent the PC-side Super Note from obtaining the last dragged image. Figure 7The solution shown will not cause the problem of misaligned display of dragged images.

[0193] For example, Figure 8 A flowchart of a method for displaying a cross-device image by dragging is provided in an embodiment of the present application. The method involves, for example Figure 4 The interaction between the first electronic device 101 and the second electronic device 102, and the internal execution of the first electronic device 101 are shown. For ease of understanding, the following example takes the first electronic device 101 as a PC, the second electronic device 102 as a mobile phone, and the web-type application installed on the PC as a super note application as an example to illustrate the solution. Figure 8 As shown, the display method includes:

[0194] S801. After the PC is turned on, the PC starts the PC-side super keyboard and mouse service.

[0195] After starting SuperNote on the PC, you can execute:

[0196] S802a. Initialize the first mark bit and the second mark bit of the PC-side super note.

[0197] In this step, the first flag is used to indicate whether to block the drag event of the Google kernel on the PC side, and the second flag is used to indicate whether the file (such as a picture) dragged across devices is successfully transmitted.

[0198] Exemplarily, the first flag bit can be recorded as flag, flag = false means that the drag event of the Google kernel on the PC side is not blocked (not ignored), and flag = true means that the drag event of the Google kernel on the PC side is blocked (ignored). The second flag bit can be recorded as dataflag, dataflag = false means that the image transmission of cross-device dragging fails, and dataflag = true means that the image transmission of cross-device dragging is successful. Based on this, the initial first flag bit flag can be set to false, and the initial second flag bit dataflag can be set to false.

[0199] S802b. Register the PC-side Super Note for the PC-side Super Keyboard and Mouse Service.

[0200] In one example, the PC-side Super Note registers a callback interface with the PC-side Super Keyboard and Mouse Service, and the PC-side Super Keyboard and Mouse Service can send messages to the PC-side Super Note through the callback interface, such as the first notification message, the second notification message, the third notification message, etc. described later.

[0201] Among them, S802a and S802b can be executed synchronously.

[0202] In some embodiments, after S802b, the method further includes:

[0203] S802c. The PC-side Super Keyboard and Mouse Service notifies the PC-side Super Note that the PC-side Super Keyboard and Mouse Service has been successfully registered.

[0204] In this embodiment, S802c is an optional step.

[0205] After the PC-side super keyboard and mouse service completes the drag event registration, it also includes:

[0206] S803. When a preset operation is detected at the edge of the PC screen, the PC super keyboard and mouse service establishes a connection with the mobile phone super keyboard and mouse service.

[0207] In one example, the PC-side Super Keyboard and Mouse Service detects that the user moves the mouse so that the cursor on the display screen quickly slides to the edge of the PC-side screen twice, triggering the PC-side Super Keyboard and Mouse Service to establish a connection with the mobile-side Super Keyboard and Mouse Service. The PC-side Super Keyboard and Mouse Service can establish a connection with the mobile-side Super Keyboard and Mouse Service by calling the PC-side communication module (such as Bluetooth communication module / WiFi P2P module). When the PC-side Super Keyboard and Mouse Service initiates a connection to the mobile-side Super Keyboard and Mouse Service, the mobile-side Super Keyboard and Mouse Service is started. After the connection is successful, the PC-side Super Keyboard and Mouse Service and the mobile-side Super Keyboard and Mouse Service can exchange data through Bluetooth connection / WiFi P2P connection.

[0208] After the PC-side super keyboard and mouse service establishes a connection with the mobile-side super keyboard and mouse service, it also includes:

[0209] S804. When the cursor is detected to have reached the edge of the PC screen, the PC-side super keyboard and mouse service sends a cursor event and coordinates to the mobile-side super keyboard and mouse service, wherein the cursor event is the event that the cursor has reached the edge of the PC screen.

[0210] S805. The mobile phone super keyboard and mouse service responds to the cursor event and coordinates and displays a simulated cursor on the mobile phone.

[0211] S806. When a drag event is detected, the mobile phone super keyboard and mouse service sends the drag event and coordinates to the PC-side super keyboard and mouse service. In this step, the drag event refers to the event of dragging the first picture from the mobile phone to the edge of the PC screen. In this embodiment, the first dragged picture can be the first picture dragged from the mobile phone after the user opens the super note application on the PC, or it can be the i-th picture dragged from the mobile phone by the user (i is a positive integer greater than or equal to 2), which is not specifically limited in this embodiment.

[0212] S807. In response to the drag event and the coordinates, the PC-side super keyboard and mouse service sends a first notification message to the PC-side super note.

[0213] In one example, in response to the drag event and coordinates, the PC-side super keyboard and mouse service sends a first notification message to the PC-side super note through a callback interface (callback start interface). The first notification message is used to indicate that the PC-side super keyboard and mouse service detects a drag event of dragging the first image from the mobile phone to the edge of the PC-side screen.

[0214] In one example, in response to the drag event and the coordinates, a cursor and a thumbnail of the first image are displayed on the PC.

[0215] S808. The PC-side super note modifies the first mark bit based on the first notification message.

[0216] In this step, the modified first flag bit is used to indicate that the drag event of the Google kernel on the PC is blocked. Exemplarily, based on the first flag bit flag=false in S802a, in this step, the modified first flag bit flag=true.

[0217] In this embodiment, a drag event is detected (dragging the first picture from the mobile phone to the edge of the PC screen), the PC super keyboard and mouse service notifies the PC super note that the drag event is triggered, and the PC super note sets the first flag bit to true, indicating that the drag event of the PC Google kernel is blocked.

[0218] In some embodiments, after S806, the method further includes:

[0219] S809. When a drag-and-release event is detected, the PC-side super keyboard and mouse service notifies the mobile-side super keyboard and mouse service to start transmitting the first image. In this embodiment, the drag-and-release event refers to dragging the first image from the edge of the PC-side screen and releasing the first image in the window area of ​​the super note.

[0220] S810. The mobile phone-side super keyboard and mouse service transmits the first image to the PC-side super keyboard and mouse service.

[0221] S811. After the first image is successfully transmitted, the PC-side super keyboard and mouse service caches the first image.

[0222] The PC-side super keyboard and mouse service caches the first image to a temporary path (or designated path) on the PC side. In this embodiment, S811 is an optional step. In other embodiments, the first image may fail to be transmitted.

[0223] S812. The PC-side super keyboard and mouse service sends a third notification message to the PC-side super note.

[0224] In one example, the PC-side super keyboard and mouse service sends a third notification message to the PC-side super note through the callback interface, where the third notification message is used to indicate whether the first picture dragged from the mobile phone is successfully transferred. If the first picture is successfully transferred, the third notification message may include the picture path of the first picture (i.e., the temporary path where the PC side stores the dragged picture, such as D:\HONOR Shard\PhoneMirror\”), picture name, etc.

[0225] In some embodiments, while executing S809, the following may also be executed:

[0226] S813. The PC-side super keyboard and mouse service sends a second notification message to the PC-side super note.

[0227] In one example, the PC-side super keyboard and mouse service sends a second notification message to the PC-side super note through a callback interface (callback end interface), wherein the second notification message is used to indicate that the PC-side super keyboard and mouse service detects a drag release event.

[0228] S814. The PC-side super note determines whether the first mark bit is the first value or the second value.

[0229] In one implementation of S814, if the PC-side super note determines that the first flag bit is the first value (true), execute:

[0230] S815a. Start a timer. The duration of the timer can be set to, for example, 30 seconds, which is not specifically limited in this embodiment.

[0231] In some embodiments, before the timer times out, the PC-side super note receives a third notification message (S812) sent by the PC-side super keyboard and mouse service, and can execute:

[0232] S816. The PC-side super note modifies the first mark bit, or modifies the first mark bit and the second mark bit based on the third notification message.

[0233] In one example, the third notification message is used to indicate that the transmission of the first image dragged from the mobile phone has failed, and the PC-side super note only modifies the first flag bit. Exemplarily, based on the first flag bit flag=true of S808, in this step, the modified first flag bit can be recorded as flag=false, which is used to indicate that the drag event of the PC-side Google kernel is not blocked, so as not to affect the subsequent image dragging process of the PC-side super note itself. For example, if you drag a picture on the PC Windows desktop to the super note, the PC-side super note can obtain and display the picture from the PC-side Google kernel according to the Windows drag picture process. The second flag bit remains unchanged, and is still dataflag=flase, which is used to indicate that the transmission of the first image dragged across devices has failed, and no picture is displayed in the PC-side super note.

[0234] In another example, the third notification message is used to indicate that the first image dragged from the mobile phone has been successfully transmitted, and then two flag bits need to be modified at the same time. Exemplarily, based on the first flag bit flag=true of S808, in this step, the modified first flag bit can be recorded as flag=false, which is used to indicate that the drag event of the PC-side Google kernel is not blocked, so that it will not affect the subsequent image dragging process of the PC-side super note itself. The PC-side super note can obtain and display the image from the PC-side Google kernel according to the windows drag image process. Based on the second flag bit dataflag=false of S802a, in this step, the modified second flag bit can be recorded as dataflag=true, which is used to indicate that the first image dragged across devices has been successfully transmitted, so that the PC-side super note can further read the image path of the first image from the third notification message to obtain and display the first image dragged across devices.

[0235] S817. The PC-side super note determines whether the second mark bit is the first value or the second value.

[0236] If the PC-side supernote determines that the second flag is the second value (false), then end this process. Alternatively, if the PC-side supernote determines that the second flag is the first value (true), execute:

[0237] S818. The PC-side super note obtains and displays the first picture based on the third notification message.

[0238] If the third notification message is used to indicate that the first picture dragged from the mobile phone is successfully transmitted, the third notification message includes the picture path and picture name of the first picture. The PC-side super note can obtain the first picture based on the picture path and picture name in the third notification message, and display the first picture at the drag release position in the super note window area.

[0239] S819. The PC-side super note modifies the second flag bit. In this step, the PC-side super note modifies the second flag bit from the first value to the second value. Exemplarily, the PC-side super note modifies the second flag bit dataflag=true to dataflag=false, thereby restoring the initial value of the second flag bit (S802a), so that after dragging and transferring the picture across devices, the value of the second flag bit can be modified or maintained according to the transmission status of the picture.

[0240] In another implementation of S814, if the PC-side super note determines that the first mark bit is the second value (false), execute:

[0241] S815b. The PC-side Super Note executes the process of dragging and dropping pictures in windows. The process of dragging and dropping pictures in windows can be referred to Figure 6 The illustrated embodiment mainly includes: when the PC-side super note receives a drag-and-release event notified by the PC-side Google kernel, it can obtain a picture from the PC-side Google kernel and display the picture in the window area of ​​the PC-side super note.

[0242] In some embodiments, when the timer times out and the PC-side super note does not receive the third notification message sent by the PC-side super keyboard and mouse service, the following can be executed:

[0243] S820. The PC-side super note modifies the first mark position and ends this process.

[0244] In this step, the PC-side super note modifies the first flag bit from the first value to the second value, and no longer receives the third notification message from the PC-side super keyboard and mouse service. Exemplarily, the first flag bit flag=true is modified to flag=false.

[0245] In the display method of dragging images across devices shown in this embodiment, the PC-side super note registers the PC-side super keyboard and mouse service so as to obtain the drag event, drag release event, etc. of the PC-side from the PC-side super keyboard and mouse service. Specifically, the PC-side super keyboard and mouse service can notify the PC-side super note of the drag event, drag release event, etc. through the interface (callback interface) registered by the PC-side super note. In addition, the PC-side super note also sets two flags, namely the first flag and the second flag, to know whether it is necessary to shield the drag event of the PC-side Google kernel and whether the cross-device drag file is successful. In this way, the PC-side super note can determine whether to obtain the image from the PC-side Google kernel (the case where the PC-side Google kernel is not shielded) or to obtain the image from the image path indicated by the PC-side super keyboard and mouse service (the case where the PC-side Google kernel is shielded) based on the first flag; the PC-side super note can determine whether it is necessary to obtain and display the image from the PC-side super keyboard and mouse service based on the second flag. If the image transmission is successful, the image needs to be obtained and displayed. If the image transmission fails, the window area of ​​the PC-side super note does not display the image.

[0246] compared to Figure 6 In the scheme shown, since the PC-side super note can set the first flag bit and the second flag bit based on the notification sent by the PC-side super keyboard and mouse service, when the drag release event is triggered, the PC-side super note will block the PC-side Google kernel (i.e., will not obtain images from the PC-side Google kernel) based on the first flag bit (flag=true), wait for the cross-device dragged image to be transmitted to the PC-side super keyboard and mouse, and obtain the cross-device dragged image based on the image path indicated by the PC-side super keyboard and mouse service. Therefore, Figure 8 The solution shown will not cause the problem of images not being displayed when dragged across devices. In addition, since the PC-side Super Keyboard and Mouse Service will send the image path of this drag after each cross-device drag and drop, the image obtained by the PC-side Super Note will not be the image dragged last time. Figure 8 The solution shown will not cause the problem of misaligned display of dragged images. Figure 7 The scheme shown, Figure 8 The solution shown does not involve a file reading operation and does not require polling to determine whether there is a second file ( S710 ), so the PC runs more efficiently.

[0247] Based on the above embodiments, the present application also provides a method for displaying a cross-device image drag, which is applied to a first electronic device installed with a first operating system, the first electronic device is connected to a second electronic device installed with a second operating system, and the first operating system is different from the second operating system, such as Fig. 9 As shown, the method includes:

[0248] S901. A first window of a first application and a second window of a second application are displayed on a display screen of a first electronic device.

[0249] The first application is a global wide area network web-type application, and the second application is a non-web-type application.

[0250] S902. When the first picture is dragged from the display screen of the second electronic device to the first window, in response to the user's drag-and-release operation on the first picture, the first electronic device displays the first picture in the first window.

[0251] S903. When the second picture is dragged from the display screen of the second electronic device to the second window, in response to the user's drag-and-release operation on the second picture, the first electronic device displays the second picture in the second window.

[0252] The execution order of S902 and S903 is not limited.

[0253] For example, Figure 1 As shown, the first electronic device is a PC, the operating system of the PC may be a Windows system, the second electronic device is a mobile phone, the operating system of the mobile phone may be an Android system. The first application may be a web-type application such as a super note, a browser, etc., and the second application may be a non-web-type application such as a document word, a slide PPT, etc.

[0254] In this embodiment, similar to the effect of dragging and displaying pictures across devices of non-web type applications, when the user drags the picture across devices to the window of the web type application, the picture can be displayed normally, and there will be no problem of the picture not being displayed or the picture that was previously dragged across devices being displayed.

[0255] In an optional embodiment, before the first electronic device displays the first picture in the first window, the method also includes: the first application determines whether to block the drag event of the Google kernel of the first electronic device by querying a preset object; the first electronic device displays the first picture in the first window, including: when it is determined to block the drag event of the Google kernel, the first application obtains the first picture from the picture path indicated by the super keyboard and mouse service of the first electronic device, and does not obtain the picture from the Google kernel; the first application displays the first picture in the first window.

[0256] In this embodiment, the preset object may be a file (eg, a first file described below) or a mark bit (eg, a first mark bit described below).

[0257] It should be noted that shielding the drag event of the Google kernel of the first electronic device refers to shielding the drag event of the Google kernel once, that is, the first application will not obtain the image from the Google kernel of the first electronic device in this drag event.

[0258] In this embodiment, the web-type application can determine whether to block the drag event of the Google kernel this time by querying the existence of the preset object or the value of the preset object. If it is determined to block the drag event of the Google kernel, the web-type application obtains the cross-device dragged picture from the super keyboard and mouse service of the first electronic device instead of obtaining the picture from the Google kernel. Since the picture path indicated by the super keyboard and mouse service of the first electronic device is the picture path of the picture dragged across devices this time, if the picture is successfully transmitted, there will be no problem of failing to obtain the picture or obtaining the picture of the previous cross-device drag, so the picture can be displayed normally in the window area of ​​the web-type application.

[0259] In an optional embodiment, the preset object is a first file; the first application determines whether to block the drag event of the Google kernel of the first electronic device by querying the preset object, including: the first application queries whether there is a first file; when the first file is queried, the first application blocks the drag event of the Google kernel of the first electronic device.

[0260] In this embodiment, the first application determines whether it needs to block the Google kernel this time by querying whether the first file exists. If the Google kernel is blocked once, the first application will not get the image from the Google kernel this time.

[0261] In an optional embodiment, before the first application obtains the first image from the image path indicated by the super keyboard and mouse service, the above method also includes: the first application queries whether there is a second file; the second file is a file created by the super keyboard and mouse service, used to indicate whether the first image is successfully transmitted and the image path of the successfully transmitted first image; when the second file is queried, the first application obtains the image path from the second file.

[0262] In this embodiment, when the first application queries the first file, the first application can further query the second file. If the first image is successfully transmitted, the first application can obtain and display the first image through the image path in the second file.

[0263] In an optional embodiment, the first application queries whether there is a second file, including: in response to the user dragging and releasing the first image, the first application queries whether there is a first file; when the first file is queried, the first application queries whether there is a second file according to a preset query frequency.

[0264] In this embodiment, the preset query frequency may be once every 2 seconds. It should be understood that the cross-device transmission of the first image takes time. After the image transmission is completed (may be successful or fail), the super keyboard and mouse service of the first electronic device creates the second file. Before this, the first application cannot query the second file, so the first application needs to query multiple times. In some embodiments, when the first file is not queried, the first application obtains the image from the Google kernel of the first electronic device.

[0265] In this embodiment, when the first application queries the first file, the first application can query the second file according to a preset query frequency to obtain the image path of the first image that is successfully transmitted.

[0266] In an optional embodiment, the above method also includes: when the first image is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service of the first electronic device creates a first file; the first file is used to indicate that the super keyboard and mouse service is dragging the image across devices.

[0267] In this embodiment, when the super keyboard and mouse service of the first electronic device detects the cross-device dragging of the image, it creates a first file to indicate that the dragging is a cross-device dragging.

[0268] In an optional embodiment, the above method also includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service queries whether there is a third file; the third file is a file created by the super keyboard and mouse service when the third picture was dragged across devices the last time, and is used to indicate whether the third picture was successfully transmitted and the picture path of the successfully transmitted third picture; if the third file is queried, the super keyboard and mouse service deletes the third file.

[0269] In this embodiment, after the super keyboard and mouse service of the first electronic device creates the first file for this cross-device dragging, it needs to delete the third file used to indicate the image transmission status of the previous cross-device dragging, so as to avoid the situation where, after creating the second file for indicating the image transmission status of this cross-device dragging, the first application cannot determine whether to obtain the image display from the third file or from the image path indicated by the second file, that is, to avoid file confusion.

[0270] In an optional embodiment, the above method also includes: in response to the user's drag and release operation on the first picture, the super keyboard and mouse service notifies the super keyboard and mouse service of the second electronic device to start transmitting the first picture; after the first picture transmission is completed, the super keyboard and mouse service deletes the first file and creates a second file.

[0271] In this embodiment, if the first file is not deleted after the first image is transferred, the first application will think that the super keyboard and mouse service of the first electronic device has been dragging the image, so the first file is deleted to ensure that the local image dragging process on the first electronic device can run normally. A second file is created to indicate the image position of the first image dragged across devices by the first application.

[0272] In an optional embodiment, the preset object is a first flag; the first application determines whether to block the drag event of the Google kernel of the first electronic device by querying the preset object, including: the first application queries the first flag; when the first flag is a first value, the first application blocks the drag event of the Google kernel; or when the first flag is a second value, the first application does not block the drag event of the Google kernel; when the drag event of the Google kernel is not blocked, the first application obtains the image from the Google kernel. Exemplarily, the first flag can be recorded as flag, the first value is true, and the second value is false.

[0273] In this embodiment, the first application queries the value of the first flag bit to determine whether the Google kernel needs to be blocked once. If the Google kernel is blocked once, the first application will not get the image from the Google kernel this time.

[0274] In an optional embodiment, before the first application obtains the first image from the image path indicated by the super keyboard and mouse service, the above method further includes: the first application queries a second flag bit, the second flag bit is used to indicate whether the first image is successfully transmitted; when the second flag bit indicates that the first image is successfully transmitted, the first application obtains the first image from the super keyboard and mouse service. In this embodiment, the first application obtains the image path of the successfully transmitted first image from the third notification message sent by the super keyboard and mouse service.

[0275] In this embodiment, when the first application determines that the first mark bit is the first value, the first application can determine whether the first image dragged this time is successfully transmitted by querying the value of the second mark bit. If the transmission is successful, the image path can be obtained from the message sent by the super keyboard and mouse service of the first electronic device (i.e., the third notification message).

[0276] In an optional embodiment, the method further includes: after the first application is started, the first application subscribes to the super keyboard and mouse service for related events of cross-device dragging, so that when the related events are triggered, the first application obtains data of the related events from the super keyboard and mouse service; the related events of cross-device dragging include at least one of the following: the image is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device; the image is dragged from the display screen of the second electronic device to the first window. In this embodiment, the first application obtains data of the related events from the super keyboard and mouse service of the first electronic device, including but not limited to the identifier of the related event, the coordinate data of the cursor, the thumbnail of the image dragged across devices, the image path (if the image is successfully transferred), etc.

[0277] In this embodiment, the first application registers events with the super keyboard and mouse service of the first electronic device so that when related events of cross-device dragging are triggered, the first application responds to related events and makes corresponding decisions, such as determining whether to block the Google kernel.

[0278] In an optional embodiment, the method further includes: after the first application is started, the first application initializes a first flag bit and a second flag bit; the first flag bit is used to indicate whether to block the drag event of the Google kernel of the first electronic device, and the second flag bit is used to indicate whether the cross-device dragged picture is successfully transmitted. Exemplarily, the initial first flag bit can be recorded as flag=false, and the initial second flag bit can be recorded as dataflag=false.

[0279] In this embodiment, the first application creates two flags to respectively record whether the drag event of the Google kernel is currently blocked and the current transmission status of the dragged image across devices. The first application queries these two flags to determine subsequent execution actions.

[0280] In an optional embodiment, the method further includes: when the first image is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service of the first electronic device sends a first notification message to the first application; the first application modifies the first flag bit based on the first notification message, and the modified first flag bit is used to indicate the shielding of the drag event of the Google kernel of the first electronic device. In this embodiment, the first notification message is used to indicate that an event of dragging an image across devices occurs at the edge of the screen of the first electronic device.

[0281] In this embodiment, the first application modifies the first flag bit based on the first notification message to record that the drag event of the Google kernel needs to be shielded once, that is, the image is not obtained from the Google kernel this time.

[0282] In an optional embodiment, the method further includes: in response to the user's drag-and-release operation on the first image, the super keyboard and mouse service of the first electronic device sends a second notification message to the first application; the first application queries the first flag bit in response to the second notification message; when the first flag bit indicates that the drag event of the Google kernel of the first electronic device is blocked, the first application starts a timer so as to receive a third notification message from the super keyboard and mouse service within the timer duration, and the third notification message is used to indicate whether the first image is successfully transmitted. In this embodiment, the second notification message is used to indicate that the first image dragged across devices is dragged to the window of the first application and released.

[0283] In this embodiment, after receiving the second notification message, the first application queries the first flag to determine whether the image released by dragging is from the first electronic device or from another device (such as the second electronic device). After determining that it is a cross-device dragged image, a timer is started to wait for the transmission of the cross-device dragged image.

[0284] In an optional embodiment, the method further includes: before the timer times out, the first application receives a third notification message; if the third notification message is used to indicate that the first image transmission fails, the first application modifies the first flag bit; or, if the third notification message is used to indicate that the first image transmission is successful, the first application modifies the first flag bit and the second flag bit. In this embodiment, the modified first flag bit is used to indicate that the drag event of the Google kernel of the first electronic device is not blocked, which can be recorded as flag = false. The modified second flag bit is restored to the initial value (i.e., the second value), which can be recorded as dataflag = false.

[0285] In this embodiment, before the timer times out, the third notification message received by the first application may indicate two situations. The first application modifies at least one of the two flag bits based on different situations to update the flag bit value based on the picture transmission situation.

[0286] In an optional embodiment, the above method further includes: when the timer times out and the first application does not receive the third notification message, the first application modifies the first mark bit.

[0287] In this embodiment, the first application modifies the first flag bit from the first value (true) to the second value (false), and no longer receives the third notification message from the super keyboard and mouse service of the first electronic device.

[0288] In an optional embodiment, the method further includes: after the first electronic device is turned on, starting a super keyboard and mouse service of the first electronic device.

[0289] In this embodiment, the super keyboard and mouse service of the first electronic device may be used to detect, for example, a cross-device drag event, a drag release event, and the like.

[0290] In an optional embodiment, the first electronic device or the second electronic device is connected to an input device, and the above method also includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the first electronic device displays a thumbnail of the first picture, and a cursor is displayed on the thumbnail of the first picture; in response to the drag operation applied to the input device, during the process of the first picture being dragged from the edge of the display screen of the first electronic device to the first window, the first electronic device displays an animation of the thumbnail of the first picture moving with the cursor on the display screen of the first electronic device.

[0291] In this embodiment, when the second electronic device detects that the first image reaches the edge of the display screen of the second electronic device, the second electronic device sends drag data to the first electronic device, and the drag data includes, for example, a thumbnail of the first image, cursor coordinate data, etc.

[0292] It should be noted that the embodiments of the present application do not particularly limit the specific structure of the execution subject of the method for displaying a cross-device image drag, as long as the code storing the method for displaying a cross-device image drag provided by the embodiments of the present application can be run to perform processing according to the method for displaying a cross-device image drag provided by the embodiments of the present application. For example, the execution subject of the method for displaying a cross-device image drag provided by the embodiments of the present application may be a functional module in an electronic device that can call and execute a program, or a processing device applied to an electronic device, for example, the processing device is a chip.

[0293] In the above embodiments, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0294] Therefore, the modules of each example described in the embodiments of the present application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in the form of hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0295] The present application also provides an electronic device, which may correspond to the first electronic device in the above embodiment. Fig.10As shown, the first electronic device 1000 includes: a memory 1001 and at least one processor ( Fig.10 A processor 1002 is shown); the memory 1001 is used to store computer program code, and the computer program code includes instructions; when the at least one processor 1002 executes the instructions, the first electronic device executes the technical solution of the PC in any of the aforementioned embodiments, and its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, which will not be repeated here.

[0296] The memory may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0297] The memory can be independent and connected to the processor through a communication line, or it can be integrated with the processor.

[0298] The processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0299] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program runs on an electronic device, the electronic device executes the technical solution of any of the above embodiments. The implementation principle and technical effect are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0300] An embodiment of the present application provides a chip, which includes a processor. The processor is used to call a computer program in a memory to execute the technical solution in any of the above embodiments. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0301] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the technical solution in any of the above embodiments. The implementation principle and technical effect are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0302] The present application embodiment provides a cross-device dragging system, referring to Figure 4 The system includes: a first electronic device and a second electronic device, the first electronic device is connected to the second electronic device; the first electronic device executes the technical solution of the PC in any of the aforementioned embodiments, and its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, which will not be repeated here.

[0303] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for displaying a cross-device image by dragging, characterized in that: Applied to a first electronic device having a first operating system installed thereon, the first electronic device being connected to a second electronic device having a second operating system installed thereon, the first operating system being different from the second operating system, the method comprising: The display screen of the first electronic device displays a first window of a first application and a second window of a second application, the first application is a global wide area network web-type application, and the second application is a non-web-type application; When the first picture is dragged from the display screen of the second electronic device to the first window, in response to a user's drag-and-release operation on the first picture, the first electronic device displays the first picture in the first window; When the second picture is dragged from the display screen of the second electronic device to the second window, in response to the user's drag-and-release operation on the second picture, the first electronic device displays the second picture in the second window.

2. The method according to claim 1, characterized in that: Before the first electronic device displays the first picture in the first window, the method further includes: the first application determines whether to shield the drag event of the Google kernel of the first electronic device by querying a preset object; The first electronic device displays the first picture in the first window, comprising: when it is determined to shield the drag event of the Google kernel, the first application obtains the first picture from the picture path indicated by the super keyboard and mouse service of the first electronic device, and does not obtain the picture from the Google kernel; The first application displays the first picture in the first window.

3. The method according to claim 2, characterized in that The preset object is a first file; the first application determines whether to shield the drag event of the Google kernel of the first electronic device by querying the preset object, including: The first application queries whether there is a first file; When the first file is found, the first application blocks the drag event of the Google kernel of the first electronic device.

4. The method according to claim 2 or 3, characterized in that: Before the first application obtains the first picture from the picture path indicated by the super keyboard and mouse service, the method further includes: The first application queries whether there is a second file; the second file is a file created by the super keyboard and mouse service, and is used to indicate whether the first picture is successfully transmitted and the picture path of the first picture that is successfully transmitted; When the second file is found, the first application obtains the image path from the second file.

5. The method according to claim 4, characterized in that The first application queries whether there is a second file, including: In response to a user dragging and releasing operation on the first picture, the first application queries whether there is a first file; When the first file is found, the first application queries whether the second file exists according to a preset query frequency.

6. The method according to any one of claims 1 to 5, characterized in that: The method further includes: when the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service of the first electronic device creates a first file; The first file is used to indicate that the super keyboard and mouse service is performing cross-device image dragging.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service queries whether there is a third file; The third file is a file created by the super keyboard and mouse service when the third picture is dragged across devices most recently, and is used to indicate whether the third picture is successfully transmitted and the picture path of the successfully transmitted third picture; If the third file is found, the super keyboard and mouse service deletes the third file.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: In response to the user's drag-and-release operation on the first picture, the super keyboard and mouse service notifies the super keyboard and mouse service of the second electronic device to start transmitting the first picture; After the first picture is transmitted, the super keyboard and mouse service deletes the first file and creates a second file.

9. The method according to claim 2, characterized in that: The preset object is a first mark; the first application determines whether to shield the drag event of the Google kernel of the first electronic device by querying the preset object, including: The first application queries the first flag bit; When the first flag bit is a first value, the first application shields the drag event of the Google kernel; or When the first flag bit is a second value, the first application does not block the drag event of the Google kernel; when the drag event of the Google kernel is not blocked, the first application obtains the image from the Google kernel.

10. The method according to claim 2, characterized in that Before the first application obtains the first picture from the picture path indicated by the super keyboard and mouse service, the method further includes: The first application queries a second flag bit, where the second flag bit is used to indicate whether the first picture is successfully transmitted; When the second flag bit indicates that the first picture is transmitted successfully, the first application obtains the first picture from the super keyboard and mouse service.

11. The method according to any one of claims 1, 2, 9 or 10, characterized in that: The method further comprises: After the first application is started, the first application subscribes to the super keyboard and mouse service of the first electronic device for related events of cross-device dragging, so that when the related event is triggered, the first application obtains data of the related event from the super keyboard and mouse service; The cross-device drag related events include at least one of the following: The picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device; The picture is dragged from the display screen of the second electronic device to the first window.

12. The method according to any one of claims 1, 2, 9 to 11, characterized in that: The method further comprises: After the first application is started, the first application initializes a first mark bit and a second mark bit; The first flag bit is used to indicate whether to shield the drag event of the Google kernel of the first electronic device, and the second flag bit is used to indicate whether the image dragged across devices is successfully transmitted.

13. The method according to any one of claims 1, 2, 9 to 12, characterized in that: The method further comprises: When the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the super keyboard and mouse service of the first electronic device sends a first notification message to the first application; The first application modifies a first flag bit based on the first notification message, and the modified first flag bit is used to indicate shielding of a drag event of the Google kernel of the first electronic device.

14. The method according to any one of claims 1, 2, 9 to 13, characterized in that: The method further comprises: In response to the user's drag-and-release operation on the first picture, the super keyboard and mouse service of the first electronic device sends a second notification message to the first application; The first application queries a first flag bit in response to the second notification message; When the first flag bit indicates blocking the drag event of the Google kernel of the first electronic device, the first application starts a timer to receive a third notification message from the super keyboard and mouse service within the duration of the timer, and the third notification message is used to indicate whether the first picture is transmitted successfully.

15. The method according to claim 14, characterized in that The method further comprises: Before the timer times out, the first application receives the third notification message; If the third notification message is used to indicate that the first picture transmission fails, the first application modifies the first mark bit; or if the third notification message is used to indicate that the first picture transmission is successful, the first application modifies the first mark bit and the second mark bit.

16. The method according to claim 14, characterized in that The method further includes: when the timer times out and the first application does not receive the third notification message, the first application modifies the first marking bit.

17. The method according to any one of claims 1 to 16, characterized in that The method further comprises: After the first electronic device is turned on, the super keyboard and mouse service of the first electronic device is started.

18. The method according to any one of claims 1 to 17, characterized in that The first electronic device or the second electronic device is connected to an input device, and the method further includes: When the first picture is dragged from the display screen of the second electronic device to the edge of the display screen of the first electronic device, the first electronic device displays a thumbnail of the first picture, and a cursor is displayed on the thumbnail of the first picture; In response to a drag operation applied to the input device, while the first image is dragged from the edge of the display screen of the first electronic device to the first window, the first electronic device displays an animation of a thumbnail of the first image on the display screen of the first electronic device as the cursor moves.

19. An electronic device, characterized in that: The electronic device is the first electronic device according to any one of claims 1 to 18, wherein the first electronic device comprises: a memory and at least one processor; The memory is used to store computer program codes, and the computer program codes include instructions; when the at least one processor executes the instructions, the first electronic device executes the method according to any one of claims 1 to 18.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by an electronic device, the method according to any one of claims 1 to 18 is implemented.

21. A cross-device drag system, characterized in that: include: A first electronic device and a second electronic device, wherein the first electronic device is connected to the second electronic device; and the first electronic device executes the method according to any one of claims 1 to 18.

Images