Multi-screen interaction method and device

Abstract

This application provides a multi-screen interaction method and apparatus. The method includes: acquiring content to be displayed, including content corresponding to a notification message or a request message triggered by a user operation; displaying the content to be displayed on a first screen through a new window; receiving first gaze point information from an eye tracker, the first gaze point information being obtained by the eye tracker based on the user's gaze information; determining, based on the first gaze point information, that the screen the user is looking at is a second screen; and moving the new window from the first screen to the second screen. The method of this application, by moving the new window across screens based on the first gaze point information, can reduce the number of long-distance mouse movements, thereby improving operational efficiency.

Description

Technical Field

[0001] This application relates to the field of screen display technology, and in particular to a multi-screen interaction method and apparatus. Background Technology

[0002] With the rapid development of modern computers, computer hardware has increasingly stronger computing power, and software has increasingly richer functions. As a result, more and more people are connecting their computers to multiple screens to observe and operate content on multiple screens simultaneously.

[0003] Currently, when performing cross-screen operations, long mouse movements are required to move the mouse cursor to the target area. Frequent cross-screen operations require repeated long mouse movements, resulting in low operational efficiency. Summary of the Invention

[0004] This application provides a multi-screen interaction method and apparatus to solve the problem of low operation efficiency caused by repeatedly moving the mouse a long distance when frequently performing cross-screen operations.

[0005] In a first aspect, this application provides a multi-screen interaction method applied to an electronic device, the electronic device excluding a first screen and a second screen, comprising:

[0006] Get the content to be displayed, which includes the content corresponding to notification messages or request messages triggered by user actions.

[0007] The content to be displayed is shown in a new window on the first screen.

[0008] Receive first gaze point information from the eye tracker, which is obtained by the eye tracker based on the user's gaze information.

[0009] Based on the first gaze point information, the screen that the user is looking at is determined to be the second screen.

[0010] Move the newly added window from the first screen to the second screen.

[0011] Optionally, the newly added window can be moved from the first screen to the second screen, including:

[0012] When a new window is to be displayed in a fixed position, move the new window to the same position on the second screen as it was when displayed on the first screen. Or

[0013] When the newly added window is a movable window, based on the first gaze point information, the newly added window is moved from the first screen to a first preset area of ​​the second screen. The first preset area is the area within the first preset range where the first gaze point corresponding to the first gaze point information is located.

[0014] Optionally, the multi-screen interaction method also includes:

[0015] Receive second fixation information from the eye tracker.

[0016] When the first preset condition is met, the mouse cursor or mouse arrow on the first screen is moved to the position of the second gaze point corresponding to the second gaze point information on the second screen, based on the second gaze point information.

[0017] Among them, satisfying the first preset condition includes: the gaze duration corresponding to the second gaze point information is greater than the second preset duration, or, acquiring mouse-related events, wherein the mouse-related events are events triggered by user operation that move the mouse cursor or mouse arrow.

[0018] Optionally, the multi-screen interaction method also includes:

[0019] Receive third fixation information from the eye tracker.

[0020] When the second preset condition is met, a sub-screen is created in the second preset area of ​​the second screen according to the third gaze point information, and the target area in the first screen is moved into the sub-screen.

[0021] The target area is the area in the first screen where the user operates within a first preset time before the second screen receives the third gaze point information; after an object in the target area is moved to a sub-screen, its original interactive functions are retained.

[0022] The second preset region is the region within the second preset range where the third gaze point information is located.

[0023] The second preset condition includes: within a first preset time before the second screen receives the third gaze point information, the user performs an operation on the first screen, or the user's preset operation is received.

[0024] Secondly, this application provides a multi-screen interaction method applied to an eye tracker, comprising:

[0025] Obtain the user's gaze information.

[0026] Based on the gaze information, the first fixation point information is obtained.

[0027] Sending first gaze information to the electronic device, which is used by the electronic device to determine that the screen the user is looking at is the second screen based on the first gaze information, and moving the new window from the first screen to the second screen. The first screen is the screen through which the content to be displayed is displayed by the new window. The content to be displayed includes the content corresponding to a notification message or a request message triggered by the user's operation.

[0028] Optionally, the eye tracker can communicate with multiple screens.

[0029] Based on the gaze information, the first fixation point information is calculated, including:

[0030] Calculate the pose data for each screen to obtain the 3D position and orientation of each screen.

[0031] Based on gaze information, 3D position, and screen orientation, the user's gaze point is determined, and the first gaze point message is obtained.

[0032] Optionally, the user's gaze point can be determined based on gaze information, 3D position, and screen orientation, including:

[0033] Based on gaze information, 3D position, and screen orientation, determine the number of intersections between the user's gaze direction and multiple screens.

[0034] If the number of intersections is 1, then that intersection is a fixation point.

[0035] If the number of intersection points is greater than 1, then the intersection point between the line of sight and the screen closest to the user is the fixation point.

[0036] Optionally, the eye tracker communicates with a screen.

[0037] Based on the gaze information, the first fixation point information is calculated, including:

[0038] Based on the gaze information, determine whether the user's gaze direction intersects with the corresponding screen.

[0039] If so, calculate the pixel coordinates of the intersection point on the screen.

[0040] Based on pixel coordinates, the user's gaze point is determined, and the first gaze point message is obtained.

[0041] Thirdly, this application provides a multi-screen interaction device for use in electronic devices, comprising:

[0042] The acquisition module is used to acquire the content to be displayed, which includes the content corresponding to notification messages or request messages triggered by user operations.

[0043] The display module is used to display content to be displayed on the first screen by adding a new window.

[0044] The receiving module is used to receive the first gaze point information from the eye tracker, which is obtained by the eye tracker based on the user's gaze information.

[0045] The determination module is used to determine the screen that the user is looking at as the second screen based on the first gaze point information.

[0046] The move module is used to move newly added windows from the first screen to the second screen.

[0047] Optionally, this mobile module is specifically used for:

[0048] When a new window is to be displayed in a fixed position, move the new window to the same position on the second screen as it was when displayed on the first screen. Or

[0049] When the newly added window is a movable window, based on the first gaze point information, the newly added window is moved from the first screen to a first preset area of ​​the second screen. The first preset area is the area within the first preset range where the first gaze point corresponding to the first gaze point information is located.

[0050] Optionally, the receiving module is also used to receive second gaze point information from the eye tracker.

[0051] The moving module is also used to move the mouse cursor or mouse arrow on the first screen to the location of the second gaze point corresponding to the second gaze point information on the second screen when the first preset condition is met, based on the second gaze point information.

[0052] Among them, satisfying the first preset condition includes: the gaze duration corresponding to the second gaze point information is greater than the second preset duration, or, acquiring mouse-related events, wherein the mouse-related events are events triggered by user operation that move the mouse cursor or mouse arrow.

[0053] Optionally, the receiving module is also used to receive third gaze point information from the eye tracker.

[0054] The moving module is further configured to create a sub-screen within a second preset area in the second screen based on the third gaze point information when the second preset condition is met, and move the target area in the first screen into the sub-screen.

[0055] The target area is the area in the first screen where the user operates within a first preset time before the second screen receives the third gaze point information. After an object in the target area is moved to the sub-screen, its original interactive functions are retained.

[0056] The second preset region is the region within the second preset range where the third gaze point information is located.

[0057] The second preset condition includes: within a first preset time before the second screen receives the third gaze point information, the user performs an operation on the first screen, or the user's preset operation is received.

[0058] Fourthly, this application provides a multi-screen interaction device for use in an eye tracker, comprising:

[0059] The acquisition module is used to acquire the user's gaze information.

[0060] The determination module is used to obtain the first gaze point information based on the gaze information.

[0061] The sending module is used to send first gaze point information to the electronic device. Based on the first gaze point information, the electronic device determines that the screen the user is looking at is the second screen and moves the newly added window from the first screen to the second screen. The first screen is the screen through which the content to be displayed is displayed by the newly added window. The content to be displayed includes the content corresponding to a notification message or a request message triggered by user operation.

[0062] Optionally, the eye tracker can communicate with multiple screens.

[0063] This determining module is specifically used for:

[0064] Calculate the pose data for each screen to obtain the 3D position and orientation of each screen.

[0065] Based on gaze information, 3D position, and screen orientation, the user's gaze point is determined, and the first gaze point message is obtained.

[0066] Optionally, this determining module is specifically used for:

[0067] Based on gaze information, 3D position, and screen orientation, determine the number of intersections between the user's gaze direction and multiple screens.

[0068] If the number of intersections is 1, then that intersection is a fixation point.

[0069] If the number of intersection points is greater than 1, then the intersection point between the line of sight and the screen closest to the user is the fixation point.

[0070] Optionally, the eye tracker communicates with a screen.

[0071] This determining module is further used for:

[0072] Based on the gaze information, determine whether the user's gaze direction intersects with the corresponding screen.

[0073] If so, calculate the pixel coordinates of the intersection point on the screen.

[0074] Based on pixel coordinates, the user's gaze point is determined, and the first gaze point message is obtained.

[0075] Fifthly, this application provides an electronic device, including: a memory and a processor;

[0076] Memory is used to store computer programs.

[0077] The processor is configured to read a computer program stored in a memory and execute the multi-screen interaction method of the first aspect described above, or to execute the multi-screen interaction method of the second aspect described above.

[0078] Sixthly, this application provides a readable storage medium having a computer program stored thereon, the computer program storing computer execution instructions, which, when executed by a processor, are used to implement the multi-screen interaction method as described in the first aspect above, or to implement the multi-screen interaction method as described in the second aspect above.

[0079] In a seventh aspect, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the multi-screen interaction method of the first aspect described above, or is used to implement the multi-screen interaction method of the second aspect described above.

[0080] The multi-screen interaction method and apparatus provided in this application acquire content to be displayed, including content corresponding to notification messages or request messages triggered by user operations; display the content to be displayed on a first screen through a new window; receive first gaze point information from an eye tracker, the first gaze point information being obtained by the eye tracker based on the user's gaze information; determine the screen the user is looking at as a second screen based on the first gaze point information; and move the new window from the first screen to the second screen. In this application, moving the new window across screens based on the first gaze point information can reduce the number of long-distance mouse movements, thereby improving operational efficiency. Attached Figure Description

[0081] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0082] Figure 1 This is a schematic diagram of a multi-screen interaction scenario provided in an embodiment of this application;

[0083] Figure 2 A flowchart illustrating a multi-screen interaction method provided in an embodiment of this application;

[0084] Figure 3 This application provides a schematic diagram of a cross-screen operation as an embodiment of the present application.

[0085] Figure 4 A flowchart illustrating another multi-screen interaction method provided in an embodiment of this application;

[0086] Figure 5 This is a schematic diagram of the structure of a multi-screen interaction device provided in an embodiment of this application;

[0087] Figure 6 This is a schematic diagram of another multi-screen interaction device provided in an embodiment of this application;

[0088] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0089] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0090] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0091] The technical solutions provided in this application can be applied to multi-screen interaction scenarios, especially when a computer is connected to multiple screens, allowing simultaneous observation and operation of content on multiple screens.

[0092] The existing method involves making long mouse movements to move the mouse cursor to the target area. Figure 1 This is a schematic diagram of a multi-screen scene, such as... Figure 1 As shown, the user needs to simultaneously operate the content on two desktop monitors on their desk and a large monitor. Additionally, if cross-screen cursor movement is required between screens that are not arranged side-by-side, for example… Figure 1 In the context of screen 1 and screen 2, and screen 2 and screen 3, the mouse's cross-screen movement rules cannot directly correspond to the physical positions between screens, failing to meet the precise and fast operation requirements when frequently switching between multiple screens, resulting in low efficiency in cross-screen operations.

[0093] To address the above issues, this application proposes a multi-screen interaction method. The method involves: acquiring content to be displayed, including notification messages or content corresponding to request messages triggered by user actions; displaying the content to be displayed on a first screen through a newly added window; receiving first gaze point information from an eye tracker, which is obtained by the eye tracker based on the user's gaze information; determining the screen the user is looking at as a second screen based on the first gaze point information; and moving the newly added window from the first screen to the second screen. In this application, moving the newly added window across screens based on the first gaze point information reduces the number of long-distance mouse movements, thereby improving operational efficiency.

[0094] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0095] Figure 2 This is a flowchart illustrating a multi-screen interaction method provided in an embodiment of this application. This multi-screen interaction method can be executed by software and / or hardware devices, such as electronic devices like terminals or servers. For example, please refer to [link to example]. Figure 2 As shown, the multi-screen interaction method may include:

[0096] S201. Obtain the content to be displayed.

[0097] The content to be displayed includes notification messages or request messages triggered by user actions. Notification messages may include system notifications, application notifications, advertising messages, etc. The content of request messages triggered by user actions can be understood as the user opening the Start Menu or other windows through shortcuts.

[0098] S202. Display the content to be displayed on the first screen by adding a new window.

[0099] Specifically, after obtaining the content to be displayed, the content is displayed on the first screen by adding a new window.

[0100] S203, Receive the first gaze point information from the eye tracker.

[0101] The first gaze point information is obtained by the eye tracker based on the user's gaze information. When the user gazes at the screen along a certain gaze direction for a period of time exceeding a certain preset value, the user's first gaze point information is obtained. This preset value can be set according to actual conditions or experience, for example, it can be set to 5 seconds or 8 seconds, etc. The specific value of this preset value is not limited in this embodiment of the application.

[0102] S204. Based on the first gaze point information, determine that the screen the user is looking at is the second screen.

[0103] If a user keeps looking at a certain screen for a certain period of time, the time when the gaze point information is received will be all the times within that period of time.

[0104] Specifically, after obtaining the first gaze point information detected by the eye tracker, the screen that the user is looking at can be determined. Based on the first gaze point information, the screen that the user is looking at is then identified as the second screen.

[0105] S205. Move the newly added window from the first screen to the second screen.

[0106] In one possible implementation, when a new window is to be displayed in a fixed position, the new window is moved to the same position on the second screen as when it was displayed on the first screen.

[0107] Specifically, when a new window is displayed in a fixed position, such as a screen notification in the lower right corner or a Start menu in the lower left corner, its relative position on the screen remains unchanged. That is, the new window is moved to the same position on the second screen as it was on the first screen. For example, if the Start menu is displayed in the lower left corner of the first screen, it is moved to the lower left corner of the second screen.

[0108] In another possible implementation, when the newly added window is a movable window, it is moved from the first screen to a first preset area of ​​the second screen based on the first gaze point information. The first preset area is the region within a first preset range where the first gaze point corresponding to the first gaze point information is located.

[0109] Specifically, when the newly added window is a movable window, such as an application pop-up or a window opened via a shortcut key, the newly added window is moved from the first screen to the area within the first preset range of the gaze point of the second screen, based on the first gaze point information.

[0110] In one possible implementation, if a user uses an electronic device for a certain period of time and keeps looking at the second screen, then if a new window pops up on the first screen during that period of time, the new window is moved from the first screen to the second screen.

[0111] For example, if a user uses an electronic device between 13:14:09 and 13:58:15 and is looking at screen A during that time, and at 13:15:19 the electronic device receives a system notification message and displays it in a new window on screen B, then the system notification window will be moved from screen B to screen A.

[0112] In another possible implementation, if the time between receiving the user's gaze information and obtaining the content to be displayed is less than a certain preset time, the newly added window is moved from the first screen to the second screen.

[0113] The preset duration can be set according to actual conditions or experience, such as 3 seconds or 5 seconds. The specific value of the preset duration is not limited in this embodiment.

[0114] For example, if the preset duration is set to 8 seconds, and the electronic device receives a system notification message and displays it as a new window on screen A while the user is not using the device, and then half an hour later the user uses the device to perform file operations and looks at screen B, if the time between the electronic device receiving the user's gaze information and acquiring the content to be displayed is less than 8 seconds, the electronic device will not move the new window. The user can then process the new window on screen A before performing file operations on the electronic device.

[0115] In this embodiment, the newly added window is moved to the second screen, allowing the user to see the new window immediately, reducing the number of times the user switches between screens, alleviating the workload of repeatedly moving the mouse a long distance when switching between multiple screens, and improving the efficiency of multi-screen operation.

[0116] In some embodiments, the method further includes moving the mouse following the user's gaze point, specifically including: receiving second gaze point information from an eye tracker; and, when a first preset condition is met, moving the mouse cursor or mouse arrow on the first screen to the location of the second gaze point corresponding to the second gaze point information on the second screen, based on the user's second gaze point information. The first preset condition includes: the gaze duration corresponding to the second gaze point information is greater than a second preset duration; or, acquiring mouse-related events, where the mouse-related events are events triggered by user operations that move the mouse cursor or mouse arrow.

[0117] It is understandable that the triggering methods for moving the mouse cursor or mouse arrow across the screen include manual triggering and automatic triggering.

[0118] Manual triggering includes events related to moving the mouse cursor or mouse arrow based on user actions. For example, the user can pre-set a keyboard trigger key for moving the mouse cursor or mouse arrow across screens. When the user presses this trigger key, the corresponding cross-screen operation is performed. Automatic triggering can be understood as automatically triggering the cross-screen movement of the mouse cursor or mouse arrow when the user's gaze remains on the same position on the second screen for more than a second preset time.

[0119] Specifically, when the operation of moving the mouse cursor or mouse arrow across screens is triggered, the electronic device will receive the operation request to move the mouse cursor or mouse arrow, and then move the mouse cursor or mouse arrow from the first screen to the location of the second gaze point on the second screen.

[0120] For example, when the second preset duration is set to 2 seconds, if the user gazes at the operation area of ​​the second screen for more than 2 seconds, the mouse cursor will be moved from the first screen to the gaze point in the operation area of ​​the second screen.

[0121] In this embodiment, the mouse cursor or arrow can be moved across the screen based on gaze point information, thereby reducing the number of times the user moves the mouse a long distance between screens and improving operational efficiency. Simultaneously, two different triggering methods are designed, allowing users to choose the appropriate method based on their needs, and an automatic triggering time threshold can be set to ensure stable and accurate operation, preventing the mouse cursor or arrow from arbitrarily switching between screens based on eye movements.

[0122] In some embodiments, such as Figure 3 As shown, the device receives third gaze point information from the eye tracker. When the second preset condition is met, it creates a sub-screen in the second preset area of ​​the second screen based on the user's third gaze point information, and moves the target area in the first screen into the sub-screen.

[0123] In this embodiment, a sub-screen can be understood as creating a small window in a second screen. For example, the small window can be a circular window or a square window. In this embodiment, the shape and size of the small window are not specifically limited.

[0124] The target area is the area in the first screen where the user operates within a first preset time before the second screen receives the third gaze point information. After an object in the target area is moved to the sub-screen, its original interactive functions are retained.

[0125] In this embodiment, the objects within the target area may include document content, files, folders, or web pages, which can appear on the screen of an electronic device. This embodiment does not impose specific limitations on this.

[0126] It is understandable that retaining the original interactive functions means that the state, functions, and attributes of objects in the target area on the first screen are still retained after being moved to a sub-screen on the second screen.

[0127] For example, if the object in the target area is a video application that is playing a video, when the target area containing the video application is moved to a sub-screen in the second screen, the interface displayed by the video application remains unchanged, and the original video is still playing. Only the position has been moved, and the original state, attributes, and functions remain unchanged.

[0128] The second preset region is the region within the second preset range where the third gaze point information is located.

[0129] The second preset condition includes: within a first preset time before the second screen receives the third gaze point information, the user performs an operation on the first screen, or the user's preset operation is received.

[0130] It is understandable that the triggering methods for moving target objects across screens include manual triggering and automatic triggering.

[0131] Manual triggering includes receiving a preset user action, such as a keyboard trigger key for moving a target object across screens. When the user presses this key, the corresponding cross-screen action is performed. Automatic triggering can be understood as automatically triggering the cross-screen movement of the target object if the user performs an action on the first screen within a first preset time before the second screen receives the third gaze point information.

[0132] Specifically, when the operation of moving the target area across screens is triggered, the electronic device will receive the operation request to move the target area and move the target area and the objects within the target area from the first screen to the sub-screen of the second screen.

[0133] For example, if a user is creating a PowerPoint presentation on the second screen while a Word document is displayed on the first screen, and the user needs to copy and paste a portion of content from the Word document into the PowerPoint, they can locate the corresponding content in the Word document and manually or automatically trigger a cross-screen movement operation to move the target area from the first screen to the sub-screen of the second screen. This method eliminates the need for long mouse movements to copy and paste content, and also avoids moving the entire Word document across screens, thus improving the efficiency of cross-screen operations.

[0134] For example, after automatically triggering the operation of moving the mouse cursor or mouse arrow across the screen, the operation of moving the target object across the screen can also be manually triggered within a certain preset time. In this case, the operation of moving the mouse cursor or mouse arrow across the screen can be converted into the operation of moving the target object across the screen.

[0135] In this embodiment, moving the target area across the screen based on gaze point information reduces the number of long-distance mouse movements and improves operational efficiency. Simultaneously, the triggering method can be selected according to user needs, and the automatic triggering time threshold can be flexibly set, thereby ensuring stable and accurate operation and preventing the target area from moving arbitrarily across the screen following the user's gaze point, which could lead to misoperation.

[0136] For example, Figure 3 This is a diagram illustrating a cross-screen operation. For example... Figure 3 As shown, the mouse and target area can move with the gaze point. The circular symbols in the figure represent the result of the mouse or target area moving with the gaze point, while the arrow symbols represent the process of the mouse or target area moving with the gaze point.

[0137] The multi-screen interaction method provided in this application involves: acquiring content to be displayed, including content corresponding to notification messages or request messages triggered by user operations; displaying the content to be displayed on a first screen through a new window; receiving first gaze point information from an eye tracker, which is obtained by the eye tracker based on the user's gaze information; determining, based on the first gaze point information, that the screen the user is looking at is a second screen; and moving the new window from the first screen to the second screen. In this application, moving the new window across screens based on the first gaze point information reduces the number of long-distance mouse movements, thereby improving operational efficiency.

[0138] Figure 4 This is a flowchart illustrating another multi-screen interaction method provided in an embodiment of this application. This multi-screen interaction method can be executed by software and / or hardware devices, such as an eye tracker. For an example, please refer to [link to example]. Figure 4 As shown, the multi-screen interaction method may include:

[0139] S401, Obtain the user's gaze information.

[0140] In this step, gaze information includes gaze direction and gaze duration.

[0141] S402. Based on the gaze information, obtain the first gaze point information.

[0142] Specifically, when a user gazes along a certain line of sight for a duration exceeding a certain preset value, the user's first gaze point information is obtained. This preset value can be set according to actual conditions or experience, for example, it can be set to 5 seconds or 8 seconds, etc. The specific value of this preset value is not limited in this embodiment of the application.

[0143] S403. Send first gaze point information to the electronic device, so that the electronic device can determine the screen that the user is looking at as the second screen based on the first gaze point information, and move the newly added window from the first screen to the second screen.

[0144] The first screen is a screen that displays content to be displayed by adding a new window. The content to be displayed includes the content corresponding to notification messages or request messages triggered by user operations.

[0145] In one possible implementation, the eye tracker is communicatively connected to multiple screens. When the eye tracker calculates the first gaze point information based on the gaze information, it first calculates the pose data of each screen to obtain the 3D position and screen orientation of each screen, and determines the user's gaze point based on the gaze information, 3D position, and screen orientation, thus obtaining the first gaze point message.

[0146] Specifically, the eye tracker is placed in a fixed position to calculate the pose data of each screen, including the 3D position and orientation of each screen. Based on the user's gaze information and the 3D position and orientation of each screen, the location of the user's gaze point is determined.

[0147] In this embodiment, the eye tracker is connected to multiple screens and obtains gaze point messages based on gaze information, 3D position, and screen orientation, which can reduce the number of eye trackers used and save costs.

[0148] For example, when determining the user's gaze point based on gaze information, 3D position, and screen orientation, the number of intersections between the user's gaze direction and multiple screens is first determined. If the number of intersections is 1, then the intersection is the gaze point; if the number of intersections is greater than 1, then the intersection of the gaze direction and the screen closest to the user is the gaze point.

[0149] Specifically, the gaze direction is represented by a 3D direction vector. The system determines whether the gaze direction intersects with any of the screen planes; the intersection of the gaze direction with each screen plane is the location of the user's gaze point. When the gaze direction intersects with one screen, that intersection is the gaze point; when the gaze direction intersects with more than one screen, since the gaze direction is unique, the screen behind will be obscured by the screen in front, so the intersection closest to the user is taken as the gaze point.

[0150] In this embodiment, the gaze point is determined by judging the number of intersections between the gaze direction and each screen plane, which can accurately obtain the position of the gaze point and thus perform precise cross-screen operations.

[0151] In another possible implementation, the eye tracker is connected to a screen. When calculating the first gaze point information based on the gaze information, it first determines whether the user's gaze direction intersects with the corresponding screen. If so, it calculates the pixel coordinates of the intersection point on the screen, and determines the user's gaze point based on the pixel coordinates, thus obtaining the first gaze point message.

[0152] Specifically, an eye tracker is placed at each screen. Each eye tracker first determines whether the user is looking at the corresponding screen. If so, it further calculates the pixel coordinates of the user's gaze point and determines the specific location of the gaze point based on the pixel coordinates.

[0153] In this embodiment, the eye tracker is connected to a screen and obtains the first gaze point message based on the gaze information, 3D position and screen orientation. It does not require calculating the 3D position and screen orientation of each screen, which can reduce the amount of calculation and save time.

[0154] The multi-screen interaction method provided in this application obtains the user's gaze information, determines a first gaze point based on the gaze information, and sends the first gaze point information to the electronic device. In this method, obtaining the first gaze point information based on the gaze information allows for accurate determination of the gaze point's location, thereby enabling precise cross-screen operations and improving the efficiency of cross-screen operations.

[0155] Figure 5 This is a schematic diagram of the structure of a multi-screen interaction device 50 provided in an embodiment of this application. For example, please refer to [link / reference]. Figure 5 As shown, the multi-screen interaction device 50 includes:

[0156] The acquisition module 501 is used to acquire the content to be displayed, which includes the content corresponding to the notification message or the request message generated by the user operation.

[0157] Display module 502 is used to display content to be displayed on the first screen by adding a new window.

[0158] The receiving module 503 is used to receive first gaze point information from the eye tracker, which is obtained by the eye tracker based on the user's gaze information.

[0159] The determination module 504 is used to determine the screen that the user is looking at as the second screen based on the first gaze point information.

[0160] The moving module 505 is used to move a newly added window from the first screen to the second screen.

[0161] Optionally, the mobile module 505 is specifically used for:

[0162] When a new window is to be displayed in a fixed position, move the new window to the same position on the second screen as it was when displayed on the first screen. Or

[0163] When the newly added window is a movable window, based on the first gaze point information, the newly added window is moved from the first screen to a first preset area of ​​the second screen. The first preset area is the area within the first preset range where the first gaze point corresponding to the first gaze point information is located.

[0164] Optionally, the receiving module 503 is also configured to receive second gaze point information from the eye tracker.

[0165] The moving module 505 is also used to move the mouse cursor or mouse arrow on the first screen to the location of the second gaze point corresponding to the second gaze point information on the second screen when the first preset condition is met.

[0166] Among them, satisfying the first preset condition includes: the gaze duration corresponding to the second gaze point information is greater than the second preset duration, or, acquiring mouse-related events, wherein the mouse-related events are events triggered by user operation that move the mouse cursor or mouse arrow.

[0167] Optionally, the receiving module 503 is also used to receive third gaze point information from the eye tracker.

[0168] The moving module 505 is further configured to create a sub-screen in a second preset area of ​​the second screen according to the third gaze point information when the second preset condition is met, and move the target area in the first screen into the sub-screen.

[0169] The target area is the area in the first screen where the user operates within a first preset time before the second screen receives the third gaze point information; after an object in the target area is moved to a sub-screen, its original interactive functions are retained.

[0170] The second preset region is the region within the second preset range where the third gaze point information is located.

[0171] The second preset condition includes: within a first preset time before the second screen receives the third gaze point information, the user performs an operation on the first screen, or the user's preset operation is received.

[0172] The multi-screen interaction device 50 shown in this application embodiment can execute the technical solution of the multi-screen interaction method in the above embodiment. Its implementation principle and beneficial effects are similar to those of the multi-screen interaction method. Please refer to the implementation principle and beneficial effects of the multi-screen interaction method. It will not be repeated here.

[0173] Figure 6 For an example, please refer to the structural schematic diagram of another multi-screen interaction device 60 provided in the embodiments of this application. Figure 6 As shown, the multi-screen interaction device 60 includes:

[0174] The acquisition module 601 is used to acquire the user's gaze information.

[0175] The determination module 602 is used to obtain the first gaze point information based on the gaze information.

[0176] The sending module 603 is used to send first gaze point information to the electronic device, so that the electronic device determines the screen that the user is looking at as the second screen based on the first gaze point information, and moves the newly added window from the first screen to the second screen. The first screen is the screen through which the content to be displayed is displayed by the newly added window. The content to be displayed includes the content corresponding to a notification message or a request message triggered by user operation.

[0177] Optionally, the eye tracker can communicate with multiple screens.

[0178] The determining module 602 is specifically used for:

[0179] Calculate the pose data for each screen to obtain the 3D position and orientation of each screen.

[0180] Based on gaze information, 3D position, and screen orientation, the user's gaze point is determined, and the first gaze point message is obtained.

[0181] Optionally, the determining module 602 is specifically used for:

[0182] Based on gaze information, 3D position, and screen orientation, determine the number of intersections between the user's gaze direction and multiple screens.

[0183] If the number of intersections is 1, then that intersection is a fixation point.

[0184] If the number of intersection points is greater than 1, then the intersection point between the line of sight and the screen closest to the user is the fixation point.

[0185] Optionally, the eye tracker communicates with a screen.

[0186] The determining module 602 is further used for:

[0187] Based on the gaze information, determine whether the user's gaze direction intersects with the corresponding screen.

[0188] If so, calculate the pixel coordinates of the intersection point on the screen.

[0189] Based on pixel coordinates, the user's gaze point is determined, and the first gaze point message is obtained.

[0190] The multi-screen interaction device 60 shown in this application embodiment can execute the technical solution of the multi-screen interaction method in the above embodiment. Its implementation principle and beneficial effects are similar to those of the multi-screen interaction method. Please refer to the implementation principle and beneficial effects of the multi-screen interaction method. It will not be repeated here.

[0191] Figure 7 This is a schematic diagram of the structure of an electronic device 70 provided in an embodiment of this application. For example, please refer to [link / reference]. Figure 7 As shown, the electronic device 70 may include a processor 701 and a memory 702; wherein,

[0192] Memory 702 is used to store computer programs.

[0193] The processor 701 is used to read the computer program stored in the memory 702 and execute the multi-screen interaction method in the above embodiments according to the computer program in the memory 702.

[0194] Optionally, the memory 702 can be either standalone or integrated with the processor 701. When the memory 702 is a device independent of the processor 701, the electronic device 70 may further include a bus for connecting the memory 702 and the processor 701.

[0195] Optionally, this embodiment further includes a communication interface, which can be connected to the processor 701 via a bus. The processor 701 can control the communication interface to realize the acquisition and transmission functions of the aforementioned electronic device 70.

[0196] For example, in this embodiment of the application, the electronic device 70 can be a terminal or a server, which can be set according to actual needs.

[0197] The electronic device 70 shown in the embodiments of this application can execute the technical solution of the multi-screen interaction method in the above embodiments. Its implementation principle and beneficial effects are similar to those of the multi-screen interaction method. Please refer to the implementation principle and beneficial effects of the multi-screen interaction method. It will not be repeated here.

[0198] This application also provides a computer-readable storage medium storing computer-executable instructions. When the processor executes the computer-executable instructions, it implements the technical solution of the multi-screen interaction method in the above embodiments. The implementation principle and beneficial effects are similar to those of the multi-screen interaction method, and can be found in the implementation principle and beneficial effects of the multi-screen interaction method. They will not be repeated here.

[0199] This application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the technical solution of the multi-screen interaction method in the above embodiments. Its implementation principle and beneficial effects are similar to those of the multi-screen interaction method. Please refer to the implementation principle and beneficial effects of the multi-screen interaction method. It will not be repeated here.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0201] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units.

[0202] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0203] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0204] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0205] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0206] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-screen interaction method, characterized in that, Applied to an electronic device, the electronic device including a first screen and a second screen, the method includes: Obtain the content to be displayed, which includes the content corresponding to a notification message or a request message generated by a user operation; The content to be displayed is shown in a new window on the first screen; Receive first gaze point information from the eye tracker, wherein the first gaze point information is obtained by the eye tracker based on the user's gaze information; Based on the first gaze point information, the screen that the user is gazing at is determined to be the second screen; If the time between receiving the user's gaze point information and obtaining the content to be displayed is less than a preset time, the newly added window will be moved from the first screen to the second screen. Moving the newly added window from the first screen to the second screen includes: When the newly added window is a window displayed in a fixed position, move the newly added window to the second screen to the same position where the newly added window was displayed on the first screen; or When the newly added window is a movable window, the newly added window is moved from the first screen to a first preset area of ​​the second screen according to the first gaze point information. The first preset area is the area within a first preset range where the first gaze point corresponding to the first gaze point information is located.

2. The method according to claim 1, characterized in that, The method further includes: Receive second gaze point information from the eye tracker; When the first preset condition is met, the mouse cursor or mouse arrow on the first screen is moved to the position of the second gaze point corresponding to the second gaze point information on the second screen, according to the second gaze point information. The first preset condition includes: the gaze duration corresponding to the second gaze point information is greater than the second preset duration, or, obtaining mouse-related events, wherein the mouse-related events are events triggered by the user operation that move the mouse cursor or the mouse arrow.

3. The method according to claim 1, characterized in that, The method further includes: Receive third gaze point information from the eye tracker; When the second preset condition is met, a sub-screen is created in the second preset area of ​​the second screen according to the third gaze point information, and the target area in the first screen is moved to the sub-screen. The target area is the area in the first screen where the user operates within a first preset time period before the second screen receives the third gaze point information; after an object in the target area moves to the sub-screen, it retains its original interactive functions. Wherein, the second preset region is the region within the second preset range where the third gaze point corresponding to the third gaze point information is located; The second preset condition includes: within a first preset time period before the second screen receives the third gaze point information, the user performs an operation on the first screen, or the second screen receives the user's preset operation.

4. A multi-screen interaction method, characterized in that, Applied to an eye tracker, the method includes: Obtain the user's gaze information; Based on the gaze information, the first gaze point information is obtained; The first gaze point information is sent to the electronic device, which determines that the screen the user is looking at is the second screen based on the first gaze point information. If the time between receiving the user's gaze point information and obtaining the content to be displayed is less than a preset time, the new window is moved from the first screen to the second screen. The first screen is the screen through which the content to be displayed is displayed. The content to be displayed includes the content corresponding to a notification message or a request message triggered by the user's operation. Moving the newly added window from the first screen to the second screen includes: When the newly added window is a window displayed in a fixed position, move the newly added window to the second screen to the same position where the newly added window was displayed on the first screen; or When the newly added window is a movable window, the newly added window is moved from the first screen to a first preset area of ​​the second screen according to the first gaze point information. The first preset area is the area within a first preset range where the first gaze point corresponding to the first gaze point information is located.

5. The method according to claim 4, characterized in that, The eye tracker is communicatively connected to multiple screens; The step of calculating the first gaze point information based on the gaze information includes: Calculate the pose data for each screen to obtain the 3D position and orientation of each screen; Based on the gaze information, the 3D position, and the screen orientation, the user's gaze point is determined, and the first gaze point information is obtained.

6. The method according to claim 5, characterized in that, Determining the user's gaze point based on the gaze information, the 3D position, and the screen orientation includes: Based on the gaze information, the 3D position, and the screen orientation, determine the number of intersections between the user's gaze direction and the multiple screens; If the number of intersection points is 1, then the intersection point is the fixation point; If the number of intersection points is greater than 1, then the intersection point of the gaze direction and the screen closest to the user is the gaze point.

7. The method according to claim 4, characterized in that, The eye tracker is connected in communication with a screen; The step of calculating the first gaze point information based on the gaze information includes: Based on the gaze information, determine whether the user's gaze direction intersects with the corresponding screen. If so, calculate the pixel coordinates of the intersection point on the screen; Based on the pixel coordinates, the user's gaze point is determined, and the first gaze point information is obtained.

8. A multi-screen interactive device, characterized in that, Applied to electronic devices, including: The acquisition module is used to acquire the content to be displayed, which includes the content corresponding to the notification message or the request message generated by the user operation. The display module is used to display the content to be displayed on the first screen by adding a new window; A receiving module is configured to receive first gaze point information from an eye tracker, wherein the first gaze point information is obtained by the eye tracker based on the user's gaze information; The determining module is used to determine, based on the first gaze point information, that the screen the user is gazing at is the second screen; The moving module is used to move the new window from the first screen to the second screen if the time between receiving the user's gaze information and obtaining the content to be displayed is less than a preset time. The moving module is specifically used to move the newly added window to the same position on the second screen as when the newly added window is displayed on the first screen, when the newly added window is a window displayed at a fixed position; or When the newly added window is a movable window, the newly added window is moved from the first screen to a first preset area of ​​the second screen according to the first gaze point information. The first preset area is the area within a first preset range where the first gaze point corresponding to the first gaze point information is located.

9. A multi-screen interactive device, characterized in that, Applications in eye trackers include: The acquisition module is used to acquire the user's gaze information; The determining module is used to obtain the first gaze point information based on the gaze information; The sending module is used to send the first gaze point information to the electronic device. The electronic device determines, based on the first gaze point information, that the screen the user is gazing at is the second screen. If the time interval between receiving the user's gaze point information and acquiring the content to be displayed is less than a preset time interval, the new window is moved from the first screen to the second screen. The first screen is the screen through which the content to be displayed is displayed via the new window. The content to be displayed includes content corresponding to a notification message or a request message triggered by the user's operation. Moving the new window from the first screen to the second screen includes: when the new window is a window displayed in a fixed position, moving the new window to the same position on the second screen as when it was displayed on the first screen; or when the new window is a movable window, moving the new window from the first screen to a first preset area on the second screen based on the first gaze point information. The first preset area is the area within a first preset range where the first gaze point corresponding to the first gaze point information is located.

Images