Sub-display notification processing

By specifying and matching sub-displays in large display systems based on gestures, the problem of users' difficulty in quickly layout and selecting content is solved, a flexible and easy user interface is realized, and the design of the display control system is simplified.

CN118939170APending Publication Date: 2024-11-12MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410933143.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-12-16
Filing Date
2020-11-05
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing large-scale display systems lack flexibility and ease in shared spaces, making it difficult for users to quickly select and layout content, especially in a multi-person interaction environment. The user interface of traditional display control systems is complex and inconvenient to non-professional. User use.

Method used

Using gesture-based methods to specify and pair sub-displays, users draw the outline of the sub-display on the display through touch or gestures, and pair it with the content source device through PIN, biometrics, etc. to realize content sharing and input routing. , simplifies the user interface and allows users to layout and select content naturally.

Benefits of technology

Improves user flexibility and ease of content selection and layout on the display, reduces the complexity and load of the display control system, provides an intuitive and easy-to-use interface, and supports the pairing and content of multiple input and output devices. shared.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939170A_ABST
    Figure CN118939170A_ABST
Patent Text Reader

Abstract

Display systems, methods, devices, and machine-readable media are disclosed in some examples that provide a gesture-based method for specifying an area of a display screen in which shared content is shown. A pairing method for associating a region of a display screen with a content source device, a method for sharing content within the region, and a method for providing input from the region back to the content source device are also disclosed.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with application number 202080087067.7 and name “Sub-display screen notification processing” filed on November 5, 2020. Background Art

[0002] Large display screens, such as those that cover entire walls, billboards, and sides of buildings, are used to provide content during conferences, trade shows, lectures, advertisements, and other events. These display screens may be made of a single display screen, or may be composed of multiple smaller display screens connected by hardware and software to form a large display screen. In some examples, the display screen may have input capabilities such as detecting touch input or gestures using a touch screen or a camera that detects gestures and / or touch events. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] In the accompanying drawings, which are not necessarily drawn to scale, like numbers may describe similar components in different views. Like numbers with different letter suffixes may represent different instances of similar components. The accompanying drawings generally illustrate various embodiments discussed in this document by way of example and not limitation.

[0004] Figure 1-3 Display systems according to some examples of the present disclosure are illustrated.

[0005] Figure 4 The use of gestures to designate a sub-display of a display screen according to some examples of the present disclosure is illustrated.

[0006] Figure 5 Illustrated is gesture-based designation of a sub-display according to some examples of the present disclosure.

[0007] Figure 6 The designation and pairing process according to some examples of the present disclosure is illustrated.

[0008] Figure 7 The designation and pairing process using biometrics according to some examples of the present disclosure is illustrated.

[0009] Figure 8 The designation and pairing process for a remote content source device according to some examples of the present disclosure is illustrated.

[0010] Fig. 9 Data flows according to some examples of the present disclosure are illustrated for a remote content source device that is not physically close to a display screen and shares content on the display screen through a communication service intermediary according to some examples of the present disclosure.

[0011] Fig.10A display screen with two designated sub-display screens (one for input) is illustrated according to some examples of the present disclosure.

[0012] Fig.11 An example of a sub-display screen with a notification area according to some examples of the present disclosure is illustrated.

[0013] Fig.12 An example of a graphical user interface (GUI) for a communication application according to some examples of the present disclosure is illustrated.

[0014] Fig.13 A display screen having multiple sub-display screens corresponding to multiple communication modalities of a communication application according to some examples of the present disclosure is illustrated.

[0015] Fig.14 Illustrations of suggested additional sub-displays according to some examples of the present disclosure are illustrated.

[0016] Fig.15 The data flow of a suggested sub-display screen according to some examples of the present disclosure is illustrated.

[0017] Fig.16 A diagram illustrating suggested display screen size changes according to some examples of the present disclosure is illustrated.

[0018] Fig.17 A logic diagram of a display screen control system according to some examples of the present disclosure is illustrated.

[0019] Fig.18 The method flow of service advertisement according to some examples of the present disclosure is illustrated.

[0020] Fig.19 The diagram illustrates a method flow for processing a sub-display screen creation notification according to some examples of the present disclosure.

[0021] Fig. 20 A flow chart of a method for processing content from a content source device at a display screen control system according to some examples of the present disclosure is illustrated.

[0022] Fig.21 A flow chart of pairing operations according to some examples of the present disclosure is illustrated.

[0023] Fig. 22 A flow chart of a method of routing input to a content source device according to some examples of the present disclosure is illustrated.

[0024] Fig.23 A flowchart of a method of providing a notification to a sub-display screen according to some examples of the present disclosure is illustrated.

[0025] Fig.24A flowchart of a method in which a notification component processes input to a notification according to some examples of the present disclosure is illustrated.

[0026] Fig.25 A logical diagram of a display screen according to some examples of the present disclosure is illustrated.

[0027] Fig.26 A flowchart of a method of registering a display screen with a controller according to some examples of the present disclosure is illustrated.

[0028] Fig. 27 A flowchart of a method of creating a sub-display screen performed by a sub-display screen creation component according to some examples of the present disclosure is illustrated.

[0029] Fig.28 A flowchart of a method in which a content display component causes content to be displayed according to some examples of the present disclosure is illustrated.

[0030] Fig.29 A flowchart of a method of processing input directed to a sub-display screen according to some examples of the present disclosure is illustrated.

[0031] Fig.30 A flowchart of a method in which a notification processing program component processes a notification message from a content source device or a display screen control system according to some examples of the present disclosure is illustrated.

[0032] Fig.31 A flowchart of a method in which a notification handler component processes a gesture directed to the notification area according to some examples of the present disclosure is illustrated.

[0033] Fig.32 A diagram of a content source device 3210 is illustrated according to some examples of the present disclosure.

[0034] Fig.33 A flow chart of a method for sharing content on a sub-display of a remote display according to some examples of the present disclosure is illustrated.

[0035] Fig.34 A flowchart of a method of processing input from a sub-display screen according to some examples of the present disclosure is illustrated.

[0036] Fig.35 A flowchart of a method of presenting a notification to a sub-display screen according to some examples of the present disclosure is illustrated.

[0037] Fig.36 A flow chart of a method of processing input such as a gesture directed to a notification displayed on a secondary display screen is illustrated.

[0038] Fig.37 A flowchart of a method of specifying a sub-display screen according to some examples of the present disclosure is illustrated.

[0039] Fig.38 A flowchart of a pairing method according to some examples of the present disclosure is illustrated.

[0040] Fig.39 A flowchart of a method of inputting a sub-display screen according to some examples of the present disclosure is illustrated.

[0041] Fig.40 A flowchart of a method of providing notification according to some examples of the present disclosure is illustrated.

[0042] Fig.41 A flow chart of a method of designating a sub-display screen area for a remote content source device that is not physically close to a display screen according to some examples of the present disclosure is illustrated.

[0043] Fig.42 is a block diagram illustrating an example of a machine upon which one or more embodiments may be implemented.

[0044] Fig.43 The diagram illustrates the content splitting process according to some examples of the present disclosure. DETAILED DESCRIPTION

[0045] In order to specify the content for presentation on the display screen, a graphical user interface (GUI) is usually provided on a controller device (e.g., a computing device). The user selects content, placement, and layout options for the display screen through the GUI. The content is sent from the controller device to the display screen and is displayed according to the selected layout and placement. Organizations that deploy these systems often have administrators or some other users who are familiar with and can access the GUI of the controller device. Although these systems are often powerful and provide flexibility in layout options, they are not practical for shared spaces such as conference rooms, where it is expected that many different people will interact with and use the display screen at various times. It may be cost-effective to hire an administrator to set up a display screen that will run for a long time in a specific configuration (e.g., as in a digital billboard or trade show), but it is not cost-effective to have an administrator on call to customize the system for users in a shared space (e.g., a conference room). Therefore, many organizations make the display control system GUI publicly accessible to users to customize the layout for their personal purposes. This places a burden on users who may not be familiar with the GUI of the display control system, which may result in a suboptimal experience with the display screen.

[0046] To address this problem, other display systems have simplified the user interface for specifying content, placement, and layout options. This simplification is easier on the user because it provides a more limited interface for the user to control, but the trade-off is less flexibility. For example, a large television may allow multiple different content sources to be displayed on a single screen, but these systems typically only provide a limited number of preset layout options that can be accessed via a remote control interface. In addition, these systems lack sophisticated input handling mechanisms.

[0047] Current large display systems therefore lack the flexibility and ease of use of content selection, layout, and placement options to allow users to quickly and easily place content on their desired display screens. As mentioned, current display systems are often characterized by proprietary interfaces with static layout options that are not available to end users, or they may be characterized by more limited interfaces that do not allow for appropriate flexibility in specifying content for display. As used herein, content may refer to one or more of the following: documents, document data, video, audio, screen-shared data, application-shared data (e.g., video frames from an application), etc.

[0048] In some examples, display systems, methods, devices, and machine-readable media are disclosed that provide a gesture-based method for designating a sub-display screen (e.g., an area of ​​a display screen in which shared content is shown). Also disclosed are pairing methods for linking (e.g., associating) a sub-display screen with a content source device, methods for sharing content within the area, and methods for providing input from the area back to the content source device.

[0049] A sub-display screen is a defined area in the display screen that displays content from a specific content source device. The sub-display screen can be independent of any other sub-display screen because the size, shape, position, settings and content source device of each sub-display screen can be different. A user can perform a gesture above, on or pointing to a display area to designate that area of ​​the display screen as a sub-display screen for sharing content. The gesture can be a touch gesture, or can be an air gesture that can be detected by a camera. The input that specifies the sub-display screen can specify the coordinates and shape of the sub-display screen on the display screen. For example, a user can touch the display screen and drag his finger on the display screen to draw the outline of the sub-display screen at the location touched by the user. As used herein, a sub-display screen is a part of a larger display screen that can be paired (i.e., linked) with a content source device to display content selected by the content source device. Each sub-display screen can display content different from one or more other sub-display screens on the same display screen. The content source device paired with the sub-display screen can transmit content to the display screen by direct connection or by network connection. Input from the sub-display screen can be directed to the content source device paired with the sub-display screen.

[0050] Each sub-display can be graphically distinguished from the surrounding display area (which may include other sub-displays) by a graphical outline, border, frame, window, etc. An exemplary input that results in the designation of a sub-display includes a user tracing his finger over the display at the location and in the shape of the desired sub-display. The shape can be a square, a circle, a triangle, a polygon, etc. Other exemplary gestures may include a drag operation, wherein the point at which the user's finger contacts the display is a first corner of the sub-display, and the user drags his finger at a second corner of the sub-display and releases his finger - the second corner is diagonal to the first corner. The display is configured to detect gestures by using a touch screen, a camera, or other input device.

[0051] To display content in the sub-display screen, after designating the sub-display screen (or in some examples, before designating the sub-display screen), the user can designate the content source device by pairing the sub-display screen with the content source device that provides the content displayed in the sub-display screen using a PIN, biometrics, QR code, or other method. Once paired, the content source device can send content to the display screen for sharing on the sub-display screen. The designation of the sub-display screen and the linking of the sub-display screen with one or more applications are all accomplished without requiring a separate user interface of a controller device.

[0052] The disclosed display system can allow one or more applications on a single content source device to simultaneously project the same or different content to multiple sub-displays on the display screen. Similarly, multiple content source devices can simultaneously project the same or different content to multiple different sub-displays of the same display screen.

[0053] The disclosed display system may also be configured to detect user input, including touch input, gesture input, etc. directed to one or more of the sub-displays or within one or more of the sub-displays. Input may be detected simultaneously on multiple sub-displays or in multiple sub-displays. These inputs may be sent back to an application linked to the sub-display on the content source device and processed by an application on the content source device (such as an application providing content shared on the sub-display). In some examples, the input may be processed as if it were generated by an input device directly coupled to the content source device. In some examples, a sub-display may be designated as an input-only sub-display and may be separated from any sub-display that displays content. An input sub-display may display input for any input, or may hide input for input in order to provide a private area for input.

[0054] In some examples, the input to the sub-display screen can be provided by a third device. For example, a content source device (e.g., a laptop computer) can provide content displayed in the sub-display screen. The content source device can be any computing device that can share content in the sub-display screen. A third device (such as a tablet computer) can allow a user to provide input to the sub-display screen. To achieve this, a separate input device (e.g., a tablet computer) can be paired with the sub-display screen (using the same method as disclosed herein for the content source device). The pairing can be an input-only pairing or an input and / or output pairing. The input from the tablet computer can be sent by the display screen control system (or in some examples, the display screen control system can facilitate the P2P connection between the input device and the content source device by providing the peer-to-peer (P2P) address of the content source device to the input device) to the content source device, in which the input is processed and reflected in any changes in the content sent to the display screen control system.

[0055] More generally, it is contemplated that multiple input and / or output devices may be paired with a single sub-display. In the case of multiple output and input devices, user-configurable rules on the display control system may govern how conflicting inputs and / or outputs are handled. For example, different content sources may be displayed side-by-side in the sub-display. Although multiple input and / or output devices may be paired with a single sub-display, in other examples, the sub-display may accept content and send input to a single content source device.

[0056] The sub-display screen may also display one or more notifications in the notification area. The notifications may be sent by a paired content sharing device. Notifications may appear within the sub-display screen, or may be visually attached to the sub-display screen or to a notification area adjacent to the sub-display screen. A user may interact with the notifications, such as dismissing a notification, replying to a notification, obtaining more information about an event related to the notification, and the like.

[0057] Additionally, although the foregoing designation and pairing process describes designation and pairing for users who are in the same room as a display screen, in some examples, a remote user may share content using an intermediary server (such as a communications server) and a sub-display screen of a display screen that is not in the same area as the remote user.

[0058] By utilizing gesture-based designation of sub-displays, a user can quickly and easily designate content for a portion of the display. Because gestures are drawn relative to the display itself rather than on the controller's user interface, the user can visually layout the display to more naturally achieve the desired layout and content selection. This avoids problems with conventional display control system user interfaces, where it may be difficult to visualize each layout selection when applied to a larger display (rather than a user interface that may be on a smaller screen). This also avoids problems that plague simpler user interfaces, which may offer less flexible layout options, because in the disclosed system, the size and shape of each sub-display is "drawn" by the user using gestures, and is therefore entirely within the user's control.

[0059] Therefore, the present disclosure discloses an improved user interface for computing, such as for large display screens. The improved display interface allows users to share content of content source devices more quickly in the desired location and layout on the display screen. The disclosed method is faster and easier than conventional solutions, while still maintaining flexibility. Instead of flipping through multiple screens of options in the display control system interface or dealing with more limited options for a more user-friendly interface, in the disclosed system, the user only needs to draw a sub-display on a larger display screen, select a sharing option in its content sharing device, and enter pairing information. The present disclosure also eliminates the need for a separate display control system GUI interface, thereby simplifying the design of the display control system without sacrificing the flexibility provided by those GUIs. Therefore, the present disclosure solves the technical problems of laying out, organizing and selecting content on a display screen by utilizing a technical solution for gesture-based input, pairing and content sharing. The present disclosure improves the functionality of a computer system by providing an improved user interface, by refining the complex layout and setup process into an intuitive and easy-to-use interface, and by reducing the processing load of the display control system by eliminating the GUI-while maintaining the flexibility and functionality of a more complex system.

[0060] Figure 1 A display system 100 according to some examples of the present disclosure is illustrated. A display screen 120 is communicatively coupled to a display screen control system 110. The display screen 120 and the display screen control system 110 may communicate by wire or wirelessly, and may be coupled directly or through a network (e.g., a packet-based network). In some examples, the display screen control system 110 may control one or more display screens, such as the display screen 120. In some examples, the display screen 120 registers with the display screen control system 110 to be controlled by the display screen.

[0061] For example, the display screen control system 110 can be any computing device (whether separate from the display screen 120 or integrated with the display screen 120) that manages the sub-display screens on the display screen. The display screen 120 can send a broadcast message, and the data in the message indicates that it is the display screen that is looking for the controller. The message can be sent to a specifically defined port and a specifically defined broadcast address. The display screen 120 can then listen for these messages and reply to the display screen control system 110 at the address from which the broadcast message was sent. Once the display screen 120 and the display screen control system 110 find each other and determine the network or other address, the display screen control system 110 and the display screen 120 can then negotiate one or more parameters used by the display screen control system 110 to control the display screen 120. Exemplary parameters can include the network address to send content, the network address to send input from the display screen to the display screen control system, how to perform pairing, the parameters of the display screen (e.g., the size and capabilities of the display screen), etc. The display screen control system 110 can then register the display screen 120 in a database of controlled display screens. For example, the display screen control system 110 can create a database record with the parameters, address, and capabilities of the display screen. Although the display screen control system 110 broadcasts the message in the above example, in other examples, the display screen 120 broadcasts a message to find the display screen control system.

[0062] The display screen 120 and / or the display screen control system 110 can recognize one or more inputs directed to the display screen 120 for creating one or more sub-display screens. For example, the display screen 120 can be a touch screen display screen. In other examples, the display screen 120 can have a camera directed to the display screen or to a user directed to the display screen. The camera can be communicatively coupled to the display screen 120 or the display screen control system 110 that can detect gestures. The user touches the display screen outside the created sub-display screen and the drag gesture can be recognized as a command to create a new sub-display screen. The geometry of the gesture can determine the geometry of the sub-display screen. For example, the user can outline the outline of the desired sub-display screen. In some examples, the geometry can be corrected by the system (for example, a circle with uneven lines and variable radius attempted by the user can be corrected to a circle with a constant radius).

[0063] In some examples, visual feedback can be provided to the user as the user creates a sub-display that shows the boundaries of the sub-display as it is created. For example, as the user moves their finger on the display to create a sub-display, an outline of the sub-display currently being created can be shown. In other examples, instead of an outline, the sub-display can be created as a box, where the opposite corners are the first and last touch points in the drag operation. When the user drags from one corner to the opposite corner, the display can show an expanded window or box.

[0064] In an example where an input device communicatively coupled to display control system 110 recognizes a gesture to designate a sub-display (e.g., a camera coupled to display control system 110 can capture video of the display to detect the input), display control system 110 can send a message indicating the input designating a new sub-display and the coordinates of the input to display 120 for displaying visual feedback (e.g., for displaying an outline or other visual designation of a boundary of the sub-display during the gesture).

[0065] Once the size and shape of the sub-display are specified, display screen 120 and display screen control system 110 may create the sub-display and pair the sub-display with a content source device, such as content source device 130 .

[0066] In an example where the display screen 120 detects a gesture to designate a sub-display screen, the display screen 120 may record the coordinates of the sub-display screen input by the user and may transmit the coordinates to the display screen control system 110. The coordinates may be sample points along the user's trajectory, or may be coordinates of a box designated by the user by determining that the user first presses a first corner of the screen and the user lifts a finger off a second corner of the screen. In an example where an input device coupled to the display screen control system 110 recognizes a gesture to designate a sub-display screen, the display screen control system 110 may send the coordinates to the display screen 120. Once a sub-display screen is designated, additional user input may move, resize, change shape (e.g., from a first polygon to a different polygon or circle), and close the sub-display screen.

[0067] When specifying a sub-display, the display screen 120 and / or the display screen control system 110 may create a data structure that stores information about the sub-display. For example, the coordinates of the sub-display, a unique identifier of the sub-display (e.g., a PIN number of the sub-display), information about the display screen on which the sub-display was created, etc. Once a content source device is paired with a sub-display, the data structure may have fields that may later be populated with information about the content source device. For example, the address of the content source device, the application identifier of the content source device, etc. are used to accept content from the content source device and match the content to the sub-display, and also route input from the sub-display and route it to the correct application on the content source device.

[0068] To pair the sub-display screen on the display screen 120 with the content source device 130, one or more pairing methods may be used. For example, the content source device 130 (e.g., a computing device) may discover the display screen control system 110. The display screen control system 110 may broadcast advertisements for display services, and the content source device 130 may listen for these advertisements. The broadcast may indicate the address of the display screen control system 110. The content source device 130 may save the address for future streaming of content and pairing with the display screen control system 110.

[0069] When a sub-display is created, the display 120 may display a PIN, QR code, password, or other unique identifier for the sub-display. This may be displayed in a graphical user interface that is located in the sub-display, near the sub-display, visually in contact with but not overlapping the sub-display, etc. For example, in a box that is below the sub-display and visually connected to but not overlapping the sub-display. For example, Figure 6 The PIN is shown in the pairing label 630 below the sub-display. The pin can be created by the display 120 or the display control system 110.

[0070] The user may then enter the PIN into an application on the content source device 130. For example, based on the discovery of the display service, the content source device may provide the user with an option to share the content of a particular application to a display screen, such as the display screen 120. For example, through a context or share menu. The application may then send the entered PIN to the discovered display screen control system 110. The display screen control system 110 may then match the application of the content source device 130 with the sub-display screen on the display screen 120. The content source device 130 may then begin sharing content to the sub-display screen by sending the content to the display screen control system 110, and may receive input from the display screen 120 via the display screen control system 110.

[0071] Content source device 130 can communicate with display screen control system 110 and / or display screen 120 in various wired and wireless ways, including through a network such as a packet-based network. Although display screen control system 110 is shown, those of ordinary skill in the art will appreciate that the functionality of display screen control system 110 can be implemented in one or more other devices as well as display screen 120.

[0072] Display screen 120 may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a plasma display, etc. Display screen 120 may include a touch screen. Display screen 120 or display screen control system 110 may also be coupled to a camera that detects gestures directed to the display screen. Display screen control system 110 may be a computing device such as a server computing device, a laptop computer, a desktop computer, a tablet computer, a smart phone, a dedicated hardware controller, etc. Display screen control system 110 may be attached to display screen 120 or the same device as display screen 120 and communicatively coupled to display screen 120 via a local connection, or may be a different device communicatively coupled to display screen 120 via a network. Content source device 130 may be a computing device such as a laptop computer, a desktop computer, a tablet computer, a smart phone, a smart watch, etc.

[0073] Figure 2 FIG. 2 illustrates a display system 200 according to some examples of the present disclosure. Figure 2 In the illustration of FIG. 1 , once the content source device 130 is paired with the sub-display, the content source device 130 can send the content directly to the display screen 120, rather than Figure 1 The display screen control system 110 sends it to the content source device 130 in the address of the display screen 120 during pairing. This can be achieved by the display screen control system 110 providing the address of the display screen 120 to the content source device 130 during pairing.

[0074] Figure 3 A display system 300 according to some examples of the present disclosure is illustrated. Figure 3 In the embodiment, the display screen control system is a part of the display screen 120. Figure 3 The display screen 120 in the iPod communicates directly with the content source device 130 for discovery, pairing, and content sharing.

[0075] Sub-display designation

[0076] Figure 4 The diagram illustrates the use of gestures to designate sub-displays of display screen 120 according to some examples of the present disclosure. When a user first touches display screen 120 at first point 410-A and drags his finger to points 410-B and 410-C in a diagonal motion, Figure 4 The gesture may be a drag gesture. The designated sub-display will be a rectangle with diagonal vertices at first point 410-A and point 410-C where the user removes their finger from the display. In some examples, after the first touch at first point 410-A and the subsequent drag toward point 410-B, display 120 may provide an outline of the window or other graphical representation, such as representation 420. As the user moves their finger, the representation updates to match the current size and shape of the sub-display. Although in Figure 4A rectangle is shown in , but in other examples other shapes can be specified in the same manner, with their sizes scaled to occupy a bounding box with vertices at the start and end points of the gesture.

[0077] exist Figure 4 , other sub-display screens are also shown. For example, sub-display screen 430 shows content A shared from content source device 1. The content may come from a specific application on content source device 1—for example, from a first application on content source device 1. Sub-display screen 440 displays content A from content source device 2, for example, from a first application on content source device 2. Sub-display screen 450 displays content B from content source device 1, for example, from a second application on content source device 1.

[0078] Figure 5 Other gesture-based designations of sub-display screens according to some examples of the present disclosure are illustrated. For example, instead of a drag operation in which diagonal vertices are specified by the start and end points of the drag, the user can specify the outline of the sub-display screen. For example, the user can draw a square, rectangle, polygon, circle, etc., and the sub-display screen can use the path drawn by the user as the coordinates of the sub-display screen. The sub-display screen shape can be the exact shape drawn by the user, that is, it may not be updated to a smooth shape, so that the imperfections of the user's drawing of the shape may be reflected in the sub-display screen. In other examples, the system can automatically or in response to user input - to correct the user's drawing to match the closest shape from the shape library. For example, as in Figure 5 As shown in , the user's sketch 510 is corrected into a square 520. The user's sketch 530 is corrected into an octagon 540, and the user's sketch 550 is corrected into a circle 560.

[0079] The system can achieve the correction by straightening the lines between the vertices in the user's sketch. The vertices can be identified by identifying changes in direction, for example, by calculating the slope of the line between consecutive coordinates of the user's sketch, and periodically comparing the slope of the entire user's sketch and comparing it to a threshold. In other examples, the completed shape is compared with a shape library. The system can calculate the total difference in (x, y) coordinate points between the points and / or vertices of each library shape and the outline drawn by the user. The system can select the shape with the smallest total difference. In some examples, in order to properly normalize the size, the shape drawn by the user can be scaled to a normalized size for comparison, or the shapes in the shape library can be normalized to a size corresponding to the user's sketch.

[0080] In some examples, the display screen may visually indicate the geometric correction by animating the movement of a line drawn by the user's sketch to the corrected geometric shape.The sub-display screen may adopt the size and boundaries indicated by the corrected geometric shape.

[0081] In some examples, whether the outline is geometrically corrected can be set by one or more settings. A global setting that is applied to each sub-display designation without user input during the designation can be used. In some examples, the setting can be set based on each sub-display designation. For example, a user can apply user input before, during, or after designating a sub-display to specify whether the system should correct the shape drawn by the user. For example, during the designation of a sub-display (e.g., during the designation of a shape), if the user taps the display with a different finger (while the user is still sketching with another finger)-a menu may be displayed that allows the user to activate geometric correction. In other examples, the menu can allow the user to select the shape to which the sketch is to be corrected.

[0082] To make the content conform to the selected shape, the content source device, display screen and / or display screen control system can apply one or more transformations to the content, such as resizing the content, stretching the content, skewing the content, cropping the content (e.g., to the shape of a sub-display screen), rotating the content, etc.

[0083] Pairing Operation

[0084] In order to route content from a content source device to the correct sub-display on a display screen, the content source device may be paired (i.e., linked) with the sub-display. Pairing (i.e., linking) is used herein to describe a process for associating a content source device with a specific sub-display to allow content to be routed from the content source device to the specific sub-display. The pairing process may occur before or after specifying the coordinates of the sub-display. Figure 6 A flow chart of a designation and pairing process according to some examples of the present disclosure is illustrated. Figure 6 In the example of , the user specifies the coordinates of the sub-display screen 610 by specifying the corners of the sub-display screen. The remaining coordinates are calculated by the system based on those two corners. The sub-display screen may include a GUI window with controls for resizing, minimizing, maximizing, and closing the sub-display screen. In other examples, the sub-display screen may be frameless with no visible controls. Those of ordinary skill in the art who have the benefit of applicant's disclosure will appreciate that other visual configurations are also contemplated.

[0085] In some examples, a pairing label 630 may be shown adjacent to a sub-display screen 620 having pairing information. Figure 6 As shown in , pairing label 630 is connected to one edge but does not overlap with the sub-display. In some examples, pairing label 630 can be displayed in sub-display 620. In other examples, the pairing information in pairing label 630 can be displayed as text in the sub-display. For example, as shown in Figure 6As shown in , the sub-display currently displays the text "Please enter a PIN in your shared application". In some examples, the PIN may be displayed together with the message, or may be displayed in place of the message, etc. The pairing information displayed in the pairing label 630 may be a PIN number. The PIN number may be created by a display screen, a display screen control system, etc. The PIN number may be a sub-display screen identifier that uniquely identifies the sub-display screen to the display screen and / or the display screen control system.

[0086] The content source device may be running an application that may allow for projection (i.e., sharing) of content. For example, a communication application, a photo application, a productivity application (e.g., a word processing application, a spreadsheet application, a drawing application, etc.), etc. Exemplary communication applications may be teleconferencing applications, video chat applications, video playback applications, photo applications, etc. In some examples, the content source device may allow for sharing of one or more of the display screens of the remote display device. For example, sharing the desktop of the remote display device.

[0087] To share content, a user may select a share content option—either in an application with the content they wish to share, or via a sharing utility (e.g., provided by an operating system of a remote content display device). The share option may be enabled for a display screen in response to a content source device discovering the display service (e.g., by receiving a broadcast message sent by a display screen control system). Figure 6 In the example of , a GUI 640 of a communication application according to some examples of the present disclosure is shown. Figure 6 In the embodiment of the present invention, the communication application is participating in a network-based communication session (such as an online meeting) and the user has indicated that the user wants to share a window. An option is displayed that allows the user to determine where to share content with other remote participants of the network-based communication session ("Conference") or on a remote display screen ("Remote Display Screen"). If Display Screen is selected, a menu or window is displayed that allows the user to enter a PIN displayed on the display screen.

[0088] As already described, an application on the content source device (e.g., an application that allows content sharing, or an operating system service that provides sharing services to other applications that share content) discovers a display service advertised by a display screen control system. When a display service is discovered, the application enables content to be shared to a display screen. When a user indicates to share content, the application or service on the content source device sends a content sharing request with an input PIN to the display screen control system address discovered during the discovery process of the display service. If the PIN matches the displayed PIN, the display service can pair the content source device with the sub-display screen. The displayed PIN can be sent by the display screen to the display screen control system. In other examples, the displayed PIN can be determined by a display service provided by the display screen control system (and sent to the display screen for display). For example, the display screen control system and / or the display screen can record information about the content source device in a data structure describing the sub-display screen. In addition, content from the content source device can be routed to the sub-display screen, and input from the sub-display screen is routed to the content source device based on pairing. The display service can send a pairing confirmation to the content source device. The content source device then sends the content to the display screen control system or the display screen itself - depending on the implementation. The shared content can be video, audio, still images, application displays, etc. Once the displays are paired and streaming begins, at 660 , the content is displayed in the secondary display.

[0089] In other examples, pairing can operate in reverse. That is, the application on the content source device can specify a PIN and the user can enter the PIN in a pairing input area (such as a pairing label) of the display screen. In these examples, the PIN can be sent by the content source device to the display control system. The PIN is then matched with the PIN entered into the pairing label of the sub-display screen.

[0090] pair

[0091] Figure 7 A flow chart of a designation and pairing process using biometrics according to some examples of the present disclosure is illustrated. Figure 6, the user starts by specifying the coordinates of the sub-display screen 710. The displayed sub-display screen 720 may have a pairing tag 730. The pairing tag 730 may be a biometric input area. For example, a fingerprint input area. The display screen may have an integrated fingerprint reader that may allow fingerprint input in one or more locations of the display screen. In some examples, the display screen may have the ability to scan fingerprints across the entire display screen, and the pairing tag 730 may be anywhere on the display screen. In some examples, the pairing tag 730 may be adjacent to and connected to the border of the sub-display screen 720. In these examples, the pairing tag 730 is used to limit the processing power required for scanning fingerprints by limiting the area that needs to be scanned to the coordinates of the pairing tag 730. Instead of scanning the entire surface of the display screen, the display screen only needs to scan the biometrics in the pairing tag 730. In addition, the pairing tag 730 is used to link the sub-display screen with the scanned biometrics so that multiple sub-display screens can be created and linked simultaneously.

[0092] When a user attempts to pair with a sub-display, the user enters a biometric at the display (such as in the pairing tag 730). The biometric template can be sent to the display control system that stores the biometric template and associates it with the specified sub-display (e.g., based on which sub-display's pairing tag scanned the biometric). The user chooses to share content with the display by selecting a share option in the content sharing application (such as through a menu 750 of the GUI of the content sharing application 740). The content source device can send a biometric template (e.g., a fingerprint scan) of the same type as that adopted by the content source device (at pairing, or during the registration or setup process) to the display control system. The display service then searches for a match between the biometric template sent by the content source device and the biometric template entered into the pairing tag (or other location) of the unpaired sub-display. If a match is found, the content source device and the sub-display can be paired, and a message can be sent to the content source device that a match is found and content can begin to be shared, such as shown at the GUI 760.

[0093] In some examples, to protect the privacy of the user, fingerprints and / or other biometrics can use non-standard template types so that if a malicious party were to obtain the biometric template (or be stored in an application or with a display service), the template would not be able to bypass the biometric authentication of other services or applications. In some examples, the biometric template can be encrypted. In some examples, the biometric template stored at the display control system can be deleted from the controller after the pairing process. If a pairing process with a particular biometric template is not generated, the template can be deleted within a short period of time.

[0094] In some examples, pairing tag 730 can be used to capture biometrics, but in other examples, other specific locations for entering biometrics can be utilized. For example, a retinal scanner may be located at a specific location on the display screen. For example, the biometric scanner is located in a central location and the system can limit the creation of sub-displays so that a single sub-display can be paired immediately. This allows a specific sub-display to be linked to a specific captured biometric. The biometric in these examples can be obtained at the display screen by the user presenting themselves at the display screen or near the display screen (e.g., within a defined distance) for scanning. The biometric scanner can be in the same room, adjacent to the display screen (e.g., mounted on the same wall surface as the display screen), facing the display screen (e.g., the user can turn it so that its back is facing the display screen for face or eye scanning). The biometric scanner can be connected via a local interface (such as a universal serial bus interface, a serial interface, etc.), and in other examples, the biometric scanner can be connected via a network interface.

[0095] In other examples, the system may allow for multiple sub-displays to be created simultaneously using a central biometric scanner location. In these examples, methods may be utilized for specifying which sub-display is associated with a biometric scan. For example, a user may enter an identifying name and / or number of a sub-display into an input area of ​​the display settings before or after scanning a biometric for this purpose.

[0096] Remote display sharing

[0097] Although the pairing process described herein is for users of content source devices that have physical access to the display screen when the sub-display screen is specified and paired, in other examples, users of content source devices that do not have physical access to the display screen when the sub-display screen is specified and paired can share content in the sub-display screen of the display screen. These content source devices are referred to herein as remote content source devices. A remote content source device is a content source device whose user does not have physical access to the display screen to pair with the sub-display screen.

[0098] Such as MICROSOFT The communication application of the conference room can allow a remote user to share content on a display screen in a conference room for local participants to view and interact with. In some examples, this can be achieved by one of the local participants receiving the shared content through a network-based communication session (e.g., an online meeting) of the communication application. The local participant can specify a sub-display screen on the display screen and pair the communication application running on the computing device of the local participant with the sub-display screen. This causes the sub-display screen to display any content that the local participant's device is displaying. Therefore, if the local participant is viewing the shared content of the remote user in the communication application, the sub-display screen will also display the content. This method has disadvantages. For example, if the local participant leaves or changes the content displayed in the paired application, this may stop or interrupt the content sharing of other local participants who rely on content sharing. Therefore, the continuous display of the remote user's content depends on the local participant. In addition, some communication applications may not be set to allow the shared content of one user to be re-shared to another location.

[0099] In some examples, the system can pair a remote content source device that is not physically close to a sub-display of a display screen without relying on a local computing device for continued content delivery. Physically not close means that a user of the remote content source device cannot see the display screen. Figure 8 840 . The communication application may have an option, such as within menu 850, for specifying a content source device of the remote user to share to the sub-display of the local display. In the example of FIG. 850 , the user physically close to the display screen first specifies the sub-display screen by using gesture 810 , as previously described. The system may then display a PIN or other pairing mechanism as previously described, such as within a pairing tab 830 of the sub-display screen 820 . The local user on the local computing device may be executing a communication application, such as application 840 . The communication application may have an option, such as within menu 850 , for specifying a content source device of the remote user to share to the sub-display screen of the local display screen. Figure 8 , the option is listed as "Remote Display Sharing" and when this option is selected, a second menu may be displayed that allows the user to enter a pin at the first input area 860 of the created sub-display and specify the remote user's content source device at the second input area 870. When these options are selected, communications may be sent to the remote user's computing device.

[0100] The instance of the communication application running on the remote computing device then allows the remote user to select applications, windows, or other content to share with the remote display screen, such as through menu 890 of GUI 875. Once the remote user selects the appropriate content to share, the remote user's communication application can stream the content to the sub-display screen of the display screen, such as shown at 880. In some examples, a communication server associated with the communication application can facilitate communications between various instances of the communication application and between the display screen and / or display screen control system.

[0101] Fig. 9 The data flow according to some examples of the present disclosure of a remote content source device that is not physically close to a display screen according to some examples of the present disclosure is illustrated, and the remote content source device shares content on the display screen through a communication service intermediary. As previously described, a local computing device 920 (which can also be used as a content source device in a different context) can discover a display service by communicating with a display screen control system 940 at operation 902. As part of the discovery process, the local computing device 920 can discover the address of the display screen control system 940. In some examples, since the remote content source device 910 is not physically close to the display screen control system 940, the remote content source device 910 may not be able to discover the display screen control system 940. For example, a broadcast message of the display screen control system 940 may not reach the remote content source device 910 because it may only be broadcast on a specific subnet or other network area that is not accessible to the remote content source device 910. In other examples, the broadcast message may be wireless, and the remote content source device 910 may be outside the wireless range.

[0102] Next, at operation 904, the user of the local computing device 920 can specify the sub-display screen on the display screen using the previously described method (e.g., by using a gesture to draw the outline of the sub-display screen). At operation 906, the display screen then communicates with the display screen control system 940 to establish a record of the pairing information and the sub-display screen. For example, by creating a data structure such as shown at 950, which has the coordinates of the sub-display screen, a PIN number (which uniquely identifies the sub-display screen on the display screen), content source device information (which may be empty before pairing), and a display screen identifier (which uniquely identifies the display screen). As previously noted in some examples, the PIN is assigned by the display screen control system 940, but is created by the display screen in other examples. In some examples, the PIN is a unique identifier for the sub-display screen across the display screen 930, but in other examples, the PIN is a unique identifier across all display screens managed by the display screen control system 940. In yet other examples, the PIN may not be a unique identifier, but the display screen control system 940 and / or the display screen 930 may assign different unique identifiers.

[0103] Then, as previously described, the pairing information (e.g., PIN) can be displayed on the display screen at operation 908. The user at the local computing device 920 can then select an option in the communication application to allow a remote user of a content source device (e.g., remote content source device 910) to share content on a designated sub-display screen. The user of the local computing device 920 can enter the sub-display screen PIN into a GUI element of the communication application running on the local computing device 920 and enter either a designation of a content source device or a designation of a remote user. In the example of specifying a remote user (as opposed to the remote content source device itself), the system determines the remote content source device on which the user is currently authenticated.

[0104] At operation 912, the communication application on the local computing device 920 then sends a message with the PIN (or other sub-display screen identifier), the remote user id (or the content source device id), and information (e.g., address) about the display screen control system 940 to the communication service 960. The communication service 960 provides communication functions (such as chat, video call, audio call, online conference, etc.) to various devices running communication applications, such as the local computing device 920 and the remote content source device 910.

[0105] The communication service 960 receives the message and determines the address of the remote content source device 910 selected by the local computing device 920 at operation 914 - for example, by consulting a mapping between the remote content source device 910 identifier and / or the user identifier and the network address. Then, a notification is sent to the communication application instance running on the remote content source device 910 using the network address of the remote content source device 910. The communication application instance running on the remote content source device 910 can prompt the user to select an application or other content to be shared (if the user does not want to share the content, then reject the sharing proposal). Once the user of the remote content source device 910 selects the content, then at operation 916, the remote content source device 910 can begin streaming the content to the communication service 960, which can use the address transmitted by the local computing device 920 to establish a connection with the display screen control system 940.

[0106] At operation 918, the communication service 960 may forward the streamed content to the display screen control system 940 along with the PIN or other identifier of the sub-display screen provided by the application on the local computing device 920. The display screen control system 940 may then associate the communication service 960 with the sub-display screen identified by the PIN or other identifier (e.g., populate the content source device information with information about the remote content source device 910 or the communication service 960). At operation 918, the content sent from the remote content source device 910 may be sent to the display screen 930, where it is displayed in the sub-display screen.

[0107] Thus, content sent from remote content source device 910 can be displayed on display screen 930 by being subjected to communication service 960 and display screen control system 940. As described, a PIN or some other unique sub-display screen identifier can be used to associate specific content with a specific sub-display screen. By using communication service 960 as an intermediary, in addition to being displayed on a sub-display screen of display screen 930, content can also be sent to other participants of a network-based communication session (e.g., an online meeting) in parallel.

[0108] In some examples, rather than having a designation for a remote content source device 910, content currently being presented in a network-based communication session (e.g., an online meeting) may be automatically sent by the communication service 960 to the display screen control system 940. The display screen control system 940 associates the sub-display screen with the network-based communication session and causes the content sent by the communication service 960 to be displayed.

[0109] exist Fig. 9 In the embodiment of the present invention, communication service 960 is used to facilitate the display of content from remote content source device 910 to display screen 930. In other implementations, communication service 960 may not be involved. That is, a message with the PIN and display screen control system information may be sent directly from local computing device 920 to remote content source device 910. Then, the content may be sent directly from remote content source device 910 to display screen control system 940.

[0110] Once set up, actions taken by the local computing device 920 do not affect the content displayed in the secondary display screen by decoupling the content from being delivered through the local computing device 920. The local computing device 920 can leave the network-based communication session, and the content will continue to be displayed as long as the communication session continues for other participants.

[0111] Input from the sub-display screen may also be delivered to the remote content source device 910. For example, the input may be routed back through the display control system 940, routed back through the communication service 960, and finally routed to the application on the remote content source device 910. Thus, input entered by a local user—on the sub-display screen of the display screen 930—may appear on the remote content source device 910. In some examples, the communication service 960 and / or the communication application may have various rules regarding which inputs are allowed and which are not allowed. Although Fig. 9 The example uses a communication application and a communication service 960, but those skilled in the art having the benefit of this disclosure will appreciate that if other applications and services are implemented Fig. 9 If you have enabled the functionality of the other apps and services, you can use the other apps and services.

[0112] As in Fig. 9 As shown in , the local computing device 920 provides a PIN or other pairing information of the sub-display screen. In some examples, the local computing device 920 can provide any pairing information instead of a PIN. For example, biometric pairing. That is, the user of the local computing device 920 can enter its biometrics into the pairing input area of ​​the display screen. The display screen can scan the biometrics and send the biometric scan data to the display screen control system. The display screen control system can store the biometric scan data in the sub-display screen information 950. The communication service 960 can then pair with the sub-display screen by sending the biometric data of the user of the local computing device 920. The biometric data can be scanned by the local computing device 920 before or when the sub-display screen is created. In some examples, the biometric data of the user of the local computing device 920 can be stored at the communication service 960.

[0113] As previously described, the local computing device 920 can provide any pairing information. This can be further generalized by stating that the local computing device 920 can provide any sub-display screen identifier. Exemplary sub-display screen identifiers include a PIN number, an alphanumeric sequence, biometric information, a sub-display screen title, and the like.

[0114] In addition, as in Fig. 9As described in , the local computing device 920 can send the address of the display screen control system 940 discovered using a broadcast message (or using some other method) to the communication service 960. In other examples, the communication service 960 may already have the display screen control system address. For example, the display screen control system 940 can be an indoor conference room system that can be registered with the communication service 960. In some examples, the communication service 960 may know multiple display screen control systems 940 and can select an appropriate display screen control system to send content to based on the location of the local computing device 920 and the display screen 930 (and the display screen 930 controlled by the display screen control system 940), the selection of the local computing device 920, etc.

[0115] In some examples, the local computing device 920 may not be Fig. 9 930. That is, the local user can specify the sub-display screen and select the remote content source device from the UI of the display screen 930. The display screen 930 can pass information about the sub-display screen, including the specified remote content source device 910 (or the user of the remote content source device 910), to the display screen control system 940. The display screen control system can then send a message (e.g., message 912) to the communication service 960. The display screen control system 940 and the communication service 960 can communicate because the display screen control system 940 is registered with the communication service 960 and / or the display screen control system 940 is permitted to participate in network-based communications (e.g., online meetings) provided by the communication service 960.

[0116] Input sub display

[0117] As previously described, the sub-display screen can accept input that is forwarded to an application on a content source device paired with the sub-display screen. In some examples, a sub-display screen that does not display content but is dedicated to accepting input can be designated. In some examples, such a sub-display screen can show input (e.g., permanently or temporarily - such as for a predetermined time), but in other examples, it may not show input.

[0118] Fig.10A display screen with two designated sub-display screens, one dedicated to input, is illustrated according to some examples of the present disclosure. Sub-display screen 1040 may be a sub-display screen showing content streamed by application A 1030 on content source device 1020. Application A 1030 runs on content source device 1020. Input directed to sub-display screen 1040 (such as a touch gesture within sub-display screen 1040) may be sent to application A 1030. For example, the input may be input processed by application A 1030 as if it were local to content source device 1020. Input that is "local" to a device is input generated by an input device coupled through a local interface (e.g., Universal Serial Bus (USB), Peripheral Component Interconnect (PCI), etc.).

[0119] In contrast, the sub-display screen 1050 may be an input-only sub-display screen. That is, input from the sub-display screen 1050 may be sent to one or more applications running on a linked content source device (such as content source device 1020). This may include application A 1030 or an application different from application A 1030. In some examples, input directed to the sub-display screen 1050 may be processed in the same manner as input to the sub-display screen 1040—that is, processed as local input to application A 1030. In other examples, input to the sub-display screen 1050 may be related to content displayed in the sub-display screen 1040 and streamed by application A 1030, but the input may not be an edit to the content. For example, input to the sub-display screen 1050 may edit content that is supplemented to the content displayed in the sub-display screen 1040. Supplemental content is content that comments, adds to, or otherwise supplements the displayed content. As a more specific example, a user may display a slide show presentation in the sub-display screen 1040 and write notes for each slide in the sub-display screen 1050. The annotations may be saved in an annotation file for each slide. That is, the annotations may be saved in an annotation file corresponding to a specific slide shown in the sub-display screen 1040. More generally, input to the sub-display screen 1050 may be related to specific content shown in the sub-display screen 1040. Other specific examples include annotations about a displayed video (e.g., annotations related to the video based on playback time), a document being edited (e.g., the input is a comment in the document inserted at an active point of editing), etc.

[0120] In other examples, the content may be completely unrelated to the content displayed in the sub-display screen 1040. For example, input directed to the sub-display screen 1050 may simply be stored in a file or other storage device of the content source device 1020. Input directed to the sub-display screen 1050 may be stored in the data storage device 1042 and may be stored with the content streamed by the application A 1030 or stored separately.

[0121] In some examples, input directed to the sub-display screen 1050 may be displayed in the sub-display screen 1050. In other examples, input directed to the sub-display screen 1050 is not displayed in the sub-display screen 1050. In examples where input directed to the sub-display screen 1050 is not displayed in the sub-display screen 1050, this facilitates unbiased annotations because the input is not reproduced for public viewing. That is, during a presentation, the presenter can record thoughts about the presentation and then save them for later use without the audience being able to read the annotations.

[0122] In some examples, the destination of the input of the sub-display screen can be user-controllable. For example, once the content source device 1020 is linked to a given sub-display screen, the user can select from a plurality of options about where to send the input on any given sub-display screen and whether to display any input of the input. If the input modifies the content shared to different sub-display screens (for example, the input to the sub-display screen 1050 modifies the content shared in the sub-display screen 1040), the input can be displayed by updating the shared content. However, the input that does not edit the content shared on different sub-display screens may be displayed or may not be displayed-it depends on the implementation and / or user preferences. Whether to display the user selection of the input can be performed on the content source device 1020 (for example, in the user interface of the application communicating with the display screen control system (such as a streaming application or an operating system)) or on the display screen. For example, it can be selected when setting the sub-display screen, in the context menu of the sub-display screen, etc.

[0123] To create an input-only sub-display (such as input-only sub-display 1050), the user can use different gestures to create the input-only sub-display, can call a context menu during the creation of the sub-display to indicate that the display is input-only, can change the sub-display to an input-only sub-display after the sub-display is created, and so on. The user can also choose whether to display the input on the sub-display. That is, even if the sub-display 1050 does not display content, the system (e.g., the display, the display control system, or the content source device 1020) can also show the input on the sub-display. For example, if the user writes a note in the input-only sub-display, the sub-display can show the note. In other examples, the input is not shown on the sub-display.

[0124] In order to route input to the sub-display screen 1040 or the sub-display screen 1050, the content source device 1020 may have a table that maps input from a specific sub-display screen to a specific action. For example, a rule may specify that input directed to the sub-display screen 1040 will be directed to the application A 1030, as if it were a local input, and input directed to the sub-display screen 1050 will be directed to the annotation file associated with the currently displayed item of the content displayed by the application A 1030. The table may list the sub-display screen identifiers and where the input will be sent and / or how the input will be processed. When receiving input from the display screen 1010 or the display screen control system, the identifier of the sub-display screen that received the input is used to index the table to determine how to process the input.

[0125] notify

[0126] In some examples, the sub-display screen may be configured to display one or more notifications from the content source device. These notifications may be identified at the content source device 1020. For example, new email notifications for an email account associated with a user of the content source device; application notifications; communication notifications such as new message notifications; incoming call notifications; and the like. Notifications may be generated by applications running on the content source device. In some examples, these notifications may not be related to the display service. Notifications may also be generated by the application that shares the content in the sub-display screen that displays the notifications, or may be generated by a different application.

[0127] The notification may be displayed within the sub-display, or in a notification area adjacent to the sub-display, etc. Fig.11 An example of a sub-display screen 1110 with a notification area 1130 according to some examples of the present disclosure is illustrated. The notification area 1130 may be a pull-out, pop-up, or other area. In some examples, the notification area 1130 may not be part of the sub-display screen. In other examples, the notification area 1130 may be part of the sub-display screen. The sub-display screen 1110 shows the content of application A for the content source device 1. The content source device 1 receives or otherwise identifies information to generate notifications to the user according to instructions from one or more applications - for example, via a notification application programming interface (API) of the operating system of the content source device 1. In some examples, the content source device 1 may have rules that specify which notifications will be displayed to the user on the remote display screen and how to show these notifications. For example, the content source device 1 may provide a user interface that allows the user to select which notifications from which applications are displayed on the remote display screen. For example, which notification types are displayed. These rules may be different from the rules used to determine when notifications are generated in the first instance.

[0128] The content source device 1 then sends the notification indicator together with the content of the notification and the sub-display screen identifier (e.g., PIN) to the display screen or display screen control system. The display screen or display screen control system then shows the notification—for example, in the sub-display screen 1110, in a notification area (such as notification area 1130). In some examples, the notification area may be a visually distinct area that is adjacent to the outline of the sub-display screen 1110 and shares a border with it. The notification area 1130 may not overlap with the sub-display screen 1110. The notification may be displayed until the user dismisses it by pointing to an input (e.g., a gesture of a specific specified type) of the notification on the sub-display screen, dismissed on the content source device, or may be displayed for a predetermined period of time. The content source device may also send a message to the display screen or display screen control system to remove or otherwise modify the notification.

[0129] In some examples, notification area 1130 may allow input, such as a gesture that allows an action to be taken on the notification. For example, a predefined gesture may dismiss the notification - causing it to no longer be displayed. Other gestures may cause the application to be displayed on the content source device that generated the application. In response to a particular gesture, information about the notification may be displayed on the sub-display to which the notification is attached (on the Fig.11 The notification may be in a sub-display 1110 in the example, a different sub-display, or a new (automatically created) sub-display. For example, if the notification is a new email notification, a specific gesture may open the email.

[0130] In some examples, whether information about a notification is displayed in the sub-display associated with the notification, in a different sub-display, or whether it is displayed in a new sub-display can be based on a gesture. One gesture can open a new sub-display, while another gesture can display the information in an already open sub-display. For example, if the user touches the notification and drags the notification away from the sub-display (e.g., as if they are "tearing off" the notification) - the system can open a new sub-display to display information about the notification. In other examples, if the user taps the notification, the information about the notification can be opened in the current sub-display (e.g., sub-display 1110).

[0131] In some examples, a designated gesture may allow an inline reply to a notification related to a particular communication to be entered. For example, a gesture may cause an input field to be displayed to allow a user to enter text in an input box in the notification area to reply to an email or text message.

[0132] Notifications may be content-sensitive in that specific content on the notification may be obscured, or the notification may not be displayed based on the content. For example, if the notification is an email, the user may set the system so that emails from specific users, with specific keywords, or otherwise matching defined criteria may be obscured, not displayed, etc.

[0133] In addition to receiving notification indications from the content source device, the display screen and / or display screen control system can detect a notification window within the content shared by the content source device. The notification can be split into a notification sub-display screen. The notification sub-display screen can be docked to the main sub-display screen in the form of a notification area 1130. The display screen control system can modify, crop, or otherwise remove notifications from the shared content so that the notifications shown in the notification sub-display screen are the only instances of the notifications shown. The display screen control system can detect notifications as features using the methods described herein for detecting communication modalities.

[0134] Different sub-displays for different modalities

[0135] In some examples, the display system can automatically create one or more sub-display screens based on the context of activities, applications, and / or applications running on a specific content source device paired with the sub-display screen. For example, the content source device may be running a communication application that can provide multiple communication modalities. Exemplary modalities include network-based communications, such as chat communications, voice communications (e.g., Voice over Internet Protocol - VoIP), video calls, online conferences, file sharing, group chats, topic-based group chats, etc. A topic-based group chat can be a chat with one or more people organized by a specific topic or group (e.g., a team).

[0136] Fig.12An example of a graphical user interface (GUI) 1200 of a communication application according to some examples of the present disclosure is shown. A command and search bar 1210 allows a user to type one or more commands or search for files, communications, application features, etc. On the left, there are various icons for accessing various communication modalities provided by the communication application. The activity icon 1220, when selected, provides a list of recent communications involving the user or the group to which the user belongs. The chat icon 1230, when selected, provides the user with the function of initiating or participating in text communications with one or more other users. The team icon 1240, when selected, provides the user with the function of creating, modifying, managing, and participating in topic-based, group-based communications. The conference icon 1245, when selected, provides the user with the function of creating, managing, and participating in network-based communications (e.g., network-based conferences), wherein the user can share voice, video, and / or content with one or more users and with a display screen. The call icon 1250, when selected, provides the user with the function of calling another user and / or video calling other users using VoIP. The file icon 1260, when selected, allows the user to view files associated with one or more other communication modalities available to the user through the communication application. For example, such as files exchanged in one or more teams, files in a chat, files shared during a call or meeting, etc. In some examples, the file icon 1260 also allows users to share files between each other. The GUI 1200 shows a selected conference icon and a GUI of an active conference with a conference control bar 1265.

[0137] Fig.13 A display screen 1305 having multiple sub-display screens corresponding to multiple communication modalities of a communication application according to some examples of the present disclosure is illustrated. As can be appreciated and as disclosed herein, multiple independent sub-display screens can display different content from the same application of the same content source device. Fig.13 As shown in , the display screen 1305 has multiple sub-display screens, which are linked to the same application on the same content source device and are displaying different communication modalities (e.g., different content) of the application. Fig.13 middle, Fig.12 The communication application is to share different communication modes to different sub-display screens. For example, sub-display screen 1310 can show the content of the main display area of ​​the meeting in which the user is currently participating. At the same time, sub-display screen 1314 can show a chat window for a chat session corresponding to the meeting. The chat session corresponding to the meeting can be an automatically set up chat session that allows meeting participants to conduct text chats with each other while presenting content. Sub-display screen 1316 can display a chat thread that is not related to the meeting displayed in sub-display screen 1310. Sub-display screen 1312 can display one or more conversations from one or more teams in the team of the communication application. Fig.13 In the example above, the conversation is from Team Y.

[0138] Sub-displays showing different communication modalities can be created manually—for example, a user can select different modalities within a communication application and use the Figure 7 The mechanism shown in , where the "app" selection can include various communication mechanisms. Figure 7 In addition to the selection of "APP 1", "APP 2", "APP 3" shown in the figure, the selection screen may include various communication modalities, such as teams, chats, conferences, etc. In addition, the user may have an option in the context menu corresponding to the communication modality to share the modality in the sub-display. When selecting a specific modality, the communication application prompts the user for the pairing information of the sub-display, and then starts sharing the communication modality on the sub-display.

[0139] In other examples, the communication application and / or the display control system can automatically create one or more sub-displays to display one or more of the various communication modalities. For example, if a first sub-display linked to a communication application is set, other sub-displays can be automatically created and linked to the communication application, wherein each sub-display displays a different communication modality in the communication modalities provided by the communication application. In some examples, the system can prompt the user before automatically creating one or more sub-displays for one or more communication modalities, but in other examples, the system can automatically create sub-displays without user confirmation or input.

[0140] In some examples, the system can create a sub-display screen for each provided communication modality provided by the communication application. In other examples, the system can create a sub-display screen only for a specific communication modality. For example, a sub-display screen can be created for a communication modality based on a usage history. In these examples, a sub-display screen can be created for a pre-specified number of recently used modalities x. In some examples, x can be based on the amount of space available on the display screen. That is, the free space not occupied by the sub-display screen can be used to determine how many sub-display screens are opened for the communication modality, and the size of the sub-display screen. The sub-display screen may change based on the usage history. Therefore, if the first three most recently used communication modalities are shown, the content shown in the three sub-display screens can change as the use of the communication modality changes. Therefore, the content of the most recently used communication modality can replace the content of the less recently used communication modality in a specific sub-display screen. In some examples, a user can "fix" a specific modality so that it cannot be replaced by other communication modalities. As the use of the display screen changes (for example, the user turns off the sub-display screen), x may increase as the free space on the display screen increases, and additional modalities can be selected for display.

[0141] In other examples, other methods of selecting communication modalities, such as user preferences, can be utilized. A user can set one or more rules for sharing one or more of the modalities. For example, a user can specify that when a display screen control system is discovered and a first modality is linked, specific other modalities can be automatically displayed. In some examples, the conditions for determining which modalities to display may include a manually linked first modality (e.g., the displayed first modality may have other associated modalities to be displayed), communication activity and usage history of the modality, the user's location, the content of the modality (e.g., the system may exclude content with specific words, images, or user-defined sensitive materials), etc.

[0142] In some examples, the user's past sub-display history may be utilized to automatically select one or more modalities. For example, if the user has created three sub-displays in the past and linked modalities A, B, and C, the system may automatically create a sub-display and link modalities B and C (or prompt the user to ask if they would like to create sub-displays for modalities B and C) upon recognizing that the user has linked a sub-display to modal A.

[0143] Which modalities a user may be interested in sharing on a sub-display may be predicted by one or more machine learning models trained using training data that may be labeled with (or not labeled with) modalities that a user has manually paired with a sub-display. The training data may be per user, across all users, across user groups, and the like. The training data may include communication activity about available modes, such as last activity, total activity, frequency of activity, recent activity level, and the like; content of the modality, such as keywords, pictures, and the like; type of modality; and the like. The model may be a logistic regression, a neural network, a decision forest, a boosted decision tree, a support vector machine, a singular value decomposition (SVD) recommender, and the like. During use, current feature data about the modality, such as current communication activity, current content of the modality, and the like, may be used as input to the model, which may output a list of modalities that may be automatically created and / or suggested to the user and / or linked to an automatically created sub-display. The characteristic data may be current communication activity regarding available modalities, such as last activity, total activity, activity frequency, recent activity level, etc.; content of the modality, such as keywords, pictures, etc.; type of modality; and the like.

[0144] In some examples, the size of the automatically created sub-display screen can be predetermined or can be determined based on one or more features. For example, each modality may have a specific defined size. In other examples, each modality can have a specific minimum and / or maximum size, where the actual size is set based on the activity of the modality. For example, a chat modality can have a specific minimum size to display a chat. In addition, the size can be set or modified by the communication history of the modality, such as setting the size of the sub-display screen to display all messages received within a defined time period. In some examples, the size can be based on the available space on the display screen. For example, if the display screen has enough space to open a sub-display screen that can display ten past messages, the ten most recently received messages can be displayed.

[0145] As mentioned above, the system can suggest adding or automatically adding additional sub-display screens to the user, for example, to display additional communication modalities. More generally, the same process described above can also be applied to other content sources running on the content source device. The content source can include multiple content items (such as different communication modalities) on the same application, different applications, etc. For example, the system can suggest opening a sub-display screen for another application running on the content source device, or can suggest opening a sub-display screen for different content segments in the same application. Thus, if a web browser is displaying two videos, the first video is shared in a sub-display screen, the system can suggest sharing the second video in a second sub-display screen.

[0146] Fig.14 An illustration of a suggested additional sub-display screen 1410 according to some examples of the present disclosure is shown. Fig.14 In the example of , the system can suggest another sub-display screen 1410 that displays a different content source (e.g., a different communication modality) of the communication application. For example, the user may have defined a sub-display screen 1420 that shows the content of the meeting and a sub-display screen 1430 that shows the chat thread. The system can suggest another chat thread based on a combination of available sub-display screen space and additional content to be shared. For example, a conversation thread can become active by receiving a message from another participant in the thread or by a user sending a message in the thread.

[0147] As in Fig.14 As shown in , the system can provide suggestions by showing an outline or shaded version of the suggested sub-display screen, and can include a shaded version of the content to be displayed. Fig.14As shown in , an area of ​​the sub-display is designated to obtain the user's approval to accept the suggested sub-display. In some examples, if the user does not accept within a threshold time period, it is assumed that the user does not accept the suggested sub-display and the suggested sub-display 1410 disappears. In other examples, a rejection control is displayed that allows the user to reject the suggested sub-display. In some examples, the user can close these suggestions.

[0148] Fig.15 1500 . The data flow of a proposed sub-display according to some examples of the present disclosure is illustrated. The display control system 1540 can determine that additional space is available on the display 1530 (e.g., a minimum amount of space not currently occupied by a sub-display) for the additional sub-display. The display control system 1540 can send a notification 1502 to the content source device 1520 that the additional space is available.

[0149] In response, content source device 1520 may determine whether any additional content is available for display and whether the content should be displayed (automatically or suggested to the user). Content source device 1520 may apply the models, rules, etc. mentioned above to determine the suggested content. Content source device 1520 may also apply user-selectable settings and rules that may prevent certain content from being displayed or suggested, to prevent embarrassing or sensitive content from being suggested or displayed.

[0150] If additional content that meets the user's settings is available, the content source device 1520 can begin streaming the content to the display screen control system 1540. The display screen control system 1540 can send a message 1550 suggesting a sub-display to the display screen 1530. The message can provide the coordinates and size of the suggested sub-display. The display screen control system 1540 can create a sub-display record (such as in Fig. 9 The sub-display record may have one or more fields indicating that only the sub-display is suggested. The message may also include a preview of the content 1547. The display 1530 may display the sub-display as described in Fig.14 The recommended display is described in .

[0151] Display screen 1530 sends a response 1549 indicating whether the user accepts the suggested sub-display. If the user accepts the suggested sub-display, the sub-display record may be changed to reflect that it is no longer a suggested sub-display, but an actual sub-display. Display screen control system 1540 may begin sending additional content 1545 to display screen 1530. The display screen may then cause the sub-display to be displayed with the additional content 1545.

[0152] In addition to suggesting content for display in a sub-display, the system may suggest increasing or decreasing the size of an already created sub-display. Fig.16 A diagram of a display screen size change according to some examples of the present disclosure is shown. The user may have begun to specify a display area 1615 by touching the first upper left corner of the display screen 1610. When the user slides his finger to the lower right corner 1617 of the sub-display screen, the system recognizes that additional space is available on the display screen. A pop-up box 1619 may be displayed, indicating that additional space is available and providing a suggested display size 1620. If the user does not release his finger, they can continue to slide their finger to enlarge the sub-display screen until the suggested size (or some other size) is reached, and then by releasing his finger, the sub-display screen may be created as a rectangle with opposite vertices as the start and end touch points. In other examples, the user may release his finger (or use another finger) and tap the pop-up box 1619 to automatically increase the size of the sub-display screen to match the suggested sub-display screen size.

[0153] In other examples, the sub-display size may be suggested after the sub-display has been created. For example, the sub-display may be created, and then a prompt may allow the user to accept a suggested resize of the sub-display by using an input field (e.g., an on-screen button). The suggestion to resize the sub-display may be made based on the available space on the display and / or the suitability of the current size for the displayed content.

[0154] For example, if the size of the content is such that scroll bars exist, or if the content is scaled in a sub-display and there is enough room on the display to increase the size of the sub-display so that the content is no longer scaled or scroll bars are no longer needed, the system can suggest a larger sub-display.

[0155] In response to the display screen gaining free space by closing one or more sub-display screens, the suggested sub-display screens may increase. The display screen and / or the display screen control system may monitor the display screen and suggest one or more sub-display screens to increase their size based on the new free space and / or the content displayed in the particular sub-display screens.

[0156] In a previous example, a system was described that provides suggestions to a user to share additional communication modalities in separate sub-displays. These suggestions are identified by an application running on a content source device. In an additional example, instead of the application on the content source device identifying the additional modalities, the display control system can identify separate features (such as separate modalities or notifications) in the shared content provided by the content source device. The display control system can automatically split those modalities (with or without user approval) into different sub-displays.

[0157] For example, a content source device may share a window of a communication application on a sub-display screen. A display screen control system may receive the content and determine one or more features of the content before causing the content to be displayed. The display screen control system may split one or more of those features into its own automatically created separate and independent sub-display screens. The decision may be based on detected features, settings of the display screen control system, user settings, and the like. For example, user settings may specify features that will be split into their own sub-display screens and / or conditions for doing so (e.g., using if-then rules). The display screen control system may also cause the content displayed in the first sub-display screen to omit visual representations of features that are split into their own sub-display screens (e.g., by cropping the features). That is, the display screen control system may be trained to detect specific features in the content that are to be split into independent sub-display screens, and may not display those features in the content in the sub-display screen where the original content is to be displayed.

[0158] In some examples, features to be split can be specified via explicit user preferences. In other examples, past user history can be used to determine the features to be split. That is, if a user often manually splits a chat session of a communication application into its own sub-display screen, the system can predict that the user will do the same thing in this case and can automatically perform the action. For example, the system can train a machine learning model based on detected shared content features and labels indicating whether the user has split those features into its own sub-display screen in the past. The model can then predict whether the user is likely to split those features into its own sub-display screen in a given situation. If the user is likely to split those features into its own sub-display screen, the system can automatically split those features-or ask the user whether he wants the system to split those features. Exemplary machine learning algorithms may include logistic regression, neural networks, decision forests, decision trees, support vector machines, and the like.

[0159] Fig.43The content splitting process according to some examples of the present disclosure is illustrated. The screen of the communication application 4320 is shared by the content source device. The screen currently shows a network-based conference with conference content and a sub-display screen with a chat thread 4322. The chat thread 4322 can be related to the conference or can be a separate chat thread. The display screen control system 4325 can receive the content. As mentioned previously, the content can be information about the display screen (e.g., a graphical description of the display screen, a video frame, etc.), application data for recreating the display screen, conference content, chat content, etc. The display screen control system 4325 can detect the chat thread 4322 as a feature of the content, and can assign the chat thread 4322 to be displayed in the second sub-display screen 4335 on the display screen 4310. The rest of the content from the screen of the communication application 4320 is shown in the sub-display screen 4330-without the chat thread 4335. For example, the detected chat thread 4322 can be cut out from the screen of the communication application 4320 before being displayed in the sub-display screen 4330. In other examples, detected chat thread 4322 can be redacted (e.g., blacked out), deleted from content, or otherwise not displayed. In some examples, detected chat thread 4322 can still be displayed in sub-display 4330.

[0160] In an example where the content is video frame content, the display screen control system can utilize a computer image recognition algorithm to identify features (e.g., objects) within one or more video frames or images. For example, one or more features are detected by adopting exemplary models of various features. Features can include communication modalities, notifications, different aspects of a graphical user interface, help prompts, comments, revisions, edits, individuals (e.g., different people displayed in a communication application), etc. For example, the system can split the video feeds of multiple different people or roles (e.g., active speaker roles) of a network-based communication (such as an online meeting) into their own sub-display screens. For example, the system can split the video feeds of each individual in the network-based communication into their own sub-display screens.

[0161] Exemplary algorithms for detecting features in a video may include machine learning methods that use methods such as the Viola-Jones object detection framework based on Haar features, scale-invariant feature transform (SIFT), or histogram of oriented gradients (HOG) features to define the features that the system looks for. These techniques can use a support vector machine classifier on the video to classify the video and find features. Other exemplary algorithms may include deep learning schemes, such as neural networks or convolutional neural networks.

[0162] As previously described, the location of the features can be detected and used as content for the independent sub-display. The size of the independent sub-display can be set based on the size of the extracted features. The extracted features can be scaled up and / or down based on the available space on the display.

[0163] In other examples where the content is not video-based content, the display screen control system 4325 can search the content for features specified by user settings, system settings, or otherwise specified for splitting into separate sub-display screens. For example, the description of the content used to render the content (e.g., content describing the user interface 4320) can be searched for various feature indicators, such as a title or other portion depicting a notification or various communication modalities.

[0164] Although the features described above are communication modalities, other features may also be utilized, such as notifications. For example, when a user shares their screen and a notification from an application pops up on their desktop, the display control system may detect the notification and create a new sub-display. In some examples, the new sub-display may be displayed with information about the Fig.11 The notification label shown in 1130 is "docked" to the old sub-display. According to the discussion in this article, the user can provide one or more gestures and inputs to the notification.

[0165] In some examples, in the case of a communication modality, the sub-display screen may be paired with a communication server providing a communication service. That is, a device running an instance of a communication application associated with a communication service may input instructions into the communication application to cause the communication server providing the communication service (or some other server accessing the communication session) to pair with the sub-display screen. For example, Fig. 9 , but without the remote content source and instead with the communication service 960. The communication service 960 can then share various content associated with the communication session on the sub-display screens. In these examples, the communication service 960 can communicate with the display screen control system to open additional sub-display screens based on various communication modalities that may be of interest to the user who can view the display screen. These predictions can be made in the same manner using the same methods and factors as described for the communication application example for making these predictions.

[0166] Display screen control system

[0167] Fig.17The logical diagram of the display screen control system 1700 according to some examples of the present disclosure is illustrated. The service advertising component 1710 can provide broadcast information about the display service provided by the display screen control system to both the display screen and the content source device. The display screen registration component 1705 can register the display screen and create a display screen information data structure 1750. The sub-display screen creation component 1715 can create a sub-display screen information data structure 1770 and assist in creating a sub-display screen. The content routing component 1720 can receive content and route it to the correct sub-display screen. The input routing component 1725 receives input from the sub-display screen and routes it to a specific content source device. The sub-display screen pairing 1730 can provide a sub-display screen pairing with a content source device. The notification component 1735 can provide one or more notifications on the display screen. The suggested sub-display screen component 1740 can suggest one or more additional sub-display screens and / or change the size or configuration of one or more sub-display screens. Each of these components will be discussed in more detail in the following discussion.

[0168] Fig.18 The method flow of service advertisement according to some examples of the present disclosure is illustrated. At operation 1805, the advertisement component 1710 may broadcast a service availability message to a broadcast address through a network. The service availability message may include a network address of a display screen control system, an indication that the display screen control system can be used to control a display screen / or an indication that the display screen control system can be used to provide a sub-display screen sharing service to a content source device.

[0169] The display screen that is looking for the display screen control system can respond with its capabilities and requests to be controlled by the display screen control system 1700. Note that the content source device does not need to reply to the broadcast message, because the content source device only needs to communicate with the display service provided by the display screen control system when it is paired with a child display screen. The display screen registration component 1705 can handle the response from the display screen. Return to reference Fig.18 At operation 1810, the display screen registration component 1705 may receive a registration message from the display screen. The display screen registration component 1705 may register the display screen by creating an entry with display information in the display information data structure 1750 at operation 1815. At operation 1820, a confirmation message may be sent to the display screen.

[0170] Exemplary display information may include a display screen identifier 1755 that is unique across a network or within the display screen control system 1700. The display screen identifier 1755 may be assigned by the display screen control system 1700, the display screen, or some other entity. Communications between the display screen control system 1700 and the display screen may include the display screen identifier 1755 to allow the display screen control system 1700 to find the display screen information data structure 1750 for the correct display screen. Other exemplary display screen information may include a display screen address 1760 that is the network address of the display screen, display screen capabilities 176 such as resolution, color capabilities, refresh capabilities, input capabilities, input type, etc. The sub-display screen information data structure 1770 may be empty initially, but may hold information about the sub-display screens once they are created.

[0171] The display screen can send a notification when creating a sub-display screen. The sub-display screen creation component 1715 of the display screen control system 1700 can process the notification.

[0172] Fig.19 A method flow for processing a sub-screen creation notification according to some examples of the present disclosure is illustrated. At operation 1905, the display screen control system 1700 may receive a sub-screen creation message from a display screen. The sub-screen creation component 1715 may process the message at operation 1910 by determining a display screen information data structure 1750 associated with the display screen using a display screen identifier sent by the display screen in the sub-screen creation message. At operation 1915, the sub-screen creation component 1715 may create a sub-screen information data structure 1770 within a display screen information structure associated with the display screen from which the message was received determined at operation 1910, or a sub-screen information data structure 1770 linked to the display screen information structure. At operation 1920, the sub-screen creation component 1715 may send a confirmation to the display screen. Fig. 9 An example of sub-display screen information 950 that may be stored is shown.

[0173] Sub-display pairing component 1730 can process pairing requests from content source devices.

[0174] Fig.21A flow chart of a pairing operation according to some examples of the present disclosure is illustrated. At operation 2105, the sub-display screen pairing component 1730 may receive a pairing request from a content source device. At operation 2110, the sub-display screen pairing component 1730 may match the pairing information provided by the display screen and stored in the sub-display screen information data structure 1770 with the pairing information sent by the content source device to determine which sub-display screen is paired with the content source device. If a match is not found, an error may be returned to the content source device. If a match is found, at operation 2115, the sub-display screen pairing component 1730 may update the sub-display screen information data structure corresponding to the matched sub-display screen to store information about the content source device. At operation 2120, the sub-display screen pairing component 1730 may send a confirmation to the content source device.

[0175] As previously described, the pairing information may be a PIN that may be created by the display and provided by the display to the display control system in the sub-display creation notification. Alternatively, the display control system 1700 may create the PIN and provide it to the display. The PIN created by the display control system 1700 may then be passed back to the display in the sub-display creation confirmation message (e.g., operation 1920) to allow the display to display the PIN in the pairing tag. The PIN may be a unique identifier used to match a content source device to a specific sub-display.

[0176] In other examples, the pairing information may be based on biometrics. In these examples, since a PIN number is no longer used to uniquely identify a sub-display, the sub-display creation component 1715 may issue a unique identifier (ID) for the sub-display, store the ID in the sub-display information, and provide the ID to the display. In examples where the sub-display identifier is unique to all sub-displays managed by the display control system 1700, the display control system 1700 may then uniquely identify any sub-display by the sub-display identifier. In examples where the sub-display identifier is unique only to each display managed by the display control system 1700, the display control system 1700 may uniquely identify any sub-display by the sub-display identifier and the display identifier.

[0177] In the example where the pairing information is a biometric template - as in Figure 7The sub-display screen creation notification may include the first biometric scan information and the sub-display screen identifier, as discussed in more detail in the pairing tag 730 in FIG. 1 . The biometric scan information (e.g., the biometric template) may be temporarily stored in the corresponding sub-display screen information. The pairing request from the content source device may also include the second biometric scan information, which may be obtained during the pairing process at the content source device, or may have been previously obtained at the content source device, such as during the account setup process.

[0178] In these examples, to match the received pairing information with the sub-display screen information structure (e.g., operation 2110), the display screen control system 1700 can traverse the display screen information data structure 1750 and the sub-display screen information, checking the unpaired sub-display screen information for the stored biometric information that matches the second biometric scan information from the content source device. If a match is found, the content source device is paired with the sub-display screen having the matching biometric scan information. If no match is found, a failure message can be sent to the content source device.

[0179] Once a matching sub-display is found, the content source device address may be stored in the sub-display information, and the sub-display identification (if it is not PIN-based) and / or the display identifier may be communicated back to the content source device.

[0180] The content routing component 1720 can process the content sent from the content source device and cause it to be displayed in the sub-display screen. The content can include a sub-display screen identifier and / or a display screen identifier.

[0181] Fig. 20 A flow chart of a method for processing content from a content source device at a display screen control system according to some examples of the present disclosure is illustrated. At operation 2005, a content routing component 1720 may receive content from a content source device. At operation 2010, the content may include a sub-display screen identifier and / or a display screen identifier, which may be used by the display screen control system to identify a display screen information data structure. If the display screen information data structure 1750 is found, the display screen address may be determined at operation 2015. If the display screen information data structure 1750 is not found, an error message may be returned to the content source device. If the display screen information data structure 1750 is found and the address is located, at operation 220, the content may be sent to an appropriate sub-display screen at the located display screen. In some examples, the display screen control system may verify that the source of the content is a content source device paired with a sub-display screen by verifying that the return address of the content matches the content source device address in the sub-display screen information. In some examples, the display screen control system may verify that the content is directed to a valid sub-display screen of the display screen.

[0182] The input routing component 1725 can receive input from a display screen or input device and route it to a content source device.

[0183] Fig. 22 A flow chart of a method of routing input to a content source device according to some examples of the present disclosure is illustrated. At operation 2205, the input routing component 1725 can receive an input indicator regarding a sub-display screen. The input indicator can be sent by the display screen (if the input device is communicatively coupled to the display screen) or via an input device (e.g., a camera) coupled to the display screen control system. The indication can include the type of input to the controller (e.g., gesture type), the coordinates of the input within the sub-display screen, a sub-display screen identifier, a display screen identifier of the display screen, etc. If the coordinates of the sub-display screen are not known to the display screen or the input device, the indication can include the coordinates of the input within the display screen itself, which are used by the controller to determine the sub-display screen.

[0184] For camera-based gesture recognition for determining the sub-display screen identifier of the input, the sub-display screen identifier may be displayed by, on, or in the display screen. The computer vision application can read the sub-display screen identifier of the sub-display screen on the display screen from the captured image of the display screen. The identifier may be displayed continuously so that humans can view and read it, or it may be displayed in a manner that humans cannot read it but the machine vision application can read it. For example, by displaying the identifier only on a limited number of consecutive frames-for example, if the refresh rate of the display screen is 60 frames per second, the identifier may be displayed only on 1 frame per second. This may be too fast for the human eye to see the identifier, but the identifier is visible to the camera. In other examples, the camera can determine when a new sub-display screen is created and capture the identifier. The camera can then track the location and position of the sub-display screen after creation. Gestures on or pointing to the sub-display screen can be captured and recognized by the camera, and the corresponding sub-display screen identifier can be determined and sent to the controller.

[0185] Similarly, if the display screen itself detects the input, the display screen can store and track the sub-display screen identifier and the sub-display screen coordinates relative to the display screen. When the gesture input is on, within, or directed to the sub-display screen, the display screen can send the input to the display screen control system.

[0186] Return to Fig. 22 At operation 2210, the input routing component 1725 may determine the content source device paired with the sub-display screen and the network address of the content source device based on the information provided by the input indicator and based on the sub-display screen information data structure 1770. At operation 2215, the controller may send the input to the paired content source device.

[0187] The notification component 1735 can provide notification functionality to the sub-display screen.

[0188] Fig.23 A flowchart of a method for providing notifications to a sub-display screen according to some examples of the present disclosure is illustrated. At operation 2305, a notification component 1735 may receive a notification indication from a content source device. The notification indication may include a sub-display screen identifier, a display screen identifier, the text of the notification, one or more graphics displayed with the notification, a graphical layout of the notification, and the like. In some examples, at operation 2310, the notification component 1735 may determine the display screen information based on the sub-display screen information. At operation 2312, the notification component 1735 may create a notification identifier and store the notification identifier together with the sub-display screen information of the sub-display screen in which the notification is to be displayed. In some examples, the notification identifier may be sent back to a content source device provided for future interactions with the notification. This allows multiple notifications to be displayed using a single sub-display screen and modifications to the notification by a content source device.

[0189] At operation 2315, notification component 1735 can forward the notification to the display screen. The notification can include a notification identifier. In some examples, the display control system or the display screen can reformat the notification to match the display capabilities of the display screen and the notification area. For example, the notification can be resized, color information can be modified, etc. Specific operations can be applied to the notification based on a series of rules. The rules can depend on notification information, display information, available notification area, etc. For example, if <notification size> is greater than <notification area size>, then <scale the notification>.

[0190] The notification component 1735 can also process received input for notifications.

[0191] Fig.24A flowchart of a notification component 1735 for processing input to a notification according to some examples of the present disclosure is illustrated. At operation 2405, the notification component 1735 may receive an indication of input directed to the notification. The indication of input directed to the notification may include the type of input to the controller (e.g., gesture type), the coordinates of the input in the notification area, a sub-display screen identifier, a display screen identifier of the display screen, a notification identifier, etc. In some examples, the notification component 1735 (or the display screen) may process the input directly. For example, the notification component 1735 may process the input of dismissing the notification by instructing the display screen to remove the notification and stop displaying the notification area. The notification component 1735 may then determine the address of the content source device (e.g., based on the display screen and / or sub-display screen identifier) ​​at operation 2410, and send the indication to the content source device at operation 2415. In other examples, the notification component 1735 may send the input indication received at operation 2405 to the content source device without taking any action. The content source device may then take action (not shown) on the notification, including dismissing the notification or other action by sending a notification indication with a notification identifier specifying the desired action.

[0192] The suggested sub-display component 1740 can monitor the available space on one or more display screens, and can determine the increase and / or decrease of the suggestion for the sub-display screen based on the content shared in the sub-display screen. The suggested sub-display component 1740 can identify one or more content transformations for adapting the content to one or more sub-display screens. The suggested sub-display component 1740 can determine that there is enough unused space on the display screen (for example, based on the space between adjacent sub-display screens) to make one or more sub-display screens larger, thereby reducing or eliminating the transformation applied to the content to adapt the content to one or more sub-display screens. For example, the suggested sub-display component 1740 can apply one or more rules, such as a decision tree, to determine the optimal sub-display screen size for the content of one or more sub-display screens. The suggested sub-display component 1740 can then use additional rules to determine the new size to be suggested to the user. The new size can be sent to the display screen to suggest to the user. If the user accepts, the display screen and the display screen control system can update its record of the sub-display screen with the new size, and update the size on the display screen. Then, the transformation applied to adapt the content in the sub-display screen can be adjusted based on the new size.

[0193] The sub-display screen component 1740 of suggestion can also suggest additional sub-display screens.For example, the sub-display screen component 1740 of suggestion can send the indication that the space on the sub-display screen is available to the content source device.The content source device can then respond with any additional content that the content source device wants to suggest to the user, for example, a preview of the content that is suggested to be displayed.The sub-display screen component 1740 of suggestion forwards it to the display screen for displaying visual suggestions to the user.If the user approves to show the content, the sub-display screen component 1740 of suggestion receives the indication of acceptance from the display screen or from the input device.The indication can then make the sub-display screen component 1740 of suggestion indicate the sub-display screen creation component 1715 to create a new sub-display screen and the sub-display screen pairing component 1730 to automatically pair the sub-display screen with the content source device.The pairing can be completed automatically, and the pairing step (for example, displaying or collecting pairing information) can be skipped.Confirmation can then be sent back to the content source device-it can start sharing the content.

[0194] Fig.25 A logical diagram of a display screen 2500 according to some examples of the present disclosure is illustrated. A display screen registration component 2505 can register a display screen with a display screen control system. A sub-display screen creation component 2515 can create a sub-display screen. An input handler 2525 can process one or more inputs to the display screen. A notification handler 2540 can process one or more received notifications. A content display screen 2520 can process displaying content in a particular sub-display screen. A suggested sub-display screen 2530 can process suggestions for additional sub-display screens or for changing the size or other parameters of an existing sub-display screen. Each of these components is described in more detail in the following discussion.

[0195] Fig.26 The method of registering a display screen with a controller according to some examples of the present disclosure is illustrated. At operation 2610, the display screen registration component 2505 can monitor broadcast messages at a specific network address and / or port. If a service broadcast message is received at operation 2620, the display screen can negotiate a display screen control system to control the display screen at operation 2630. This can include sending registration information, such as display capabilities and addresses. The display screen can store information about the display screen control system, such as the network address of the display screen control system. Although the examples mentioned in this article have a display screen control system that sends advertisements, in other examples, the display screen and / or content source device can discover the display screen control system by sending its own advertisements and / or discovery messages that the display screen control system listens for and responds to.

[0196] The sub-display creation component 2515 can recognize input to create one or more sub-displays and communicate with the display control system to create the sub-displays.

[0197] Fig. 27 A flowchart of a method for creating a sub-display screen performed by a sub-display screen creation component 2515 according to some examples of the present disclosure is illustrated. At operation 2710, the sub-display screen creation component 2515 can identify a gesture indicating the creation of a sub-display screen. For example, a specific gesture points to an outer area of ​​an existing sub-display screen or on the outer area. Specific gestures can include finger down (e.g., a finger is placed and held on the display screen) and drag. The path of the user's finger on the screen specifies the coordinates of the sub-display screen. Another gesture may include a first finger down to indicate a first diagonal point and drag and a finger up (e.g., the user lifts his finger up), wherein the position of the finger up is the diagonal of the diagonal line. As previously described, the display screen can animate the creation of the sub-display screen. Gestures or other inputs can be determined by an input device (such as a touch screen) of the display screen or an input device (such as a camera) that is communicatively coupled to the display screen or a controller (which can pass input to the display screen).

[0198] After completing the gesture, the display screen can create one or more data structures to store information about the sub-display screen and track the sub-display screen. In some examples, the data structure can include the coordinates of the sub-display screen, including the coordinates of the outline, the sub-display screen identifier, etc. In some examples, once created, the sub-display screen can be moved, resized, rotated, closed, etc. The data structure can track the location of the sub-display screen on the display screen.

[0199] At operation 2720, the display screen can determine pairing information. For example, the display screen can assign a PIN number or other unique identifier to the sub-display screen. In some examples, the unique identifier of the sub-display screen is a PIN, but in other examples, an additional sub-display screen identifier can be assigned to the sub-display screen. In other examples, the display screen can request and receive a PIN or other identifier from the display screen control system. At operation 2730, the display screen can show pairing information such as within a pairing tag. In an example where the pairing method is a biometric pairing method, operation 2720 can simply determine an identifier for the sub-display screen, and operation 2730 can display instructions for the user to enter its biometrics. In these examples, another operation can be performed, whereby the display screen communicates with a biometric scanner to obtain a biometric sample of the user. In some examples, the biometric scanner can be instructed to scan only at a specific location within the display screen-such as at a pairing tag. The coordinates of the pairing tag can be provided to the biometric scanner. At operation 2730, the sub-display screen information can be sent to the display screen control system along with any pairing information and biometric information (depending on the implementation). Sub-display information, coordinates, pairing (eg, PIN and / or sub-display identifier), and biometric information may be stored in a data structure for the display.

[0200] The content display component 2520 can determine the correct sub-display screen for the received content and cause the content to be displayed within the sub-display screen.

[0201] Fig.28A flowchart of a method in which a content display component 2520 enables content to be displayed according to some examples of the present disclosure is illustrated. At operation 2810, a display screen may receive content, for example, from a content source device and / or a display screen control system. The content may include a sub-display screen identifier (e.g., a pairing PIN or some other identifier unique to the display screen or unique to the display screen control system). At operation 2820, the sub-display screen in which the content is to be rendered may be determined based on the sub-display screen identifier received with the content (e.g., in a packet header, a stream identifier, etc.) and the address of the content source device, etc. At operation 2830, based on the record of the display screen for the sub-display screen, the coordinates of the sub-display screen may be determined. In some examples, the content may be reformatted to fit within the sub-display screen. This may include scaling, rotating, translating, stretching, cropping, or otherwise transforming the content. In some examples, the content may be decoded before transforming and / or displaying the content. For example, the content may be encoded in one or more encoding formats according to one or more encoding schemes. The display screen (or display screen control system) may decode the content into a native format for rendering on the display screen. At operation 2835, the content may be displayed in the sub-display screen.

[0202] The suggested sub-display screen component 2530 can receive one or more suggestions from the suggested sub-display screen component 2530 of the controller. The suggested sub-display screen component 2530 can advertise one or more suggestions for changing the size and / or position of one or more sub-display screens on the display screen, and present one or more new sub-display screen creation suggestions. The user can accept these suggestions by providing input. The suggestion can be displayed as a transparent, phantom, dotted line or other visually depicted prompt to preview the new size, position, shape or new sub-display screen (including the content of the suggestion). The user can tap a button or other input area to accept the change, and can have another user interface area to reject the suggested change. If the user accepts the change, the display screen adjusts the data structure of one or more sub-display screens to reflect the change.

[0203] The input handler component 2525 can process input directed to, in, within, or on one or more sub-displays.

[0204] Fig.29A flowchart of a method for processing input directed to a sub-display screen according to some examples of the present disclosure is illustrated. At operation 2910, the display screen may identify input directed to one or more sub-display screens, in one or more sub-display screens, within one or more sub-display screens, or on one or more sub-display screens. For example, the display screen may detect one or more gestures of a touch screen component of the display screen. In other examples, a camera communicatively coupled to the display screen may send a notification that a gesture or other input has been detected. The type of input (e.g., the type of gesture) and the coordinates of the gesture may be determined. At operation 2920, the coordinates may be used to determine which sub-display screen the input belongs to. The display screen may then convert the coordinates of the gesture relative to the complete display screen (e.g., display screen 120) into relative coordinates within the sub-display screen. At operation 2930, these coordinates, the input type, the coordinates of the input within the sub-display screen, etc. may be sent to the controller.

[0205] The notification handler component 2540 can process notifications sent from content source devices and / or display screen control systems and process input directed to the notifications.

[0206] Fig.30 A flowchart of a method in which a notification handler component 2540 processes a notification message from a content source device or a display screen control system according to some examples of the present disclosure is illustrated. At operation 3010, the notification processor component 2540 may receive a notification indication from a display screen control system or a content source device. The notification indication may include content for displaying the notification. The notification indication may include a notification identifier and a sub-display screen identifier. At operation 3020, the display screen may identify the sub-display screen, for example, based on the sub-display screen identifier. At operation 3030, the display screen may render a notification area (e.g., a notification label). At operation 3035, the notification may be displayed in the notification area.

[0207] Fig.31 A flowchart of a method for a notification handler component 2540 to process a gesture directed to a notification area according to some examples of the present disclosure is illustrated. At operation 3110, the display screen may recognize a gesture directed to, on, within, or other input of a notification. At operation 3120, a sub-display screen corresponding to the notification may be identified. Additionally, at operation 3130, the notification identifier may be determined. These may be determined by matching the coordinates of the gesture with the coordinates of an active sub-display screen and / or an active notification area of ​​an active sub-display screen. At operation 3135, the notification may be sent to a display screen control system and / or a content source device.

[0208] Fig.32 A diagram of a content source device 3210 is illustrated according to some examples of the present disclosure. Fig.32One or more of the components of can be part of an application, such as a communication application. The user interface component 3215 can provide one or more user interfaces, such as a graphical user interface (GUI), to allow a user to select content for sharing, select a remote display, and pair with a remote display. The user interface component 3215 can also provide a GUI for selecting which notifications to display on the remote display.

[0209] The remote display sharing component 3220 can identify content, for example, based on a user selection through the user interface component 3215, and send the content to a display control system or a sub-display of the remote display based on a pairing operation performed by the remote display pairing component 3225.

[0210] The remote display pairing component 3225 can pair with the remote display by sending the pairing information (PIN, biometrics, etc.) collected through the user interface component 3215 to the display control system or the remote display. In some examples, the remote display pairing component 3225 can send one or more biometric templates (e.g., fingerprints, eye scans, facial recognition scans, etc.) obtained by the content source device from the user. These templates can be obtained during the application setup process, before the pairing process, or as part of the pairing process. These templates can be compared by the display control system or the remote display with the templates obtained at the remote display. If the templates match, the remote display control system and / or the remote display can pair the content source device with the sub-display on the remote display.

[0211] As previously described, pairing (i.e., linking) describes a process for associating a content source device with a specific sub-display screen to allow content to be routed from the content source device to the specific sub-display screen. The process requires matching one or more pieces of pairing information (e.g., PIN, biometric template, etc.) entered or shown on the display screen with one or more pieces of pairing information entered on the content source device. In some examples, the pairing is a way of providing a unique identifier to the content source device that can be attached to the content to properly route the content to the correct sub-display screen.

[0212] In other examples, the pairing may additionally create entries in the display control system or display that identify the sub-display (e.g., using a PIN or other unique identifier) ​​and the content source device (e.g., using an IP address or other unique identifier) ​​and / or an application on the content source device, such that data from the content source device is displayed in the correct sub-display and input from the sub-display is returned to the correct content source device and / or application on the content source device.

[0213] For an example where a single content source device shares content with multiple sub-displays (e.g., different applications share different content to different sub-displays), the content source device 3210 can record which applications and which content on the content source device are associated with each sub-display. The remote display sharing component 3220 receives the content from the application, consults the data structure to determine the address of the appropriate sub-display and display or display control system, and then sends the content to that address (along with the identifier of the sub-display).

[0214] For input returned to the content source device, the display screen input component 3230 of the content source device 3210 can query the data structure to determine the application to which the input is directed based on the sub-display screen identifier contained in the input. The display screen input component 3230 can then forward the input to the application. In one example, the input triggers an event and / or subroutine of the application in the same manner as the same input entered using a local input device. Local input is any input from an input device (e.g., a mouse, a touch screen of a content source device, etc.) connected to the content source device via a local bus.

[0215] The notification component 3235 can detect one or more notifications from one or more applications running on the content source device 3210. The notification component 3235 can compare the notification type, source application and / or notification component with a set of one or more rules specifying notifications to be displayed on one or more sub-displays of the shared content. The notification can be displayed in the notification area of ​​the sub-display. The notification may not be from the same application that displays the content in the sub-display. The notification can be created by the notification component, including a graphical layout of the notification and including a notification identifier.

[0216] Input to one or more of the notifications may be sent by a display screen and / or a display screen control system to the content source device 3210 and may be processed by a notification component 3235. The notification input indication may include a notification identifier. The notification component 3235 may utilize the notification identifier to identify the notification and may process the input. A specific input may dismiss the notification. A specific input may initiate the application that sent the notification at the content source device 3210, display the notification in a sub-display screen whose notification tag displays the notification (whether the application whose content is displayed in the sub-display screen generates the notification), or initiate a new sub-display screen using the notification. A specific input may generate a quick response input area on the notification tag. The notification component 3235 may send instructions back to the display screen and / or the display screen control system to modify or update the notification. For example, in response to input to a notification on a display screen or a content source device.

[0217] The display service 3240 can listen for broadcast messages from a display screen control system or display screen advertising that the display screen can be used to share in one or more sub-display screens. The broadcast message may include the address of one or more display screen control systems. The service may be local to a specific network. In some examples, multiple services may be discovered. During pairing, the content source device may attempt to pair with all discovered display screen control systems (because the content source device may not know which display screen control system controls the desired display screen where the specified sub-display screen is located). Pairing information is unlikely to match two different unpaired sub-display screens. In an alternative, each display screen may have a globally unique identifier that may be displayed together with or as part of the pairing information that the user may also enter. The display screen control system may broadcast the display screen identifier of the display screen managed by the display screen control system. The remote display screen pairing component 3225 may then use the broadcast information to select the appropriate display screen control system.

[0218] Fig.33 A flow chart of a method for sharing content on a sub-display screen of a remote display screen according to some examples of the present disclosure is illustrated. At operation 3310, the content source device may receive a selection of content to be shared from a user. For example, through a user interface provided by user interface component 3215. At operation 3315, the content source device may identify sub-display screen pairing information. For example, a user may enter a PIN, scan a QR code, enter a password, or provide other information displayed on a display screen (e.g., in a pairing tag) in a user interface element provided by user interface component 3215. In other examples, the content source device may identify one or more biometric templates of the user - for example, by retrieving it from a database, performing a new scan of the biometrics, etc.

[0219] At operation 3320, the remote display screen pairing component 3225 of the content source device can identify the display screen control system and the address of the display screen control system. As previously described, the display screen control system information can be obtained by monitoring broadcast messages. At operation 3325, the remote display screen pairing component 3225 can send a pairing request message to the address of the display screen control system together with the pairing information. Once the pairing is established (e.g., a confirmation message from the display screen control system), the remote display screen sharing component 3220 can start sharing content at operation 3330. As previously described, if multiple display screen control systems are found, the content source device can send multiple pairing requests to multiple display screen control systems.

[0220] Fig.34A flow chart of a method for processing input from a sub-display screen according to some examples of the present disclosure is illustrated. At operation 3410, the display screen input component 3230 may receive an input notification from a display screen control system or a display screen. The input notification may include a sub-display screen identifier, input type, location, and other information of the sub-display screen associated with the input. At operation 3415, the display screen input component 3230 may identify an application associated with the input. For example, by finding a data structure stored in association between an application and a sub-display screen identifier. At operation 3420, the input is sent to or otherwise delivered to the application.

[0221] Fig.35 A flow chart of a method for presenting notifications to a sub-display according to some examples of the present disclosure is illustrated. At operation 3510, a notification component 3235 may detect a notification. For example, an operating system may have a notification service to which an application sends notifications. The notification component 3235 may interface with the component. At operation 3515, the notification component 3235 may determine whether a notification setting allows a notification to be sent to a sub-display. If the notification setting does not allow sending to one or more sub-displays, processing of the notification for display on the sub-display may be terminated.

[0222] If the notification setting allows sending to one or more sub-display screens, at operation 3520, the notification component 3235 can select a sub-display screen associated with the content source device. If there is no sub-display screen associated with the content source device, the processing of the notification for display on the sub-display screen can be terminated. If only a single sub-display screen is associated with the content source device, then the sub-display screen can be selected. If there are multiple sub-display screens associated with the content source device, one or more selection methods can be used. For example, the content source device can select the sub-display screen from which the content source device recently received input. In other examples, the largest sub-display screen can be selected. In other examples, the display control system can determine which sub-display screen associated with the content source device to display the notification. In other examples, user preferences can be used, such as matching a specific notification type with a sub-display screen showing specific shared content. For example, a sub-display screen showing a communication application can display specific communication-related notifications.

[0223] Once a sub-display screen is selected, the notification may be sent to the selected sub-display screen at operation 3530. In some examples, an identifier is associated with the notification and sent with the notification.

[0224] Fig.36A flow chart of a method for processing input such as a gesture pointing to a notification displayed on a sub-display is illustrated. At operation 3610, the notification component 3235 can receive an indication of an input pointing to a notification sent by a content source device. As previously described, the indication can include information about the input, such as the type of input, the location of the input, a notification identifier, and other information. At operation 3615, the notification component 3235 can identify the application that generated the notification, for example, by using a notification identifier. At operation 3620, the input information can be sent to the application identified at operation 3615.

[0225] Fig.37 A flow chart of a method 3700 of specifying a sub-display according to some examples of the present disclosure is illustrated. At operation 3710, a display control system and / or a display may recognize a gesture input directed to a display surface of a display. The gesture input may define a geometry of a sub-display on the display. The gesture input defines a sub-display covering coordinates different from a previously specified sub-display. For example, the display may determine that the input is directed to a location outside of an existing sub-display and send a notification to the display control system in response to detecting a gesture that specifies a sub-display. The indication may occur after the gesture is completed (e.g., defining the geometry of the sub-display) or may occur in conjunction with the gesture.

[0226] At operation 3720, in response to the gesture input, the display screen control system and / or the display screen can create a sub-display screen, and the sub-display screen is linked to the content source device using a linking process as described herein. For example, one or more components of the system can create a data structure storing information about the sub-display screen. At operation 3730, in response to creating a sub-display screen and linking the sub-display screen with the content source device, the content shared by the content source device is displayed in the sub-display screen of the display screen. In some examples, the content can be shared simultaneously with different content shared in a previously specified sub-display screen. For example, the display screen control system can send content to a display screen with a sub-display screen identifier. The display screen can receive content directly from the content source device or from the content source device via the display screen control system, and the content can be rendered within the geometric structure established at operation 3710.

[0227] Fig.38 A flow chart of a pairing method 3800 according to some examples of the present disclosure is illustrated. At operation 3810, a display control system and / or a display may recognize a gesture input directed to a display, the gesture input specifying a geometry of a sub-display. For example, the display may send a notification to the display control system in response to detecting a gesture specifying the sub-display.

[0228] At operation 3820, in response to the gesture input, the display system may create a sub-display based on the defined geometry. As previously described, this may require creating one or more data structures and, in some examples, visually indicating the sub-display on the display. At operation 3830, the display may render and display a pairing information input area. For example, a pairing tag 630, 730, or 830 that allows entry of pairing information such as biometrics, a PIN code, and the like.

[0229] At operation 3840, the display screen or other input device may detect the entry of pairing information at the pairing information input area. The pairing information may include a PIN (e.g., via the previously described reverse pairing process, where the content source device determines the PIN and displays it to the user who enters it into the pairing information input area), a biometric scan of the user, authentication credentials for a user account, an alphanumeric value, etc. The pairing information may be stored in one or more data structures, such as a sub-display screen information data structure 950 in one or more data storage areas.

[0230] At operation 3850, the system may link the sub-display screen with the content source device based on matching the first pairing information received at the pairing input area with the second pairing information provided by the content source device. For example, the content source device may send a pairing request with the second pairing information. As with the first pairing information, the second pairing information may be a PIN, an authentication credential for a user account, an alphanumeric value, biometric scan information from a biometric scan performed by the content source device, and the like. For example, the system may search for a sub-display screen information data structure with matching pairing information. Once a match is found, the sub-display screen with the matching information is linked (i.e., paired) with the content source device.

[0231] At operation 3860, the display screen and / or the display screen control system may cause the content sent by the content source device to be displayed in the sub-display screen. For example, the display screen control system may send the content to the display screen having the sub-display screen identifier. The display screen may receive the content from the content source device directly or via the display screen control system, and may render the content within the geometry established at operation 3810.

[0232] Fig.39A flowchart of a method 3900 for inputting a sub-display screen according to some examples of the present disclosure is illustrated. At operation 3910, a display screen control system and / or a display screen may recognize a gesture input directed to a display screen, the gesture input specifying the geometry of a first sub-display screen. For example, the display screen may send a notification to the display screen control system in response to detecting a gesture specifying a sub-display screen. At operation 3920, in response to the gesture input, the display screen control system and / or the display screen may create a first sub-display screen. At operation 3930, the display screen and / or the display screen control system may cause content sent by a content source device to be displayed in the first sub-display screen. For example, the display screen control system may send content to a display screen having a sub-display screen identifier. The display screen may receive content directly from a content source device or from a content source device via a display screen control system, and may render the content within the geometry established at operation 3910.

[0233] At operation 3940, a second gesture input to the display screen specifying the geometry of the second sub-display screen may be identified as previously described for operation 3910. At operation 3950, the second sub-display screen may be linked to a content source device. At operation 3960, an input directed to the second sub-display screen may be identified. For example, by comparing the coordinates of the input with the coordinates of the sub-display screen on the display screen. At operation 3970, the input directed to the second sub-display screen is sent to the content source device. In some examples, the input may be hidden—that is, it is not displayed on the first sub-display screen and / or the second sub-display screen. As previously described, the input may be content displayed on the first sub-display screen, or may be directed to content that is linked to content displayed on the first sub-display screen. Although Fig.39 An example of creating a sub-display screen showing content and pairing the sub-display screen showing content with a remote content source device before specifying and pairing an input sub-display screen is shown, but in some examples, an input-only sub-display screen may be defined first. In other examples, only an input-only sub-display screen may be defined. That is, the disclosed method does not require creating a sub-display screen showing content and pairing the sub-display screen showing content with a remote content source device.

[0234] Fig.40A flowchart 4000 of a method for providing notifications according to some examples of the present disclosure is illustrated. At operation 4010, the display screen control system and / or the display screen may recognize a gesture input directed to the display screen, the gesture input specifying the geometry of the first sub-display screen, as previously described. At operation 4020, in response to the gesture input, the display screen control system and / or the display screen may create a sub-display screen and link the sub-display screen to a content source device using a linking (i.e., pairing) process as described herein. At operation 4030, content sent by the content source device is caused to be displayed in the sub-display screen, as described elsewhere herein.

[0235] At operation 4040, the content source device may send, and the display screen control system may receive information about notifications on the content source device, which, if displayed in the sub-display screen, will overlap with the content. For example, the content sent by the content source device for display in the sub-display screen may be a content stream (e.g., screen sharing), and a notification may pop up on the screen. The content source device may detect this (e.g., through an API of an operating system) and may send a notification to the display screen control system. The notification may include content below the notification position (e.g., content obscured by the displayed notification). The content source device may obtain content below the notification position by accessing a rendering pipeline for the content and copying the area below the notification location, for example, through an operating system. In some examples, the content sent by the content source device for display in the sub-display screen may have one or more layers, which may include a notification layer to be rendered on the content. The notification at operation 4040 may indicate to the display screen control system that there is a notification layer in the content. In yet other examples, the content received from the content source device may not be a notification, and operation 4040 is not a message from the content source device, but rather a machine vision technique as described herein is used to detect the notification.

[0236] At operation 4045, the display screen control system may cause the notification received at operation 4040 or within the content in the notification area to be displayed. The notification area may be another sub-display screen (which may be automatically created), a notification area such as notification area 1130, and the like.

[0237] At operation 4050, the sub-display screen is caused to display the content sent by the content source device, and the notification does not overlap with the content. That is, the notification layer may not be shown, and the notification in the content may be replaced by the received content (e.g., from the computing device) below the notification, etc. In some examples, the display screen control system or the content source device may determine the content below the notification based on the content that was in that position before the notification. In other examples, the content source device does not include the notification in the content sent for display in the sub-display screen, and the notification is not displayed therein, but is displayed in the notification area (region).

[0238] Fig.41 A flow chart of a method 4100 of designating a sub-display screen area for a remote content source device that is not physically close to a display screen is illustrated according to some examples of the present disclosure. Fig.41 The communication server that provides the communication service (for example, MICROSOFT Service server) (e.g., communication service 960) to execute.

[0239] At operation 4110, the communication server may receive a first message over a network. The first message may be received from a computing device (e.g., a computing device local to a display screen) or a display screen control system. In some examples, the first message may include a sub-display screen identifier and an identifier of a remote content source device. In some examples where the first message is not received from the display screen control system and / or the communication server may not yet know the display screen control system, the first message may include an address of the display screen control system, which is used by the communication server to reach the display screen control system. In some examples, the first message links the sub-display screen to the remote content source device. That is, the message may be an indication that the display screen control system has linked the sub-display screen to the remote content source device. In other examples, the message may indirectly link the sub-display screen to the remote content source device (via later message delivery).

[0240] As noted, the first message may be received from a computing device that is not a display screen control system (e.g., a computing device of a user of a designated display screen), or may be received directly from the display screen control system (e.g., a user of a designated sub-display screen may also input a remote content source device through the display screen's U / I). In some examples, the identifier of the remote content source device may be a user identifier—that is, the remote content source device is identified by a user who authenticates to the device. The authentication may be determined by a communication server providing a communication service.

[0241] At operation 4120, the communication server may send a second message to the remote content source device. The message may indicate that the sub-display screen is linked to the remote content source device. The remote content source device may receive the message and prompt the user to select content to be shared (or refuse to share).

[0242] At operation 4130, the content is received by the server from a remote content source device. At operation 4140, the communication server may cause the content to be displayed in the sub-display screen by sending the content to a display screen control system over a network. The communication server may include an identifier of the sub-display screen as well as the content. By including the sub-display screen identifier within the content, the communication server specifies in which sub-display screen the content is to be displayed. Including the sub-display screen identifier within the content may include placing the sub-display screen identifier within the content, as metadata for the content, in a header of a packet sent with the content, and the like.

[0243] Fig.41 Covering such as Fig. 9 , and scenarios such as where the local computing device 920 does not participate in the processes described herein, but instead a message (e.g., message 912) is sent by the display control system 940. The display control system 940 may be known to the communication service 960 because the display control system 940 participates in one or more network-based communication sessions.

[0244] As used herein, content sharing refers to the process of sharing a piece of content on a first device with another device. The sharing can be performed simultaneously with the presentation of the content on the content source device, so that the presentation of the content on the other device can occur within a determined time period after the content is displayed on the content source device or sent from the content source device. In some examples, content sharing can be content streaming or content projection.

[0245] As used herein, communication modalities refer to different communication methods. For example, email, text messaging, chat messaging, video calls, and telephone calls are all different communication modalities. A communication application may provide multiple communication modalities, and may even provide multiple communication modalities during the same communication session.

[0246] As used herein, pairing information can be a PIN for referring to a sequence of numbers and / or letters for matching a specific sub-display screen with a specific content source device. In some examples, the PIN and the content sent by the remote content source device can then be provided to uniquely identify the destination sub-display screen. In other examples, once the pairing is completed, some other identifiers can be utilized. For example, the display screen control system can have a mapping between the Internet Protocol (IP) address of the content source device and the sub-display screen determined by the pairing process (e.g., by requesting to share the content including the PIN sent by the remote content sharing device). In other examples, the PIN may be unique or may not be unique, and after pairing, a new unique identifier is created and transmitted to the content source device and stored with the sub-display screen information. The content source device can attach the unique identifier to the content.

[0247] For example, when biometric pairing is utilized, the display control system, display, and / or content source device may create a unique identifier for the sub-display during pairing and then use the unique identifier to reference the sub-display when sending content or performing other display operations. In other examples, as mentioned above, the IP address of the content source device is used to link the content source device with the sub-display, and an identifier for the sub-display is not required.

[0248] In the example where the content source device sends content without a sub-display screen identifier, the display screen control system can still use the unique sub-display screen identifier when sending the content to the display screen so that the display screen can display the content in the appropriate sub-display screen. In other examples, the display screen control system can simply specify the coordinates at which to display the content on the display screen.

[0249] The present disclosure may also allow for sub-display screens for showing content and collaboration. For example, a user can draw input on a display wall and send the input to a virtual whiteboard of a content source device. Other users of the content source device can draw on the whiteboard on the content source device, and those updates can be reflected on the display screen. When the source device is remote (e.g., Fig. 9 ), you can also use this virtual shared whiteboard.

[0250] Those of ordinary skill in the art who benefit from this disclosure will appreciate that, although various functions are attributed to specific devices and specific components or modules within those devices, such description is exemplary. Other configurations are possible and are contemplated such that the functions attributed to specific devices and / or components can be performed by different devices and / or components.

[0251] Fig.42A block diagram of an exemplary machine 4200 on which any one or more of the techniques (e.g., methods) discussed herein may be implemented is illustrated. In alternative embodiments, machine 4200 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 4200 may operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, machine 4200 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 4200 may be a display screen, a display screen control system, a content source device, a remote content source device, a computing device, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a network device, a network router, a switch or a bridge, or any machine capable of running instructions (sequentially or otherwise) specifying the actions that a machine will take. Machine 4200 may be configured to implement a process, create a GUI, and be configured to have Figure 1-43 Further, while only a single machine is illustrated, the term “machine” should also be understood to include any collection of machines that individually or jointly run a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein (such as cloud computing, software as a service (SaaS), other computer cluster configurations).

[0252] As described herein, examples may include or may operate on a logic unit or multiple components, modules, or mechanisms (hereinafter referred to as "modules"). A module is a tangible entity (e.g., hardware) that can perform a specified operation and may be configured or arranged in a particular manner. In an example, a circuit may be arranged in a particular manner (e.g., internally or relative to an external entity such as other circuits) as a module. In an example, all or part of one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application parts, or applications) as a module that performs a specified operation. In an example, the software may reside on a machine-readable medium. In an example, the software, when run by the underlying hardware of the module, causes the hardware to perform the specified operation.

[0253] Thus, the term "module" is understood to include a tangible entity, i.e., an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a particular manner or to perform part or all of any operation described herein. Consider an example in which modules are temporarily configured, each of which need not be instantiated at any time. For example, in the case where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as various different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time instance and to constitute different modules at different time instances.

[0254] The machine (e.g., computer system) 4200 may include a hardware processor 4202 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 4204, and a static memory 4206, some or all of which may communicate with each other via an interconnection link (e.g., a bus) 4208. The machine 4200 may also include a display unit 4210, an alphanumeric input device 4212 (e.g., a keyboard), and a user interface (UI) navigation device 4214 (e.g., a mouse). In an example, the display unit 4210, the input device 4212, and the UI navigation device 4214 may be a touch screen display. The machine 4200 may additionally include a storage device (e.g., a drive unit) 4216, a signal generating device 4218 (e.g., a speaker), a network interface device 4220, and one or more sensors 4221 (such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors). The machine 4200 may include an output controller 4228, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0255] The storage device 4216 may include a machine-readable medium 4222 on which is stored one or more sets of data structures or instructions 4224 (e.g., software) that embody or are used by any one or more of the techniques or functions described herein. The instructions 4224 may also reside, completely or at least partially, within the main memory 4204, within the static memory 4206, or within the hardware processor 4202 during execution by the machine 4200. In an example, one or any combination of the hardware processor 4202, the main memory 4204, the static memory 4206, or the storage device 4216 may constitute a machine-readable medium.

[0256] Although the machine-readable medium 4222 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 4224.

[0257] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 4200 and causing the machine 4200 to perform any one or more of the techniques of the present disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, the machine-readable medium may include non-temporary machine-readable media. In some examples, the machine-readable medium may include a machine-readable medium that is not a temporarily propagated signal.

[0258] The instructions 4224 may also be sent or received over a communication network 4226 using a transmission medium via the network interface device 4220. The machine 4200 may communicate with one or more other machines using any of a variety of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Exemplary communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network known as a cellular network). The Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as IEEE 802.16 series of standards), IEEE 802.15.4 series of standards, Long Term Evolution (LTE) series of standards, Universal Mobile Telecommunications System (UMTS) series of standards, peer-to-peer (P2P) networks, etc. In an example, the network interface device 4220 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to the communication network 4226. In an example, the network interface device 4220 may include multiple antennas to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology for wireless communication. In some examples, the network interface device 4220 may use multi-user MIMO technology for wireless communication.

[0259] Additional Notes and Examples

[0260] Example 1 is a method for designating a sub-display area, the method comprising: using one or more hardware processors: recognizing a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display on the display screen, the sub-display including coordinates different from those of a previously designated sub-display; in response to recognizing the gesture input, creating a sub-display based on the geometry and linking the sub-display with a content source device; and in response to creating the sub-display and linking the sub-display with the content source device, causing content shared by the content source device to be displayed in the sub-display of the display screen simultaneously with different content shared in the previously designated sub-display.

[0261] In Example 2, the subject matter of Example 1 includes: wherein the content shared in the previously designated sub-display screen is shared by a second content source device.

[0262] In Example 3, the subject matter of Examples 1-2 includes: wherein the method further includes: identifying a corrected geometric shape based on a first shape generated by the geometric structure of the sub-display, the corrected geometric shape being a geometric shape among multiple corrected geometric shapes that is most similar to the first shape; and wherein creating the sub-display based on the geometric structure includes creating the sub-display using the corrected geometric shape.

[0263] In Example 4, the subject matter of Example 3 includes: wherein the corrected geometric shape is a polygon having more than four sides.

[0264] In Example 5, the subject matter of Examples 3-4 includes: wherein the corrected geometric shape is a circle.

[0265] In Example 6, the subject matter of Examples 3-5 includes: wherein identifying the corrected geometric shape occurs while the gesture input is in progress, and wherein the method includes updating a contour of the gesture input to match the corrected geometric shape.

[0266] In Example 7, the subject matter of Examples 1-6 includes wherein the different content shared in the previously designated sub-display screen is shared by the content source device.

[0267] In Example 8, the subject matter of Examples 1-7 includes: identifying an input directed to the display surface of the display screen in a position corresponding to the sub-display screen; and in response to determining that the input is directed to the sub-display screen, sending the input to the content source device based on the linking of the sub-display screen with the content source device.

[0268] In Example 9, the subject matter of Example 8 includes: wherein sending the input to the content source device includes sending an identifier of the sub-display screen to the content source device, the content source device routing the input to an application based on the sub-display screen identifier.

[0269] In Example 10, the subject matter of Examples 1-9 includes: identifying a first simultaneous input and a second simultaneous input, both of which are directed toward the display surface; determining that the first simultaneous input is directed toward the sub-display screen; in response to determining that the first simultaneous input is directed toward the sub-display screen, sending the first simultaneous input to the content source device; determining that the second simultaneous input is directed toward the previously designated sub-display screen; and in response to determining that the second simultaneous input is directed toward the previously designated sub-display screen, sending the second simultaneous input to a second content source device linked to the previously designated sub-display screen.

[0270] In Example 11, the subject matter of Example 10 includes: wherein determining that the first simultaneous input is directed to the sub-display screen includes determining that coordinates of the first simultaneous input on the display screen correspond to coordinates belonging to the sub-display screen.

[0271] In Example 12, the subject matter of Examples 1-11 includes causing display of a geometric shape defining a boundary of the sub-display screen.

[0272] In Example 13, the subject matter of Examples 1-12 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that content shared by the content source device is displayed in the sub-display screen of the display screen includes: providing the address of the display screen to the content source device, and wherein the content source device sends the content to the address.

[0273] In Example 14, the subject matter of Examples 1-13 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen includes: receiving the content from the content source device; and sending the content to the display screen.

[0274] In Example 15, the subject matter of Examples 1-14 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen includes: receiving the content from the content source device; and rendering the content in the sub-display screen on the display screen.

[0275] In Example 16, the subject matter of Examples 1-15 includes: wherein the one or more hardware processors are communicatively coupled to the display screen via a local bus.

[0276] In Example 17, the subject matter of Examples 1-16 includes wherein the one or more hardware processors are communicatively coupled to the display screen via a packet-based Internet Protocol network.

[0277] In Example 18, the subject matter of Examples 1-17 includes: wherein the display is a touch screen display, and wherein the gesture input is a touch input on the display surface.

[0278] In Example 19, the subject matter of Examples 1-18 includes: wherein the display surface is a projection screen and the display screen is provided by a projector.

[0279] In Example 20, the subject matter of Examples 1-19 includes: displaying a prompt on the display screen, the prompt indicating that the display screen has additional free space not occupied by the sub-display screen, the prompt providing a user with the option of automatically increasing the size of the sub-display screen to fill the additional free space; and in response to receiving an indication to increase the size of the sub-display screen, increasing the size of the sub-display screen.

[0280] In Example 21, the subject matter of Example 20 includes: wherein the prompt is displayed based on determining that the content is being scaled to fit on the sub-display screen.

[0281] In Example 22, the subject matter of Examples 1-21 includes: wherein the content is content from a first source and the different content is from a second source, and wherein the method further includes: displaying a prompt on the display screen, the prompt suggesting automatically creating a new sub-display screen with content from a third source, both of the first source and the third source being from the content source device; and in response to receiving an indication to create the new sub-display screen, creating the new sub-display screen and displaying the content from the third source.

[0282] In Example 23, the subject matter of Example 22 includes: wherein the first source and the third source are different communication types provided by a communication application running on the content source device.

[0283] In Example 24, the subject matter of Examples 22-23 includes: wherein displaying the prompt is in response to determining that the display screen has additional free space not occupied by the sub-display screen and the content source device includes the third content source that is not displayed in the sub-display screen.

[0284] In Example 25, the subject matter of Examples 1-24 includes: wherein recognizing the gesture input includes receiving an indication of the gesture input from the display screen.

[0285] In Example 26, the subject matter of Examples 1-25 includes: wherein recognizing the gesture input includes detecting the gesture input at a touch screen of the display screen.

[0286] Example 27 is a computing device for specifying a sub-display area, the device comprising: one or more hardware processors; a memory comprising instructions, which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising: recognizing gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display on the display screen, the sub-display including coordinates different from those of a previously specified sub-display; in response to recognizing the gesture input, creating the sub-display based on the geometry and linking the sub-display with a content source device; and in response to creating the sub-display and linking the sub-display with the content source device, causing content shared by the content source device to be displayed in the sub-display of the display screen simultaneously with different content shared in the previously specified sub-display.

[0287] In Example 28, the subject matter of Example 27 includes: wherein the content shared in the previously designated sub-display screen is shared by a second content source device.

[0288] In Example 29, the subject matter of Examples 27-28 includes: wherein the operation further includes: identifying a corrected geometric shape based on a first shape generated by the geometric structure of the sub-display, the corrected geometric shape being a geometric shape among multiple corrected geometric shapes that is most similar to the first shape; and wherein creating the sub-display based on the geometric structure includes creating the sub-display using the corrected geometric shape.

[0289] In Example 30, the subject matter of Example 29 includes: wherein the corrected geometric shape is a polygon having more than four sides.

[0290] In Example 31, the subject matter of Examples 29-30 includes: wherein the corrected geometric shape is a circle.

[0291] In Example 32, the subject matter of Examples 29-31 includes: wherein the operation of identifying the corrected geometric shape occurs while the gesture input is in progress, and wherein the operation further includes updating a contour of the gesture input to match the corrected geometric shape.

[0292] In Example 33, the subject matter of Examples 27-32 includes: wherein the different content shared in the previously designated sub-display screen is shared by the content source device.

[0293] In Example 34, the subject matter of Examples 27-33 includes: wherein the operation further includes: identifying an input directed to the display surface of the display screen in a position corresponding to the sub-display screen; and in response to determining that the input is directed to the sub-display screen, sending the input to the content source device based on the link between the sub-display screen and the content source device.

[0294] In Example 35, the subject matter of Example 34 includes: wherein the operation of sending the input to the content source device includes sending an identifier of the sub-display screen to the content source device, and the content source device routes the input to an application based on the sub-display screen identifier.

[0295] In Example 36, the subject matter of Examples 27-35 includes: wherein the operation further includes: identifying a first simultaneous input and a second simultaneous input, both of which are directed to the display surface; determining that the first simultaneous input is directed to the sub-display screen; in response to determining that the first simultaneous input is directed to the sub-display screen, sending the first simultaneous input to the content source device; determining that the second simultaneous input is directed to the previously designated sub-display screen; and in response to determining that the second simultaneous input is directed to the previously designated sub-display screen, sending the second simultaneous input to a second content source device linked to the previously designated sub-display screen.

[0296] In Example 37, the subject matter of Example 36 includes: wherein determining that the operation of the first simultaneous input is directed to the sub-display screen includes determining that coordinates of the first simultaneous input on the display screen correspond to coordinates belonging to the sub-display screen.

[0297] In Example 38, the subject matter of Examples 27-37 includes: wherein the operations further include: causing a geometric shape defining a boundary of the sub-display screen to be displayed.

[0298] In Example 39, the subject matter of Examples 27-38 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen, the operation includes: providing the address of the display screen to the content source device, and wherein the content source device sends the content to the address.

[0299] In Example 40, the subject matter of Examples 27-39 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen, the operation includes: receiving the content from the content source device; and sending the content to the display screen.

[0300] In Example 41, the subject matter of Examples 27-40 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen, the operation includes: receiving the content from the content source device; and rendering the content in the sub-display screen on the display screen.

[0301] In Example 42, the subject matter of Examples 27-41 includes: wherein the one or more hardware processors are communicatively coupled to the display screen via a local bus.

[0302] In Example 43, the subject matter of Examples 27-42 includes: wherein the one or more hardware processors are communicatively coupled to the display screen via a packet-based Internet Protocol network.

[0303] In Example 44, the subject matter of Examples 27-43 includes: wherein the display is a touch screen display, and wherein the gesture input is a touch input on the display surface.

[0304] In Example 45, the subject matter of Examples 27-44 includes: wherein the display surface is a projection screen and the display screen is provided by a projector.

[0305] In Example 46, the subject matter of Examples 27-45 includes: wherein the operation further includes: displaying a prompt on the display screen, the prompt indicating that the display screen has additional free space not occupied by a sub-display screen, the prompt providing the user with the option of automatically increasing the size of the sub-display screen to fill the additional free space; and in response to receiving an indication to increase the size of the sub-display screen, increasing the size of the sub-display screen.

[0306] In Example 47, the subject matter of Example 46 includes: wherein the prompt is displayed based on determining that the content is being scaled to fit in the sub-display screen.

[0307] In Example 48, the subject matter of Examples 27-47 includes: wherein the content is content from a first source and different content from a second source, and wherein the operation further includes: displaying a prompt on the display screen, the prompt suggesting automatically creating a new sub-display screen with content from a third source, both of the first source and the third source being from the content source device; and in response to receiving an indication to create the new sub-display screen, creating the new sub-display screen and displaying the content from the third source.

[0308] In Example 49, the subject matter of Example 48 includes: wherein the first source and the third source are different communication types provided by a communication application running on the content source device.

[0309] In Example 50, the subject matter of Examples 48-49 includes: wherein the operation of displaying the prompt is in response to determining that the display screen has additional free space not occupied by the sub-display screen and the content source device includes the third content source that is not displayed in the sub-display screen.

[0310] In Example 51, the subject matter of Examples 27-50 includes: wherein the operation of recognizing the gesture input includes receiving an indication of the gesture input from the display screen.

[0311] In Example 52, the subject matter of Examples 27-51 includes: wherein the operation of recognizing the gesture input includes detecting the gesture input at a touch screen of the display screen.

[0312] Example 53 is a machine-readable medium storing instructions for specifying a sub-display area, which when executed by a machine causes the machine to perform operations including: recognizing gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display on the display screen, the sub-display including coordinates different from those of a previously specified sub-display; in response to recognizing the gesture input, creating the sub-display based on the geometry and linking the sub-display with a content source device; and in response to creating the sub-display and linking the sub-display with the content source device, causing content shared by the content source device to be displayed in the sub-display of the display screen simultaneously with different content shared in the previously specified sub-display.

[0313] In Example 54, the subject matter of Example 53 includes: wherein the content shared in the previously designated sub-display screen is shared by a second content source device.

[0314] In Example 55, the subject matter of Examples 53-54 includes: wherein the operation further includes: identifying a corrected geometric shape based on a first shape generated by the geometric structure of the sub-display, the corrected geometric shape being a geometric shape among multiple corrected geometric shapes that is most similar to the first shape; and wherein creating the sub-display based on the geometric structure includes creating the sub-display using the corrected geometric shape.

[0315] In Example 56, the subject matter of Example 55 includes: wherein the corrected geometric shape is a polygon having more than four sides.

[0316] In Example 57, the subject matter of Examples 55-56 includes: wherein the corrected geometric shape is a circle.

[0317] In Example 58, the subject matter of Examples 55-57 includes: wherein the operation of identifying the corrected geometric shape occurs while the gesture input is in progress, and wherein the operation further includes updating the contour of the gesture input to match the corrected geometric shape.

[0318] In Example 59, the subject matter of Examples 53-58 includes: wherein the different content shared in the previously designated sub-display screen is shared by the content source device.

[0319] In Example 60, the subject matter of Examples 53-59 includes: wherein the operation further includes: identifying an input directed to the display surface of the display screen in a position corresponding to the sub-display screen; and in response to determining that the input is directed to the sub-display screen, sending the input to the content source device based on the link between the sub-display screen and the content source device.

[0320] In Example 61, the subject matter of Example 60 includes: wherein the operation of sending the input to the content source device includes sending an identifier of the sub-display screen to the content source device, and the content source device routes the input to an application based on the sub-display screen identifier.

[0321] In Example 62, the subject matter of Examples 53-61 includes: wherein the operation also includes: identifying a first simultaneous input and a second simultaneous input, both of which are directed to the display surface; determining that the first simultaneous input is directed to the sub-display screen; in response to determining that the first simultaneous input is directed to the sub-display screen, sending the first simultaneous input to the content source device; determining that the second simultaneous input is directed to the previously designated sub-display screen; and in response to determining that the second simultaneous input is directed to the previously designated sub-display screen, sending the second simultaneous input to a second content source device linked to the previously designated sub-display screen.

[0322] In Example 63, the subject matter of Example 62 includes: wherein determining that the operation of the first simultaneous input is directed to the sub-display screen includes determining that coordinates of the first simultaneous input on the display screen correspond to coordinates belonging to the sub-display screen.

[0323] In Example 64, the subject matter of Examples 53-63 includes: wherein the operation further includes: causing a geometric shape defining a boundary of the sub-display screen to be displayed.

[0324] In Example 65, the subject matter of Examples 53-64 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen, the operation includes: providing the address of the display screen to the content source device, and wherein the content source device sends the content to the address.

[0325] In Example 66, the subject matter of Examples 53-65 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen, the operation includes: receiving the content from the content source device; and sending the content to the display screen.

[0326] In Example 67, the subject matter of Examples 53-66 includes: wherein, in response to creating the sub-display screen and linking the sub-display screen with the content source device so that the content shared by the content source device is displayed in the sub-display screen of the display screen, the operation includes: receiving the content from the content source device; and rendering the content in the sub-display screen on the display screen.

[0327] In Example 68, the subject matter of Examples 53-67 includes: wherein the machine is communicatively coupled to the display screen via a local bus.

[0328] In Example 69, the subject matter of Examples 53-68 includes: wherein the machine is communicatively coupled to the display screen via a packet-based Internet Protocol network.

[0329] In Example 70, the subject matter of Examples 53-69 includes: wherein the display is a touch screen display, and wherein the gesture input is a touch input on the display surface.

[0330] In Example 71, the subject matter of Examples 53-70 includes: wherein the display surface is a projection screen and the display screen is provided by a projector.

[0331] In Example 72, the subject matter of Examples 53-71 includes: wherein the operation further includes: displaying a prompt on the display screen, the prompt indicating that the display screen has additional free space not occupied by a sub-display screen, the prompt providing the user with the option of automatically increasing the size of the sub-display screen to fill the additional free space; and in response to receiving an indication to increase the size of the sub-display screen, increasing the size of the sub-display screen.

[0332] In Example 73, the subject matter of Example 72 includes: wherein the prompt is displayed based on determining that the content is being scaled to fit in the sub-display screen.

[0333] In Example 74, the subject matter of Examples 53-73 includes: wherein the content is content from a first source and the different content is from a second source, and wherein the operation further includes: displaying a prompt on the display screen, the prompt suggesting automatically creating a new sub-display screen with content from a third source, both of the first source and the third source being from the content source device; and in response to receiving an indication to create the new sub-display screen, creating the new sub-display screen and displaying the content from the third source.

[0334] In Example 75, the subject matter of Example 74 includes: wherein the first source and the third source are different communication types provided by a communication application running on the content source device.

[0335] In Example 76, the subject matter of Examples 74-75 includes: wherein the operation of displaying the prompt is in response to determining that the display screen has additional free space not occupied by the sub-display screen and the content source device includes the third content source that is not displayed in the sub-display screen.

[0336] In Example 77, the subject matter of Examples 53-76 includes: wherein the operation of recognizing the gesture input includes receiving an indication of the gesture input from the display screen.

[0337] In Example 78, the subject matter of Examples 53-77 includes: wherein the operation of recognizing the gesture input includes detecting the gesture input at a touch screen of the display screen.

[0338] Example 79 is a device for specifying a sub-display area, the device comprising: a unit for recognizing a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display on the display screen, the sub-display including coordinates different from those of a previously specified sub-display; a unit for creating a sub-display based on the geometry and linking the sub-display with a content source device in response to recognizing the gesture input; and a unit for causing content shared by the content source device to be displayed in the sub-display of the display screen simultaneously with different content shared in the previously specified sub-display in response to creating the sub-display and linking the sub-display with the content source device.

[0339] In Example 80, the subject matter of Example 79 includes: wherein the content shared in the previously designated sub-display screen is shared by a second content source device.

[0340] In Example 81, the subject matter of Examples 79-80 includes: wherein the device also includes: a unit for identifying a corrected geometric shape based on a first shape generated by the geometric structure of the sub-display, the corrected geometric shape being a geometric shape among multiple corrected geometric shapes that is most similar to the first shape; and wherein creating the sub-display based on the geometric structure includes creating the sub-display using the corrected geometric shape.

[0341] In Example 82, the subject matter of Example 81 includes: wherein the corrected geometric shape is a polygon having more than four sides.

[0342] In Example 83, the subject matter of Examples 81-82 includes: wherein the corrected geometric shape is a circle.

[0343] In Example 84, the subject matter of Examples 81-83 includes: wherein the means for identifying the corrected geometric shape occurs while the gesture input is in progress, and wherein the device further includes means for updating the outline of the gesture input to match the corrected geometric shape.

[0344] In Example 85, the subject matter of Examples 79-84 includes: wherein the different content shared in the previously designated sub-display screen is shared by the content source device.

[0345] In Example 86, the subject matter of Examples 79-85 includes: a unit for identifying an input directed to the display surface of the display screen in a position corresponding to the sub-display screen; and a unit for sending the input to the content source device based on the link between the sub-display screen and the content source device in response to determining that the input is directed to the sub-display screen.

[0346] In Example 87, the subject matter of Example 86 includes: wherein the means for sending the input to the content source device includes means for sending an identifier of the sub-display screen to the content source device, and the content source device routes the input to an application based on the sub-display screen identifier.

[0347] In Example 88, the subject matter of Examples 79-87 includes: a unit for identifying a first simultaneous input and a second simultaneous input, both of which are directed toward the display surface; a unit for determining that the first simultaneous input is directed toward the sub-display screen; a unit for sending the first simultaneous input to the content source device in response to determining that the first simultaneous input is directed toward the sub-display screen; a unit for determining that the second simultaneous input is directed toward the previously designated sub-display screen; and a unit for sending the second simultaneous input to a second content source device linked to the previously designated sub-display screen in response to determining that the second simultaneous input is directed toward the previously designated sub-display screen.

[0348] In Example 89, the subject matter of Example 88 includes: wherein the unit for determining that the first simultaneous input is directed to the sub-display screen includes a unit for determining that the coordinates of the first simultaneous input on the display screen correspond to the coordinates belonging to the sub-display screen.

[0349] In Example 90, the subject matter of Examples 79-89 includes means for causing display of a geometric shape defining a boundary of the sub-display screen.

[0350] In Example 91, the subject matter of Examples 79-90 includes: wherein the unit for causing content shared by the content source device to be displayed in the sub-display screen of the display screen includes: a unit for providing the address of the display screen to the content source device, and wherein the content source device sends the content to the address.

[0351] In Example 92, the subject matter of Examples 79-91 includes: wherein the unit for causing the content shared by the content source device to be displayed in the sub-display screen of the display screen includes: a unit for receiving the content from the content source device; and a unit for sending the content to the display screen.

[0352] In Example 93, the subject matter of Examples 79-92 includes: wherein, the unit for causing the content shared by the content source device to be displayed in the sub-display screen of the display screen includes: a unit for receiving the content from the content source device; and a unit for rendering the content in the sub-display screen on the display screen.

[0353] In Example 94, the subject matter of Examples 79-93 includes: wherein the device is communicatively coupled to the display screen via a local bus.

[0354] In Example 95, the subject matter of Examples 79-94 includes: wherein the device is communicatively coupled to the display screen via a packet-based Internet Protocol network.

[0355] In Example 96, the subject matter of Examples 79-95 includes: wherein the display is a touch screen display, and wherein the gesture input is a touch input on the display surface.

[0356] In Example 97, the subject matter of Examples 79-96 includes: wherein the display surface is a projection screen and the display screen is provided by a projector.

[0357] In Example 98, the subject matter of Examples 79-97 includes: a unit for displaying a prompt on the display screen, the prompt indicating that the display screen has additional free space not occupied by the sub-display screen, the prompt providing a user with the option of automatically increasing the size of the sub-display screen to fill the additional free space; and a unit for increasing the size of the sub-display screen in response to receiving an indication to increase the size of the sub-display screen.

[0358] In Example 99, the subject matter of Example 98 includes: wherein the prompt is displayed based on determining that the content is being scaled to fit in the sub-display screen.

[0359] In Example 100, the subject matter of Examples 79-99 includes: wherein the content is content from a first source and the different content is from a second source, and wherein the device further includes: a unit for displaying a prompt on the display screen, the prompt suggesting automatically creating a new sub-display screen with content from a third source, both of the first source and the third source being from the content source device; and a unit for creating the new sub-display screen and displaying the content from the third source in response to receiving an indication to create the new sub-display screen.

[0360] In Example 101, the subject matter of Example 100 includes: wherein the first source and the third source are different communication types provided by a communication application running on the content source device.

[0361] In Example 102, the subject matter of Examples 100-101 includes wherein displaying the prompt is in response to determining that the display screen has additional free space not occupied by the sub-display screen and the content source device includes the third content source that is not displayed in the sub-display screen.

[0362] In Example 103, the subject matter of Examples 79-102 includes: wherein the means for recognizing the gesture input includes means for receiving an indication of the gesture input from the display screen.

[0363] In Example 104, the subject matter of Examples 79-103 includes: wherein the means for recognizing the gesture input includes means for detecting the gesture input at a touch screen of the display screen.

[0364] Example 105 is a computing device for pairing a designated sub-display, the computing device comprising: one or more hardware processors; a memory storing instructions, the instructions, when executed by the one or more hardware processors, causing the one or more hardware processors to perform operations comprising: recognizing a gesture input directed to a display surface of a display, the gesture input defining a geometry of a sub-display on the display, the sub-display comprising coordinates different from those of a previously designated sub-display; in response to recognizing the gesture input, creating the sub-display based on the defined geometry; displaying a pairing information input area corresponding to the sub-display on the display; receiving a pairing input comprising first pairing information at the pairing information input area; linking the sub-display with a content source device based on a match between the first pairing information from the pairing input and second pairing information provided by the content source device; and in response to linking the sub-display with the content source device, causing content shared by the content source device to be displayed in the sub-display of the display.

[0365] In Example 106, the subject matter of Example 105 includes: wherein the first pairing information is first biometric scan data, and wherein the operation of receiving the pairing input includes causing the biometric scanner to use an in-display biometric scanner to scan a portion of the display screen corresponding to the pairing information input area to generate the first biometric scan data.

[0366] In Example 107, the subject matter of Example 106 includes: wherein the second pairing information is second biometric scan data received together with the pairing request sent by the content source device, and wherein the operation of linking the sub-display screen with the content source device based on matching the first pairing information with the second pairing information includes: storing the first biometric scan data in a record of the sub-display screen; identifying the record of the sub-display screen from multiple other records of other sub-display screens based on matching the first biometric scan data with the second biometric scan data; updating the record of the sub-display screen using the address of the content source device from the pairing request; and wherein causing the content shared by the content source device to be displayed in the sub-display screen of the display screen includes: identifying multiple groups having content from the content source device; identifying the sub-display screen by matching the address in the header of at least one of the multiple groups with the address of the content source device stored in the record of the sub-display screen; and causing the content to be rendered in the sub-display screen of the display screen.

[0367] In Example 108, the subject matter of Examples 106-107 includes: wherein the first biometric scan data is one of the following: fingerprint scan data, human eye scan data, or facial recognition data.

[0368] In Example 109, the subject matter of Examples 105-108 includes wherein the pairing information input area includes a label visually attached to a border of the sub-display screen.

[0369] In Example 110, the subject matter of Examples 105-109 includes: wherein the first pairing information is a number, a letter, or an alphanumeric code.

[0370] In Example 111, the subject matter of Examples 105-110 includes: wherein the first pairing information is a login credential of a user account.

[0371] In Example 112, the subject matter of Examples 105-111 includes: wherein the pairing information input area corresponds to the sub-display screen by linking the sub-display screen with the pairing input.

[0372] In Example 113, the subject matter of Example 112 includes: wherein the pairing information input area corresponds to the sub display screen by visually linking the pairing information input area with the sub display screen.

[0373] Example 114 is a method for pairing a designated sub-display, the method comprising: using one or more hardware processors: identifying a gesture input directed to a display surface of a display, the gesture input defining a geometry of a sub-display on the display, the sub-display including coordinates different from a previously designated sub-display; in response to identifying the gesture input, creating the sub-display based on the defined geometry; displaying a pairing information input area corresponding to the sub-display on the display; receiving a pairing input including first pairing information in the pairing information input area; linking the sub-display with the content source device based on a match between the first pairing information from the pairing input and second pairing information provided by the content source device; and in response to linking the sub-display with the content source device, causing content shared by the content source device to be displayed in the sub-display of the display.

[0374] In Example 115, the subject matter of Example 114 includes: wherein the first pairing information is first biometric scan data, and wherein receiving the pairing input includes causing the biometric scanner to use an in-display biometric scanner to scan a portion of the display screen corresponding to the pairing information input area to generate the first biometric scan data.

[0375] In Example 116, the subject matter of Example 115 includes: wherein the second pairing information is second biometric scan data received together with the pairing request sent by the content source device, and wherein linking the sub-display screen with the content source device based on matching the first pairing information with the second pairing information includes: storing the first biometric scan data in a record of the sub-display screen; identifying the record of the sub-display screen from multiple other records of other sub-display screens based on matching the first biometric scan data with the second biometric scan data; updating the record of the sub-display screen using the address of the content source device from the pairing request; and wherein causing content shared by the content source device to be displayed in the sub-display screen of the display screen includes: identifying multiple groups having content from the content source device; identifying the sub-display screen by matching the address in the header of at least one of the multiple groups with the address of the content source device stored in the record of the sub-display screen; and causing the content to be rendered in the sub-display screen of the display screen.

[0376] In Example 117, the subject matter of Examples 115-116 includes: wherein the first biometric scan data is one of the following: fingerprint scan data, human eye scan data, or facial recognition data.

[0377] In Example 118, the subject matter of Examples 114-117 includes wherein the pairing information input area includes a label visually attached to a border of the sub-display screen.

[0378] In Example 119, the subject matter of Examples 114-118 includes: wherein the first pairing information is a number, a letter, or an alphanumeric code.

[0379] In Example 120, the subject matter of Examples 114-119 includes: wherein the first pairing information is a login credential of a user account.

[0380] In Example 121, the subject matter of Examples 114-120 includes: wherein the pairing information input area corresponds to the sub-display screen by linking the sub-display screen with the pairing input.

[0381] In Example 122, the subject matter of Example 121 includes: wherein the pairing information input area corresponds to the sub display screen by visually linking the pairing information input area with the sub display screen.

[0382] Example 123 is a machine-readable medium storing instructions for pairing a specified sub-display screen, wherein the instructions, when executed by a machine, cause the machine to perform operations including: identifying a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display screen on the display screen, the sub-display screen including coordinates different from those of a previously specified sub-display screen; in response to identifying the gesture input, creating the sub-display screen based on the defined geometry; displaying a pairing information input area corresponding to the sub-display screen on the display screen; receiving a pairing input including first pairing information at the pairing information input area; linking the sub-display screen with the content source device based on matching the first pairing information from the pairing input with second pairing information provided by the content source device; and in response to linking the sub-display screen with the content source device, causing content shared by the content source device to be displayed in the sub-display screen of the display screen.

[0383] In Example 124, the subject matter of Example 123 includes: wherein the first pairing information is first biometric scan data, and wherein the operation of receiving the pairing input includes causing the biometric scanner to use an in-display biometric scanner to scan a portion of the display screen corresponding to the pairing information input area to generate the first biometric scan data.

[0384] In Example 125, the subject matter of Example 124 includes: wherein the second pairing information is second biometric scan data received together with the pairing request sent by the content source device, and wherein the operation of linking the sub-display screen with the content source device based on matching the first pairing information with the second pairing information includes: storing the first biometric scan data in a record of the sub-display screen; identifying the record of the sub-display screen from multiple other records of other sub-display screens based on matching the first biometric scan data with the second biometric scan data; updating the record of the sub-display screen using the address of the content source device from the pairing request; and wherein causing the content shared by the content source device to be displayed in the sub-display screen of the display screen includes: identifying multiple groups having content from the content source device; identifying the sub-display screen by matching the address in the header of at least one of the multiple groups with the address of the content source device stored in the record of the sub-display screen; and causing the content to be rendered in the sub-display screen of the display screen.

[0385] In Example 126, the subject matter of Examples 124-125 includes: wherein the first biometric scan data is one of the following: fingerprint scan data, human eye scan data, or facial recognition data.

[0386] In Example 127, the subject matter of Examples 123-126 includes: wherein the pairing information input area includes a label visually attached to a border of the sub-display screen.

[0387] In Example 128, the subject matter of Examples 123-127 includes: wherein the first pairing information is a number, a letter, or an alphanumeric code.

[0388] In Example 129, the subject matter of Examples 123-128 includes: wherein the first pairing information is a login credential of a user account.

[0389] In Example 130, the subject matter of Examples 123-129 includes: wherein the pairing information input area corresponds to the sub-display screen by linking the sub-display screen with the pairing input.

[0390] In Example 131, the subject matter of Example 130 includes: wherein the pairing information input area corresponds to the sub-display screen by visually linking the pairing information input area with the sub-display screen.

[0391] Example 132 is a device for pairing a designated sub-display, the device comprising: a unit for identifying a gesture input directed toward a display surface of a display, the gesture input defining a geometry of a sub-display on the display, the sub-display including coordinates different from those of a previously designated sub-display; a unit for creating the sub-display based on the defined geometry in response to identifying the gesture input; a unit for displaying a pairing information input area corresponding to the sub-display on the display; a unit for receiving a pairing input including first pairing information at the pairing information input area; a unit for linking the sub-display with a content source device based on matching the first pairing information from the pairing input with second pairing information provided by the content source device; and a unit for causing content shared by the content source device to be displayed in the sub-display of the display in response to linking the sub-display with the content source device.

[0392] In Example 133, the subject matter of Example 132 includes: wherein the first pairing information is a first biometric scan data, and wherein the unit for receiving the pairing input includes causing the biometric scanner to use an in-display biometric scanner to scan a portion of the display screen corresponding to the pairing information input area to generate the first biometric scan data.

[0393] In Example 134, the subject matter of Example 133 includes: wherein the second pairing information is second biometric scan data received together with the pairing request sent by the content source device, and wherein the unit for linking the sub-display screen with the content source device based on matching the first pairing information with the second pairing information includes: a unit for storing the first biometric scan data in a record of the sub-display screen; a unit for identifying the record of the sub-display screen from multiple other records of other sub-display screens based on matching the first biometric scan data with the second biometric scan data; a unit for updating the record of the sub-display screen using the address of the content source device from the pairing request; and wherein causing content shared by the content source device to be displayed in the sub-display screen of the display screen includes: a unit for identifying multiple groups having content from the content source device; a unit for identifying the sub-display screen by matching the address in a header of at least one of the multiple groups with the address of the content source device stored in the record of the sub-display screen; and a unit for causing the content to be rendered in the sub-display screen of the display screen.

[0394] In Example 135, the subject matter of Examples 133-134 includes: wherein the first biometric scan data is one of the following: fingerprint scan data, human eye scan data, or facial recognition data.

[0395] In Example 136, the subject matter of Examples 132-135 includes: wherein the pairing information input area includes a label visually attached to a border of the sub-display screen.

[0396] In Example 137, the subject matter of Examples 132-136 includes: wherein the first pairing information is a number, a letter, or an alphanumeric code.

[0397] In Example 138, the subject matter of Examples 132-137 includes: wherein the first pairing information is a login credential of a user account.

[0398] In Example 139, the subject matter of Examples 132-138 includes: wherein the pairing information input area corresponds to the sub-display screen by linking the sub-display screen with the pairing input.

[0399] In Example 140, the subject matter of Example 139 includes: wherein the pairing information input area corresponds to the sub-display screen by visually linking the pairing information input area with the sub-display screen.

[0400] Example 141 is a device for specifying an input sub-display area, the device comprising: one or more hardware processors; a memory storing instructions, which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations including the following: identifying a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a first sub-display on the display screen; in response to identifying the gesture input, creating the first sub-display based on the geometry; causing content from a content source device to be displayed in the first sub-display of the display screen; identifying a second gesture input directed to the display surface of the display screen, the second gesture input defining a geometry of a second sub-display on the display screen, the second sub-display having a different position from the first sub-display on the display screen; in response to the second gesture input, linking the second sub-display to the content source device; identifying input directed to the second sub-display; and causing the input directed to the second sub-display to be sent to the content source device.

[0401] In Example 142, the subject matter of Example 141 includes: wherein nothing is displayed on the second sub-display screen, and wherein the input entered on the second sub-display screen is not displayed on the second sub-display screen.

[0402] In Example 143, the subject matter of Examples 141-142 includes: wherein the input entered on the second sub-display screen is not displayed on the first sub-display screen.

[0403] In Example 144, the subject matter of Examples 141-143 includes: wherein the operation further includes: sending the identified input directed to the second sub-display screen to the content source device; and saving, by the content source device, the input separately from the content from the content source device.

[0404] In Example 145, the subject matter of Examples 141-144 includes: wherein the operation further includes: sending the identified input directed to the second sub-display screen to the content source device; and saving the input by the content source device as supplementary data of the content from the content source device, wherein the supplementary data is related to a specific portion of the content from the content source device.

[0405] In Example 146, the subject matter of Example 145 includes: wherein the content from the content source device is a presentation, and the auxiliary data is a plurality of annotations, each of the plurality of annotations being associated with a particular slide of the presentation.

[0406] In Example 147, the subject matter of Examples 141-146 includes: wherein the operation further includes: identifying a second input directed to the first sub-display screen; sending the second input to the content source device; and causing the second input to be displayed on the first sub-display screen.

[0407] In Example 148, the subject matter of Examples 141-147 includes: wherein the operation of linking the second sub-display screen to the content source device includes: linking an application running on the content source device to the second sub-display screen, and wherein the operation of causing the input directed to the second sub-display screen to be sent to the content source device includes causing the input entered on the second sub-display screen to be sent to the application on the content source device.

[0408] In Example 149, the subject matter of Example 148 includes: wherein a second application generates the content displayed on the first sub-display screen.

[0409] In Example 150, the subject matter of Examples 141-149 includes: wherein the input is not displayed on the display screen.

[0410] In Example 151, the subject matter of Examples 141-150 includes: wherein the operation of identifying the input directed to the second sub-display screen includes: receiving input to the touch screen of the display screen at coordinates within the second sub-display screen, wherein the input is input in front of the display screen, a gesture of touching the display screen, or input to a screen keyboard on the display screen.

[0411] In Example 152, the subject matter of Examples 141-151 includes: wherein the operation of linking the second sub-display screen to the content source device includes matching a first sub-display screen identifier from the display screen with a second sub-display screen identifier from the content source device.

[0412] In Example 153, the subject matter of Example 152 includes: wherein the operation of causing the input directed to the second sub-display screen to be sent to the content source device includes: determining that the input corresponds to the second sub-display screen; determining that the content source device is linked to the second sub-display screen based on a sub-display screen information data structure; determining an address of the content source device based on the sub-display screen information data structure; and transmitting the input to the content source device at the address via a network.

[0413] In Example 154, the subject matter of Examples 141-153 includes: wherein the operation further includes: linking the first sub-display screen to the content source device by matching the first sub-display screen identifier from the display screen with the second sub-display screen identifier from the content source device.

[0414] Example 155 is a method for specifying an input sub-display area, the method comprising: using one or more hardware processors: identifying a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a first sub-display on the display screen; in response to identifying the gesture input, creating the first sub-display based on the geometry; causing content from a content source device to be displayed in the first sub-display of the display screen; identifying a second gesture input directed to the display surface of the display screen, the second gesture input defining a geometry of a second sub-display on the display screen, the second sub-display having a different position than the first sub-display on the display screen; in response to the second gesture input, linking the second sub-display to the content source device; identifying input directed to the second sub-display; and causing the input directed to the second sub-display to be sent to the content source device.

[0415] In Example 156, the subject matter of Example 155 includes: wherein nothing is displayed on the second sub-display screen, and wherein the input entered on the second sub-display screen is not displayed on the second sub-display screen.

[0416] In Example 157, the subject matter of Examples 155-156 includes: wherein the input entered on the second sub-display is not displayed on the first sub-display.

[0417] In Example 158, the subject matter of Examples 155-157 includes: sending the identified input directed to the second sub-display screen to the content source device; and saving, by the content source device, the input separately from the content from the content source device.

[0418] In Example 159, the subject matter of Examples 155-158 includes: sending the identified input directed to the second sub-display screen to the content source device; and saving, by the content source device, the input as supplemental data to the content from the content source device, the supplemental data relating to a specific portion of the content from the content source device.

[0419] In Example 160, the subject matter of Example 159 includes: wherein the content from the content source device is a presentation, and the auxiliary data is a plurality of annotations, each of the plurality of annotations being related to a particular slide of the presentation.

[0420] In Example 161, the subject matter of Examples 155-160 includes: identifying a second input directed to the first sub-display screen; sending the second input to the content source device; and causing the second input to be displayed on the first sub-display screen.

[0421] In Example 162, the subject matter of Examples 155-161 includes: wherein linking the second sub-display screen to the content source device includes linking an application running on the content source device to the second sub-display screen, and wherein causing the input directed to the second sub-display screen to be sent to the content source device includes causing the input entered on the second sub-display screen to be sent to the application on the content source device.

[0422] In Example 163, the subject matter of Example 162 includes: wherein a second application generates the content displayed on the first sub-display screen.

[0423] In Example 164, the subject matter of Examples 155-163 includes: wherein the input is not displayed on the display screen.

[0424] In Example 165, the subject matter of Examples 155-164 includes: wherein identifying the input directed to the second sub-display includes: receiving input to the touch screen of the display screen at coordinates within the second sub-display screen, wherein the input is input in front of the display screen, a gesture of touching the display screen, or input to an on-screen keyboard on the display screen.

[0425] In Example 166, the subject matter of Examples 155-165 includes: wherein linking the second sub-display screen to the content source device includes matching a first sub-display screen identifier from the display screen with a second sub-display screen identifier from the content source device.

[0426] In Example 167, the subject matter of Example 166 includes: wherein, causing the input directed to the second sub-display screen to be sent to the content source device includes: determining that the input corresponds to the second sub-display screen; determining that the content source device is linked to the second sub-display screen based on a sub-display screen information data structure; determining an address of the content source device based on the sub-display screen information data structure; and transmitting the input to the content source device at the address via a network.

[0427] In Example 168, the subject matter of Examples 155-167 includes: wherein the method further includes: linking the first sub-display screen to the content source device by matching the first sub-display screen identifier from the display screen with the second sub-display screen identifier from the content source device.

[0428] Example 169 is a machine-readable medium storing instructions for specifying an input sub-display area, the instructions, when executed by a machine, causing the machine to perform operations including: identifying a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a first sub-display on the display screen; in response to identifying the gesture input, creating the first sub-display based on the geometry; causing content from a content source device to be displayed in the first sub-display of the display screen; identifying a second gesture input directed to the display surface of the display screen, the second gesture input defining a geometry of a second sub-display on the display screen, the second sub-display having a different position than the first sub-display on the display screen; in response to the second gesture input, linking the second sub-display to the content source device; identifying an input directed to the second sub-display; and causing the input directed to the second sub-display to be sent to the content source device.

[0429] In Example 170, the subject matter of Example 169 includes: wherein nothing is displayed on the second sub-display screen, and wherein the input entered on the second sub-display screen is not displayed on the second sub-display screen.

[0430] In Example 171, the subject matter of Examples 169-170 includes: wherein the input entered on the second sub-display is not displayed on the first sub-display.

[0431] In Example 172, the subject matter of Examples 169-171 includes: wherein the operation further includes: sending the identified input directed to the second sub-display screen to the content source device; and saving, by the content source device, the input separately from the content from the content source device.

[0432] In Example 173, the subject matter of Examples 169-172 includes: wherein the operation also includes: sending the identified input directed to the second sub-display screen to the content source device; and saving the input by the content source device as supplementary data of the content from the content source device, wherein the supplementary data is related to a specific portion of the content from the content source device.

[0433] In Example 174, the subject matter of Example 173 includes: wherein the content from the content source device is a presentation, and the auxiliary data is a plurality of annotations, each of the plurality of annotations being related to a particular slide of the presentation.

[0434] In Example 175, the subject matter of Examples 169-174 includes: wherein the operation further includes: identifying a second input directed to the first sub-display screen; sending the second input to the content source device; and causing the second input to be displayed on the first sub-display screen.

[0435] In Example 176, the subject matter of Examples 169-175 includes: wherein the operation of linking the second sub-display screen to the content source device includes: linking an application running on the content source device to the second sub-display screen, and wherein the operation of causing the input directed to the second sub-display screen to be sent to the content source device includes causing the input entered on the second sub-display screen to be sent to the application on the content source device.

[0436] In Example 177, the subject matter of Example 176 includes: wherein a second application generates the content displayed on the first sub-display screen.

[0437] In Example 178, the subject matter of Examples 169-177 includes: wherein the input is not displayed on the display screen.

[0438] In Example 179, the subject matter of Examples 169-178 includes: wherein the operation of identifying the input directed to the second sub-display includes: receiving input to the touch screen of the display screen at coordinates within the second sub-display screen, wherein the input is input in front of the display screen, a gesture of touching the display screen, or input to a screen keyboard on the display screen.

[0439] In Example 180, the subject matter of Examples 169-179 includes: wherein the operation of linking the second sub-display screen to the content source device includes matching the first sub-display screen identifier from the display screen with the second sub-display screen identifier from the content source device.

[0440] In Example 181, the subject matter of Example 180 includes: wherein the operation of causing the input directed to the second sub-display screen to be sent to the content source device includes: determining that the input corresponds to the second sub-display screen; determining that the content source device is linked to the second sub-display screen based on a sub-display screen information data structure; determining an address of the content source device based on the sub-display screen information data structure; and transmitting the input to the content source device at the address via a network.

[0441] In Example 182, the subject matter of Examples 169-181 includes: wherein the operation further includes: linking the first sub-display screen to the content source device by matching the first sub-display screen identifier from the display screen with the second sub-display screen identifier from the content source device.

[0442] Example 183 is a device for specifying an input sub-display area, the device comprising: a unit for identifying a gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a first sub-display on the display screen; a unit for creating the first sub-display based on the geometry in response to identifying the gesture input; a unit for causing content from a content source device to be displayed in the first sub-display of the display screen; a unit for identifying a second gesture input directed to the display surface of the display screen, the second gesture input defining a geometry of a second sub-display on the display screen, the second sub-display having a different position from the first sub-display on the display screen; a unit for linking the second sub-display to the content source device in response to the second gesture input; a unit for identifying an input directed to the second sub-display; and a unit for causing the input directed to the second sub-display to be sent to the content source device.

[0443] In Example 184, the subject matter of Example 183 includes: wherein nothing is displayed on the second sub-display screen, and wherein the input entered on the second sub-display screen is not displayed on the second sub-display screen.

[0444] In Example 185, the subject matter of Examples 183-184 includes: wherein the input entered on the second sub-display screen is not displayed on the first sub-display screen.

[0445] In Example 186, the subject matter of Examples 183-185 includes: means for sending the identified input directed to the second sub-display screen to the content source device; and means for saving, by the content source device, the input separately from the content from the content source device.

[0446] In Example 187, the subject matter of Examples 183-186 includes: a unit for sending an identified input directed to the second sub-display screen to the content source device; and a unit for saving, by the content source device, the input as supplemental data for the content from the content source device, the supplemental data relating to a specific portion of the content from the content source device.

[0447] In Example 188, the subject matter of Example 187 includes: wherein the content from the content source device is a presentation, and the auxiliary data is a plurality of annotations, each of the plurality of annotations being related to a particular slide of the presentation.

[0448] In Example 189, the subject matter of Examples 183-188 includes: means for identifying a second input directed to the first sub-display screen; means for sending the second input to the content source device; and means for causing the second input to be displayed on the first sub-display screen.

[0449] In Example 190, the subject matter of Examples 183-189 includes: wherein the unit for linking the second sub-display screen to the content source device includes: linking an application running on the content source device to the second sub-display screen, and wherein the unit for causing the input directed to the second sub-display screen to be sent to the content source device includes causing the input entered on the second sub-display screen to be sent to the application on the content source device.

[0450] In Example 191, the subject matter of Example 190 includes: wherein a second application generates the content displayed on the first sub-display screen.

[0451] In Example 192, the subject matter of Examples 183-191 includes: wherein the input is not displayed on the display screen.

[0452] In Example 193, the subject matter of Examples 183-192 includes: wherein the unit for identifying the input directed to the second sub-display includes: a unit for receiving input to the touch screen of the display screen at coordinates within the second sub-display screen, wherein the input is input in front of the display screen, a gesture of touching the display screen, or input to a screen keyboard on the display screen.

[0453] In Example 194, the subject matter of Examples 183-193 includes: wherein the means for linking the second sub-display screen to the content source device includes means for matching a first sub-display screen identifier from the display screen with a second sub-display screen identifier from the content source device.

[0454] In Example 195, the subject matter of Example 194 includes: wherein, the unit for causing the input directed to the second sub-display screen to be sent to the content source device includes: a unit for determining that the input corresponds to the second sub-display screen; a unit for determining that the content source device is linked to the second sub-display screen based on a sub-display screen information data structure; a unit for determining an address of the content source device based on the sub-display screen information data structure; and a unit for transmitting the input to the content source device at the address via a network.

[0455] In Example 196, the subject matter of Examples 183-195 includes: wherein the device further includes: a unit for linking the first sub-display screen to the content source device by matching the first sub-display screen identifier from the display screen with the second sub-display screen identifier from the content source device.

[0456] Example 197 is a computing device for specifying a sub-display area for a remote content source device, the computing device comprising: one or more hardware processors; a memory storing instructions, which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising: receiving a first message via a network, the first message including an identifier of a sub-display of a display and an identifier of the remote content source device, the first message linking the sub-display with the remote content source device, the sub-display being specified by a gesture input directed to a display surface of the display; in response to receiving the first message, sending a second message via the network to the remote content source device, the second message indicating that the sub-display is linked to the remote content source device; receiving content from the remote content source device via the network; and displaying the content in the sub-display by sending the content via the network to a display control system to include the identifier of the sub-display.

[0457] In Example 198, the subject matter of Example 197 includes: wherein the first message is received from a second computing device via the network.

[0458] In Example 199, the subject matter of Example 198 includes: wherein the identifier is input by a user of the second computing device into an application running on the second computing device, and wherein the first message is sent by the application.

[0459] In Example 200, the subject matter of Examples 198-199 includes: wherein the first message includes an address of the display screen control system discovered by the second computing device based on a broadcast message received from the display screen control system.

[0460] In Example 201, the subject matter of Examples 198-200 includes: wherein the first message is received as part of a network-based communication session with the remote content source device.

[0461] In Example 202, the subject matter of Examples 197-201 includes: wherein the operations further include: authenticating the display screen control system to the network-based communication session; and determining an address of the display screen control system based on the authentication.

[0462] In Example 203, the subject matter of Examples 197-202 includes: wherein the operations further include providing a network-based communication session by routing one or more communications between the second computing device and the remote content source device to facilitate the network-based communication session.

[0463] In Example 204, the subject matter of Examples 197-203 includes: wherein the first message is received from the display screen control system.

[0464] In Example 205, the subject matter of Examples 197-204 includes: wherein the operation of causing the display of the content includes sending the content to the display screen control system, and wherein the operation also includes: sending the content to a third computing device, the third computing device being a participant in the network-based communication session.

[0465] In Example 206, the subject matter of Examples 197-205 includes: wherein the display screen control system is the same device as the display screen.

[0466] In Example 207, the subject matter of Examples 197-206 includes: wherein the identifier of the remote content source device includes a user identifier, and wherein the operation further includes: mapping the user identifier to the remote content source device based on authenticating the user on the remote content source device.

[0467] In Example 208, the subject matter of Examples 197-207 includes: wherein the operation further includes: receiving an indication of the gesture input; and in response to the indication of the gesture input, creating the sub-display screen by initializing a data structure storing the display.

[0468] Example 209 is a method for specifying a sub-display area for a remote content source device, the method comprising: using one or more hardware processors of a computing device: receiving a first message via a network, the first message including an identifier of a sub-display of a display and an identifier of the remote content source device, the first message linking the sub-display with the remote content source device, the sub-display being specified by a gesture input directed to a display surface of the display; in response to receiving the first message, sending a second message to the remote content source device via the network, the second message indicating that the sub-display is linked to the remote content source device; receiving content from the remote content source device via the network; and displaying the content in the sub-display by sending the content via the network to a display control system to include the identifier of the sub-display.

[0469] In Example 210, the subject matter of Example 209 includes: wherein the first message is received from a second computing device via the network.

[0470] In Example 211, the subject matter of Example 210 includes: wherein the identifier is input by a user of the second computing device into an application running on the second computing device, and wherein the first message is sent by the application.

[0471] In Example 212, the subject matter of Examples 210-211 includes: wherein the first message includes an address of the display screen control system discovered by the second computing device based on receiving a broadcast message from the display screen control system.

[0472] In Example 213, the subject matter of Examples 210-212 includes wherein the first message is received as part of a network-based communication session with the remote content source device.

[0473] In Example 214, the subject matter of Examples 209-213 includes: authenticating the display screen control system to the network-based communication session; and determining an address of the display screen control system based on the authentication.

[0474] In Example 215, the subject matter of Examples 209-214 includes: wherein the computing device provides a network-based communication session by routing one or more communications between a second computing device and the remote content source device to facilitate the network-based communication session.

[0475] In Example 216, the subject matter of Examples 209-215 includes: wherein the first message is received from the display screen control system.

[0476] In Example 217, the subject matter of Examples 209-216 includes: wherein causing display of the content includes sending the content to the display screen control system, and wherein the method further includes: sending the content to a third computing device, the third computing device being a participant in the network-based communication session.

[0477] In Example 218, the subject matter of Examples 209-217 includes: wherein the display screen control system is the same device as the display screen.

[0478] In Example 219, the subject matter of Examples 209-218 includes: wherein the identifier of the remote content source device includes a user identifier, and wherein the method further comprises: mapping the user identifier to the remote content source device based on authenticating the user on the remote content source device.

[0479] In Example 220, the subject matter of Examples 209-219 includes: receiving an indication of the gesture input; and in response to the indication of the gesture input, creating the sub-display screen by initializing a data structure storing the display.

[0480] Example 221 is a machine-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations including: receiving a first message over a network, the first message including an identifier of a sub-display of a display and an identifier of a remote content source device, the first message linking the sub-display with the remote content source device, the sub-display being specified by a gesture input directed to a display surface of the display; in response to receiving the first message, sending a second message over the network to the remote content source device, the second message indicating that the sub-display is linked to the remote content source device; receiving content from the remote content source device over the network; and displaying the content in the sub-display by sending the content over the network to a display control system to cause the content to include the identifier of the sub-display.

[0481] In Example 222, the subject matter of Example 221 includes: wherein the first message is received from the second computing device over a network.

[0482] In Example 223, the subject matter of Example 222 includes: wherein the identifier is input by a user of the second computing device into an application running on the second computing device, and wherein the first message is sent by the application.

[0483] In Example 224, the subject matter of Examples 222-223 includes: wherein the first message includes an address of the display screen control system discovered by the second computing device based on receiving a broadcast message from the display screen control system.

[0484] In Example 225, the subject matter of Examples 222-224 includes: wherein the first message is received as part of a network-based communication session with the remote content source device.

[0485] In Example 226, the subject matter of Examples 221-225 includes: wherein the operations further include: authenticating the display screen control system to the network-based communication session; and determining an address of the display screen control system based on the authentication.

[0486] In Example 227, the subject matter of Examples 221-226 includes: wherein the operations further include: providing a network-based communication session by routing one or more communications between the second computing device and the remote content source device to facilitate the network-based communication session.

[0487] In Example 228, the subject matter of Examples 221-227 includes: wherein the first message is received from the display screen control system.

[0488] In Example 229, the subject matter of Examples 221-228 includes: wherein the operation of causing the display of the content includes sending the content to the display screen control system, and wherein the operation also includes: sending the content to a third computing device, the third computing device being a participant in the network-based communication session.

[0489] In Example 230, the subject matter of Examples 221-229 includes: wherein the display screen control system is the same device as the display screen.

[0490] In Example 231, the subject matter of Examples 221-230 includes: wherein the identifier of the remote content source device includes a user identifier, and wherein the operation further includes: mapping the user identifier to the remote content source device based on authenticating the user on the remote content source device.

[0491] In Example 232, the subject matter of Examples 221-231 includes: wherein the operation further includes: receiving an indication of the gesture input; and in response to the indication of the gesture input, creating the sub-display screen by initializing a data structure storing the display.

[0492] Example 233 is a computing device for specifying a sub-display screen area for a remote content source device, the computing device comprising: a unit for receiving a first message via a network, the first message including an identifier of a sub-display screen of a display screen and an identifier of the remote content source device, the first message linking the sub-display screen with the remote content source device, the sub-display screen being specified by a gesture input directed to a display surface of the display screen; a unit for sending a second message to the remote content source device via the network in response to receiving the first message, the second message indicating that the sub-display screen is linked to the remote content source device; a unit for receiving content from the remote content source device via the network; and a unit for displaying the content in the sub-display screen by sending the content via the network to a display screen control system, the content including the identifier of the sub-display screen.

[0493] In Example 234, the subject matter of Example 233 includes: wherein the first message is received from a second computing device via the network.

[0494] In Example 235, the subject matter of Example 234 includes: wherein the identifier is input by a user of the second computing device into an application running on the second computing device, and wherein the first message is sent by the application.

[0495] In Example 236, the subject matter of Examples 234-235 includes: wherein the first message includes an address of the display screen control system discovered by the second computing device based on receiving a broadcast message from the display screen control system.

[0496] In Example 237, the subject matter of Examples 234-236 includes: wherein the first message is received as part of a network-based communication session with the remote content source device.

[0497] In Example 238, the subject matter of Examples 233-237 includes: means for authenticating the display screen control system to a network-based communication session; and means for determining an address of the display screen control system based on the authentication.

[0498] In Example 239, the subject matter of Examples 233-238 includes: wherein the computing device provides a network-based communication session by routing one or more communications between a second computing device and the remote content source device to facilitate the network-based communication session.

[0499] In Example 240, the subject matter of Examples 233-239 includes: wherein the first message is received from the display screen control system.

[0500] In Example 241, the subject matter of Examples 233-240 includes: wherein the means for causing the display of the content includes sending the content to the display screen control system, and wherein the computing device further includes: a means for sending the content to a third computing device, the third computing device being a participant in the network-based communication session.

[0501] In Example 242, the subject matter of Examples 233-241 includes: wherein the display screen control system is the same device as the display screen.

[0502] In Example 243, the subject matter of Examples 233-242 includes: wherein the identifier of the remote content source device includes a user identifier, and wherein the computing device further includes: a unit for mapping the user identifier to the remote content source device based on authenticating the user on the remote content source device.

[0503] In Example 244, the subject matter of Examples 233-243 includes: means for receiving an indication of the gesture input; and means for creating the sub-display screen by initializing a data structure storing the display in response to the indication of the gesture input.

[0504] Example 245 is a computing device for causing notifications to be displayed using a sub-display screen, the computing device comprising: one or more hardware processors; a memory storing instructions that, when executed, cause the one or more hardware processors to perform operations including: recognizing gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display screen on the display screen; in response to the gesture input, creating the sub-display screen based on the geometry and linking the sub-display screen with a content source device; causing content sent by the content source device to be displayed in the sub-display screen of the display screen; receiving a message from the content source device, the message having information about a notification on the content source device, the notification, if displayed in the sub-display screen, overlapping with the content; causing the notification to be displayed in a notification area of ​​the display screen; and causing the sub-display screen of the display screen to display the content sent by the content source device without the notification overlapping with the content.

[0505] In Example 246, the subject matter of Example 245 includes: wherein the notification area is adjacent to a border of the sub-display screen.

[0506] In Example 247, the subject matter of Examples 245-246 includes: wherein the notification area is not rendered before receiving the message.

[0507] In Example 248, the subject matter of Examples 245-247 includes: wherein the notification area is a separate sub-display screen.

[0508] In Example 249, the subject matter of Examples 245-248 includes: wherein the operation further includes: providing a user interface with selectable options for showing different notification types on the sub-display screen; receiving a selection of a first notification type to be displayed on the sub-display screen; and wherein causing the display of the notification to include determining that the notification is of the first notification type.

[0509] In Example 250, the subject matter of Example 249 includes: wherein the user interface is displayed on the content source device.

[0510] In Example 251, the subject matter of Examples 249-250 includes: wherein the user interface is displayed on the display screen.

[0511] In Example 252, the subject matter of Examples 245-251 includes: wherein the operation also includes: identifying input on the notification area of ​​the display surface, the input matching a predefined gesture to dismiss the notification; and in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to dismiss the notification, dismissing the notification by removing the notification information from the display screen.

[0512] In Example 253, the subject matter of Example 252 includes: wherein the operation of dismissing the notification includes sending a message to the content source device to dismiss the notification.

[0513] In Example 254, the subject matter of Examples 245-253 includes: wherein the operation also includes: recognizing input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and in response to recognizing that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen.

[0514] In Example 255, the subject matter of Example 254 includes: wherein the operation of displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen of the display screen includes: completely replacing the content shared by the content source device in the sub-display screen with the content corresponding to the notification.

[0515] In Example 256, the subject matter of Examples 245-255 includes: wherein the operation also includes: identifying input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, displaying content corresponding to the notification in an automatically created second sub-display screen, the second sub-display screen being at a different position from the sub-display screen.

[0516] In Example 257, the subject matter of Examples 245-256 includes: wherein the notification is not generated from an application that generates the content shared by the content source device.

[0517] Example 258 is a method for displaying a notification using a sub-display screen, the method comprising: using one or more hardware processors: recognizing gesture input directed to a display surface of a display screen, the gesture input defining a geometric structure of a sub-display screen on the display screen; in response to the gesture input, creating a sub-display screen based on the geometric structure and linking the sub-display screen with a content source device; causing content sent by the content source device to be displayed in the sub-display screen of the display screen; receiving a message from the content source device, the message having information about a notification on the content source device, the notification, if displayed in the sub-display screen, overlapping with the content; causing the notification to be displayed in a notification area of ​​the display screen; and causing the sub-display screen of the display screen to display the content sent by the content source device without the notification overlapping with the content.

[0518] In Example 259, the subject matter of Example 258 includes: wherein the notification area is adjacent to a border of the sub-display screen.

[0519] In Example 260, the subject matter of Examples 258-259 includes: wherein the notification area is not rendered before receiving the message.

[0520] In Example 261, the subject matter of Examples 258-260 includes: wherein the notification area is a separate sub-display.

[0521] In Example 262, the subject matter of Examples 258-261 includes: providing a user interface with selectable options for showing different notification types on the sub-display screen; receiving a selection of a first notification type to be displayed on the sub-display screen; and wherein causing display of the notification to include determining that the notification is of the first notification type.

[0522] In Example 263, the subject matter of Example 262 includes: wherein the user interface is displayed on the content source device.

[0523] In Example 264, the subject matter of Examples 262-263 includes: wherein the user interface is displayed on the display screen.

[0524] In Example 265, the subject matter of Examples 258-264 includes: identifying input on the notification area of ​​the display surface, the input matching a predefined gesture to dismiss the notification; and in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to dismiss the notification, dismissing the notification by removing the notification information from the display screen.

[0525] In Example 266, the subject matter of Example 265 includes: wherein causing dismissal of the notification to include sending a message to the content source device to dismiss the notification.

[0526] In Example 267, the subject matter of Examples 258-266 includes: recognizing input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and in response to recognizing that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen.

[0527] In Example 268, the subject matter of Example 267 includes: wherein displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen of the display screen includes: completely replacing the content shared by the content source device in the sub-display screen with the content corresponding to the notification.

[0528] In Example 269, the subject matter of Examples 258-268 includes: recognizing input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and in response to recognizing that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, displaying content corresponding to the notification in an automatically created second sub-display screen, the second sub-display screen being at a different position from the sub-display screen.

[0529] In Example 270, the subject matter of Examples 258-269 includes: wherein the notification is not generated from an application that generates the content shared by the content source device.

[0530] Example 271 is a machine-readable medium storing instructions that, when executed by a machine, cause the machine to perform operations including: recognizing gesture input directed to a display surface of a display screen, the gesture input defining a geometry of a sub-display screen on the display screen; in response to the gesture input, creating the sub-display screen based on the geometry and linking the sub-display screen with a content source device; causing content sent by the content source device to be displayed in the sub-display screen of the display screen; receiving a message from the content source device, the message having information about a notification on the content source device, the notification, if displayed in the sub-display screen, overlapping with the content; causing the notification to be displayed in a notification area of ​​the display screen; and causing the sub-display screen of the display screen to display the content sent by the content source device without the notification overlapping with the content.

[0531] In Example 272, the subject matter of Example 271 includes: wherein the notification area is adjacent to a border of the sub-display screen.

[0532] In Example 273, the subject matter of Examples 271-272 includes: wherein the notification area is not rendered before receiving the message.

[0533] In Example 274, the subject matter of Examples 271-273 includes: wherein the notification area is a separate sub-display screen.

[0534] In Example 275, the subject matter of Examples 271-274 includes: wherein the operation further includes: providing a user interface with selectable options for showing different notification types on the sub-display screen; receiving a selection of a first notification type to be displayed on the sub-display screen; and wherein causing the display of the notification to include determining that the notification is of the first notification type.

[0535] In Example 276, the subject matter of Example 275 includes: wherein the user interface is displayed on the content source device.

[0536] In Example 277, the subject matter of Examples 275-276 includes: wherein the user interface is displayed on the display screen.

[0537] In Example 278, the subject matter of Examples 271-277 includes: wherein the operation also includes: identifying input on the notification area of ​​the display surface, the input matching a predefined gesture to dismiss the notification; and in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to dismiss the notification, dismissing the notification by removing the notification information from the display screen.

[0538] In Example 279, the subject matter of Example 278 includes: wherein causing the operation of dismissing the notification includes sending a message to the content source device to dismiss the notification.

[0539] In Example 280, the subject matter of Examples 271-279 includes: wherein the operation also includes: recognizing input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and in response to recognizing that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen.

[0540] In Example 281, the subject matter of Example 280 includes: wherein the operation of displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen of the display screen includes: completely replacing the content shared by the content source device in the sub-display screen with the content corresponding to the notification.

[0541] In Example 282, the subject matter of Examples 271-281 includes: wherein the operation also includes: identifying input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, displaying content corresponding to the notification in an automatically created second sub-display screen, the second sub-display screen being at a different position from the sub-display screen.

[0542] In Example 283, the subject matter of Examples 271-282 includes: wherein the notification is not generated from an application that generates the content shared by the content source device.

[0543] Example 284 is a device for displaying notifications using a sub-display screen, the device comprising: a unit for recognizing gesture input directed to a display surface of a display screen, the gesture input defining a geometric structure of a sub-display screen on the display screen; a unit for creating the sub-display screen based on the geometric structure and linking the sub-display screen with a content source device in response to the gesture input; a unit for causing content sent by the content source device to be displayed in the sub-display screen of the display screen; a unit for receiving a message from the content source device, the message having information about a notification on the content source device, the notification, if displayed in the sub-display screen, overlapping with the content; a unit for causing the notification to be displayed in a notification area of ​​the display screen; and a unit for causing the sub-display screen of the display screen to display the content sent by the content source device without the notification overlapping with the content.

[0544] In Example 285, the subject matter of Example 284 includes: wherein the notification area is adjacent to a border of the sub-display screen.

[0545] In Example 286, the subject matter of Examples 284-285 includes: wherein the notification area is not rendered before receiving the message.

[0546] In Example 287, the subject matter of Examples 284-286 includes: wherein the notification area is a separate sub-display.

[0547] In Example 288, the subject matter of Examples 284-287 includes: a unit for providing a user interface having selectable options for showing different notification types on the sub-display; a unit for receiving a selection of a first notification type to be displayed on the sub-display; and wherein causing the display of the notification to include determining that the notification is of the first notification type.

[0548] In Example 289, the subject matter of Example 288 includes: wherein the user interface is displayed on the content source device.

[0549] In Example 290, the subject matter of Examples 288-289 includes: wherein the user interface is displayed on the display screen.

[0550] In Example 291, the subject matter of Examples 284-290 includes: a unit for identifying an input on the notification area of ​​the display surface, the input matching a predefined gesture to dismiss the notification; and a unit for dismissing the notification by removing the notification information from the display screen in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to dismiss the notification.

[0551] In Example 292, the subject matter of Example 291 includes: wherein causing dismissal of the notification to include sending a message to the content source device to dismiss the notification.

[0552] In Example 293, the subject matter of Examples 284-292 includes: a unit for identifying an input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and a unit for displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification.

[0553] In Example 294, the subject matter of Example 293 includes: wherein the unit for displaying content corresponding to the notification on the content shared by the content source device in the sub-display screen of the display screen includes: a unit for completely replacing the content shared by the content source device in the sub-display screen with the content corresponding to the notification.

[0554] In Example 295, the subject matter of Examples 284-294 includes: a unit for identifying input on the notification area of ​​the display surface, the input matching a predefined gesture to expand the notification; and a unit for displaying content corresponding to the notification in a second sub-display screen automatically created in response to identifying that the input on the notification area of ​​the display screen matches the predefined gesture to expand the notification, the second sub-display screen being at a different position from the sub-display screen.

[0555] In Example 296, the subject matter of Examples 284-295 includes: wherein the notification is not generated from an application that generates the content shared by the content source device.

[0556] Example 297 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-296.

[0557] Example 298 is an apparatus comprising means for implementing any of Examples 1-296.

[0558] Example 299 is a system that implements any of Examples 1-296.

[0559] Example 300 is a method of implementing any of Examples 1-296.

Claims

1. A computing device for causing a notification to be displayed using a display screen, the computing device comprising: one or more hardware processors; A memory storing instructions that, when executed, cause the one or more hardware processors to perform operations including: receiving content sent from a remote content source device; identifying a notification of the remote content source device, the notification being present in a portion of the content sent from the remote content source device, before displaying the received content sent from the remote content source device in an area of ​​a display device, the display device being remote from the remote content source device; removing the notification from the portion of the content sent by the remote content source device to create an extracted notification and content sent from the remote content source device without showing the notification; causing display of the extracted notification in a notification area of ​​the display device that would otherwise be displayed in the portion of the content, the notification area not overlapping the area of ​​the display device; as well as The content transmitted from the remote content source device with the visual representation of the notification omitted is caused to be displayed in the area of ​​the display device without displaying the notification in the area of ​​the display device.

2. The computing device of claim 1, wherein: The notification is generated by an application running on the remote content source device.

3. The computing device of claim 1, wherein: The notification area is a visually distinct region adjacent to an outline of the area of ​​the display device and sharing a border.

4. The computing device of claim 1, wherein: The operations also include: receiving an input from a user corresponding to the notification area; and At least one action is taken in response to the input.

5. The computing device of claim 4, wherein: The at least one action includes dismissing the notification.

6. The computing device of claim 4, wherein: The input is a drag gesture, and wherein the at least one action includes generating an animation showing the notification being torn off from the notification area.

7. The computing device of claim 4, wherein: The notification is for a received message sent by a sender, and wherein the input is an inline reply allowing the user to enter reply text that is sent to the sender of the received message.

8. A method for causing a notification to be displayed using a display screen, the method comprising: receiving content sent from a remote content source device; identifying a notification of the remote content source device, the notification being present in a portion of the content sent from the remote content source device, before displaying the received content sent from the remote content source device in an area of ​​a display device, the display device being remote from the remote content source device; removing the notification from the portion of the content sent by the remote content source device to create an extracted notification and content sent from the remote content source device without showing the notification; causing display of the extracted notification in a notification area of ​​the display device that would otherwise be displayed in the portion of the content sent from the remote content source device, the notification area not overlapping the area of ​​the display device; as well as The content transmitted from the remote content source device with the visual representation of the notification omitted is caused to be displayed in the area of ​​the display device without displaying the notification in the area of ​​the display device.

9. The method according to claim 8, wherein: The notification is generated by an application running on the remote content source device.

10. The method according to claim 8, wherein: The notification area is a visually distinct region adjacent to an outline of the area of ​​the display device and sharing a border.

11. The method according to claim 8, wherein: The method further comprises: receiving an input from a user corresponding to the notification area; and At least one action is taken in response to the input.

12. The method according to claim 11, wherein: The at least one action includes dismissing the notification.

13. The method according to claim 11, wherein: The input is a drag gesture, and wherein the at least one action includes generating an animation showing the notification being torn off from the notification area.

14. The method according to claim 11, wherein: The notification is for a received message sent by a sender, and wherein the input is an inline reply allowing the user to enter reply text that is sent to the sender of the received message.

15. A non-transitory machine-readable medium storing instructions for causing a notification to be displayed using a display screen, the instructions, when executed, causing a machine to perform operations comprising: receiving content sent from a remote content source device; identifying a notification of the remote content source device, the notification being present in a portion of the content sent from the remote content source device, before displaying the received content sent from the remote content source device in an area of ​​a display device, the display device being remote from the remote content source device; removing the notification from the portion of the content sent by the remote content source device to create an extracted notification and content sent from the remote content source device without showing the notification; causing display of the extracted notification in a notification area of ​​the display device that would otherwise be displayed in the portion of the content sent from the remote content source device, the notification area not overlapping the area of ​​the display device; as well as The content transmitted from the remote content source device with the visual representation of the notification omitted is caused to be displayed in the area of ​​the display device without displaying the notification in the area of ​​the display device.

16. The non-transitory machine-readable medium of claim 15, wherein: The notification is generated by an application running on the remote content source device.

17. The non-transitory machine-readable medium of claim 15, wherein: The notification area is a visually distinct region adjacent to an outline of the area of ​​the display device and sharing a border.

18. The non-transitory machine-readable medium of claim 15, wherein: The operations also include: receiving an input from a user corresponding to the notification area; and At least one action is taken in response to the input.

19. The non-transitory machine-readable medium of claim 18, wherein: The at least one action includes dismissing the notification.

20. The non-transitory machine-readable medium of claim 18, wherein: The input is a drag gesture, and wherein the at least one action includes generating an animation showing the notification being torn off from the notification area.

21. A computing device for remote content display, the computing device comprising: one or more hardware processors; A memory storing instructions that, when executed, cause the one or more hardware processors to perform operations including: identifying content for display in a first display area; causing the identified content to be displayed in the first display area; identifying a notification that needs to be displayed, the notification, if displayed in the first display area, will overlap content displayed in a previous video frame in the first display area; creating an extracted notification by separating the notification from the identified content to be displayed in the first display area; a notification indicating the extraction is displayed in a second display area; causing the extracted notification to be displayed in a second display area but not in the first display area, where the extracted notification would otherwise be displayed within the identified content of the first display area; as well as The identified content is caused to be displayed in the first display area with a visual representation of the notification omitted.

22. The computing device of claim 21, wherein: The notification is generated by an application running on the computing device.

23. The computing device of claim 21, wherein: The operation of identifying content for display in the first display area includes receiving a selection input selecting the content.

24. The computing device of claim 21, wherein: The first display area and the second display area are sub-display screens, and the sizes and positions of the sub-display screens are created by a user on the remote display device.

25. The computing device of claim 21, wherein: The operation of causing the identified content to be displayed in the first display area with the visual representation of the notification omitted includes replacing pixels showing the notification with pixels in the same area of ​​a previous frame.

26. The computing device of claim 21, wherein: The content for display is screen sharing data of the entire screen of the sharing user.

27. The computing device of claim 21, wherein: The operation of causing the identified content with the visual representation of the notification omitted to be displayed in the first display area includes: transmitting the identified content with the visual representation omitted.

28. A method for remote content display, the method comprising: Using one or more hardware processors of a computing device to perform operations including: identifying content for display in a first display area; causing the identified content to be displayed in the first display area; identifying a notification that needs to be displayed, the notification, if displayed in the first display area, will overlap content displayed in a previous video frame in the first display area; creating an extracted notification by separating the notification from the identified content to be displayed in the first display area; a notification indicating the extraction is displayed in a second display area; causing the extracted notification to be displayed in a second display area but not in the first display area, where the extracted notification would otherwise be displayed within the identified content of the first display area; as well as The identified content is caused to be displayed in the first display area with the visual representation of the notification omitted.

29. The method according to claim 28, wherein: The notification is generated by an application running on the computing device.

30. The method of claim 28, wherein: Identifying content for display in the first display area includes receiving a selection input selecting the content.

31. The method of claim 28, wherein: The first display area and the second display area are sub-display screens, and the sizes and positions of the sub-display screens are created by the user on the remote display device.

32. The method of claim 28, wherein: The operation of causing the identified content to be displayed in the first display area with the visual representation of the notification omitted includes replacing pixels showing the notification with pixels in the same area of ​​a previous frame.

33. The method of claim 28, wherein: The content for display is screen sharing data of the entire screen of the sharing user.

34. The method of claim 28, wherein: Causing display of the identified content with the visual representation of the notification omitted in the first display area includes transmitting the identified content with the visual representation omitted.

35. A computer-readable storage device storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising: identifying content for display in a first display area; causing the identified content to be displayed in the first display area; identifying a notification that needs to be displayed, the notification, if displayed in the first display area, will overlap content displayed in a previous video frame in the first display area; creating an extracted notification by separating the notification from the identified content to be displayed in the first display area; a notification indicating the extraction is displayed in a second display area; causing the extracted notification to be displayed in a second display area but not in the first display area, where the extracted notification would otherwise be displayed within the identified content of the first display area; as well as The identified content is caused to be displayed in the first display area with the visual representation of the notification omitted.

36. The computer readable storage device of claim 35, wherein: The notification is generated by an application running on the computing device.

37. The computer readable storage device of claim 35, wherein: The operation of identifying content for display in the first display area includes receiving a selection input selecting the content.

38. The computer readable storage device of claim 35, wherein: The first display area and the second display area are sub-display screens, and the sizes and positions of the sub-display screens are created by a user on the remote display device.

39. The computer readable storage device of claim 35, wherein: The operation of causing the identified content to be displayed in the first display area with the visual representation of the notification omitted includes replacing pixels showing the notification with pixels in the same area of ​​a previous frame.

40. The computer readable storage device of claim 35, wherein: The content for display is screen sharing data of the entire screen of the sharing user.