Multi-user system

Abstract

A multi-user system (1) comprising a first large-size display (110) at a first location (100) and a second display (210) at a second, remote, location (200). A tracking module (140) identifies skeletons of users (120, 130) at the first location (100) and determines respective locations of interest (g, h) in the first display (110). Awareness cues (AC120, AC130) are built and transmitted to the second display (210) to adequately convey behavioral information to the second location (200). (Figure 1)

Description

MULTI-USER SYSTEM

[0001] The present disclosure relates to a system that is intended to be used by a plurality of users and especially users located in different locations.

[0002] Real-time remote communication has become a regular way of exchanging information. Videoconference software and hardware systems enable users to exchange a video stream from one location to another location. When using a computer, the video stream can enable users to share the content of their screen. The use of a mouse pointer can greatly increase the mutual understanding between users located in remote locations.

[0003] The use of an input device (a mouse, a token, a keyboard, etc.) is however a strong drawback whenseveral userslocated at one location interact with one or more users located at a second location: either each one of the users must be holding a respective input device, or the input device must be passed along from user to user, to enable each one of the users to participate in the discussion. Passing a token along users is really cumbersome when there are a high number of users located in a large room. This is also a source of confusion for the participants located in the remote location as it is not easy for them to know in real-time who among the participants at the first location is actually holding the token.

[0004] Additionally, known systems hardly ever address the transmission of non-verbal aspects, such as gestures, gaze, facial expressions, emotions, or body postures.

[0005] When users interact with alarge-size screen, such as wall-size screens that cover a substantial portion of the walls of a room, it becomes even harder to properly communicate between two distinct locations without losing too much information between the two locations. Indeed, the size of the screen enhances the difficulties mentioned above. For instance, a mouse is not adapted for large-size screen as the cursor is easily lost and the hand movements would be larger than the human anatomy allows. Hence, large-size screens are not known to be an appropriate way of communicating in mixed-presence scenarios.

[0006] There is therefore a strong need for a system that enables mixed-presence use of large-size screens. There is also a need for such a system that conveys sufficient information from one site to the other, without suffering from the hindrance of the hardware constraints.Summary of the invention

[0007] An aim of the present disclosure is to provide for this need. In particular, the system of the present invention enables real-time communication between a location where several users interact with a large-size screen, and a remote location. The system makes it easier and more convenient to control the overall operations without being constrained by the limitations of hardware (or the number of hardware ports). Also, the system of the invention is not bound by a limit in terms of space (which could be the case when using user input devices, even wirelessly) or in terms of number of users (as it is easier in terms of logistics to use the system with many users without the need to provide them with an input device). Additionally, the present disclosure provides for a system that continuously guides the users to focus on relevant information, so as to greatly improve communication and make the decision process more efficient in a collaborative scenario. Furthermore, since the mixed-presence system of the present invention is efficient, it discourages in-presence meetings and thereby saves transportation-related costs and energy, reduces CO2 emissions, and thus is environment-friendly. Further benefits of the present invention may appear hereinafter.

[0008] According to the present disclosure, a multi-user system is provided comprising: at a first location, a first large-size display which at least two users may interact with, the first large-size display comprising a first two-dimensional coordinate system; at at least one second location remote from the first location, a second display comprising a second two-dimensional coordinate system; a data link connecting the first large-size display to the second display; a tracking module comprising: a scene acquiring device, configured to continuously capture a scene at the first location; a skeleton identifying module, configured to: detect skeletons in the captured scene; distinguish the skeletons one from another by retrieving a pre-determined identifier of a recognized skeleton or allocating a respective identifier to a non-recognized skeleton; determine a position of each identified skeleton; and determine at least one of: a gaze direction of each identified skeleton; and a hand-pointing direction of each identified skeleton; a target locator configured to continuously determine, for each identified skeleton, based on their position, and the gaze direction and / or the hand-pointing direction, and based on a known relative arrangement of the scene acquiring device with respect to the first large-size display, a gaze location of interest in the first large-size display and / or a hand-pointing location of interest in the first large-size display, each location of interest defining a pixel or a group of pixels of the first large-size display, the pixel or group of pixels being identified by their coordinates in the first coordinate system; and an awareness module configured to, continuously: allocate to each identified skeleton a gaze awareness cue and / or a hand-pointing awareness cue or retrieve previously allocated awareness cue(s), the awareness cue(s) allocated to a skeleton being visually distinct from the awareness cue(s) allocated to another skeleton; apply a transfer function to the pixel or group of pixels of each location of interest to determine their respective coordinates in the second coordinate system; constitute or update, for each identified skeleton, a gaze tuple and / or a hand-pointing tuple, each gaze tuple comprising a gaze awareness cue of an identified skeleton and the coordinates of the gaze location of interest of the same identified skeleton in the second coordinate system, and each hand-pointing tuple comprising a hand-pointing awareness cue of an identified skeleton and the coordinates of the hand-pointing location of interest of the same identified skeleton in the second coordinate system; send, for each identified skeleton, the gaze tuple and / or hand-pointing tuple to the second display via the data link, the second display being configured to display each received awareness cue of the identified skeleton at the respective coordinates in the second coordinate system.

[0009] This system enables to conveniently use large-size screens with multiple users in mixed-presence scenarios. It further frees the users from the constraint of holding hardware devices (reducing the use of physical ports) while at the same time transmitting behavioral parameters of the users from one location to another location. In particular, it makes possible for the users in the second location to identify in real-time and without confusion the places on the screen where their attention should be focused.Definitions

[0010] The system is intended for being amulti-usersystem in the sense that several human beings can simultaneously use the system. In particular, at least two users can be located at the first location while at least one user is located at the second, remote, location.

[0011] The ordinal labels “first”, “second”, “third”, … are used to distinguish the features of the system from each other. A higher ordinal number (e.g., second) does not necessarily mean that similar features of lower ordinal number (e.g., first) are part of the system. Similarly, a lower number (e.g., first) does not imply that a second item of the same kind is part of the system.

[0012] The various modules detailed herein can be combined as a single module or can be spread apart as separate, interconnected, modules. It is implicit that a module may contain the necessary hardware and software components to assure its functionality.

[0013] At a first location, a firstlarge-size displayis arranged. In the present context, large size is to be understood as a size of at least 2 meters diagonally. In some examples the large-size display occupies several walls of a room. In some examples, the large-size display is made of several planar and / or curved screens. The display comprises a multitude of pixels or a multitude of atomic visual elements which can be independently controlled one from the others. The display comprises any appropriate hardware or software elements (e.g., CPU, GPU, memory, communication port, etc.) to properly control each pixel, including but not limited to, their color, contrast, brightness, frequency, etc.

[0014] The first large-size display has atwo-dimensional coordinate system. In other words, each pixel (or the like) of the first large-size display can be uniquely located by its coordinates consisting in a unique pair of values.

[0015] The second location isremotefrom the first location. This means that both locations are spaced apart and there is no overlap between these locations. In some examples, the first and second locations are separated from one another by at least 10 meters, at least 10 km or at least 100 km. A location can be a room within a building.

[0016] The system comprises asecond displayat the second location. In some examples, the second display may be a computer display (tablet, smartphone, multiple-screens display, etc.), used by a unique user. In some examples, the second display is a large-size display. In some examples, the second location mirrors the first location and is also equipped with the equipment allowing to transmit, to the first location, behavioral parameters of the user(s) located at the second display.

[0017] The system containsat leastone second location, meaning that further displays may be arranged in further locations. These further displays may be similar to the second display discussed herein. They may be connected to the first location via the same data link or via a distinct and / or specific data link.

[0018] The second two-dimensional coordinate system is a coordinate system enabling pixels (or the like) of the second display to be located by a unique pair of values.

[0019] The first and second displays are connected through adata link. The data link may be based on any appropriate transfer protocol that can convey information from the first display to the second display (e.g., MQTT). The data link may be used to establish a video or sound connection between the users at different locations. In some examples, the data link is specific to the transfer of the tuples, and voice or video connection is established by another link between the locations.

[0020] The system comprises ascene acquiring device. This device may be made of one or several image capture devices, and / or of any other device suitable for capturing a scene, including the users located at the first location. In some examples, the scene acquiring device may comprise a depth camera, for instance using a time-of-flight technique to detect location of items or users in a 3D coordinate system. In some examples, the scene acquiring device comprises an infrared camera.

[0021] The scene is acquiredcontinuously. In the present context, continuously should be understood as a repetitive process capturing the scene at a given frequency which may be greater than 10 Hz, preferably greater than 24 Hz, more preferably greater than 50 Hz.

[0022] The skeleton building module contains adequate software and / or hardware components to build skeletons out of the scene containing human beings. Askeletonis herein to be understood as an assembly of joints, i.e., key points of a corresponding human silhouette, each joint being uniquely located in a 3D coordinate system. The skeleton can be made of 5 to 30 joints. The system works under the hypothesis that each skeleton is unique to a given human being: the static appearance is unique (a unique combination of distances between two adjacent joints), and / or dynamic aspects of the skeletons is unique (head, hand or gait movements are unique). Hence, each uniquely identified skeleton corresponds to exactly one user.

[0023] Once the skeleton building module has built skeletons out of the scene, it may retrieve known skeletons from a pre-recorded database and allocate them a previously given identifier. The non-retrieved skeletons are given a new identifier. Obviously, the system may contain appropriate functions to personalize or to pre-select the identifiers based on predetermined criteria.

[0024] The skeleton building module can determine a position, a gaze direction and a hand-pointing direction of each identified skeleton. To do so, the module has a knowledge of a set of the first location, i.e., a 3D coordinate system with predetermined position of the scene acquiring device, and potentially physical markers positioned in the scene to serve as reference for the skeleton building module. The skeleton building module is then able to determine a position of each skeleton (e.g., one combination of 3D coordinates of all the joints of the skeleton, or the coordinates of one key joint), to determine a gaze direction (e.g., a head joint direction of a skeleton in the 3D coordinate system), and to determine a hand-pointing direction (e.g., the direction between two joints of an arm of a skeleton). In some examples, the gaze direction is calculated from the orientation of the head of a skeleton. In some examples, the system further comprises image capturing devices (e.g., IR glint detection) that can capture the eyes of the users and can detect their sight direction. For hand-pointing direction determination, sufficient information is captured to create a skeleton of a hand of the user.

[0025] The target locator of the tracking module can then calculate a location of interest of each skeleton. To do so, the set of the first location (relative position of scene acquiring devices and display pixels) is known: e.g., each pixel of the first large-size display is located in the 3D coordinate system (beyond each pixel having coordinates in the first 2D coordinate system of the first display). In the 3D coordinate system, the position of a skeleton and the gaze or hand-pointing direction build a vector (an origin, a direction, and a value which may be the distance between the skeleton and the display). The intersection of that vector with the plane (or curved surface) of the display determines the location of interest. A location of interest defines a pixel or a group of pixels of the first large-size display, the pixel or group of pixels being identified by their coordinates in the first coordinate system.

[0026] A module of the system generates and transmits awareness cues. An awareness cue may be a visual cue defined by at least one of: a shape, a size, a color, a brightness, a contrast, a dynamic (blinking, modulating in size, shape, color, etc.). A database of available awareness cues can be prerecorded. The module will pick from this database, arbitrarily or according to predefined rules, one or more awareness cues for each skeleton. If a skeleton has a gaze direction and a gaze location of interest, a gaze awareness cue will be chosen for this skeleton. If a skeleton has a hand-pointing direction and a hand-pointing location of interest, a hand-pointing awareness cue will be chosen for this skeleton. The gaze and hand-pointing cues may be different for a given skeleton. The system may provide for the personalization of awareness cues, so that each or some of the users may choose or may design their awareness cue (color, shape, etc.).

[0027] Two distinct skeletons do not share the same awareness cue, i.e., at least one of: the shape, size, color, brightness, contrast, or dynamic differs between the two skeletons.

[0028] If a skeleton has been retrieved from a previous interaction of the user with the system, the awareness cues of this skeleton may be retrieved as well. A new skeleton will be allocated new cues.

[0029] The second display may be displaying content differently from the first display: their size, resolution or arrangement of windows may differ. Hence, in order to properly convey the behavioral parameters of the users to the second location, a mapping of the content that is displayed at the location of interest and the content displayed on the second display should be made. A transfer function may be used to map each pixel of the first display to one or more pixels of the second display. The transfer function may be a predetermined function (e.g., a matrix transforming a pair of coordinates of one pixel in the first 2D coordinate system into a pair of coordinates of the corresponding pixel in the second 2D coordinate system). Alternatively, the transfer function may be dynamically adjusted: just before applying the transfer function, a comparison can be made to seek for similitudes between the content displayed on the two displays, or for seeking in the second display, the location where the location of interest of the first display is located. This may require identifying the content that is displayed in the location of interest and to seek for that content in the second display. In some examples, a dedicated structure (hardware / software) can maintain up to date a transfer function, as the users of one location or the other location rearrange the windows or the resolution of their respective display.

[0030] The content sent from the first display to the second display can take the form of tuples, i.e., a series of organized elements. For example, each tuple can comprise an awareness cue which can take any appropriate form (e.g., a PNG, BPM file, enriched with parameter values, such as a size, color, brightness, contrast, dynamic behavior, etc.). Each tuple also comprises the coordinates of the location of interest in the second 2D coordinate system so that the second display (or more specifically its controller) is made aware of where to display the respective awareness cue.

[0031] The invention also relates to the corresponding method, i.e., a computer-implemented method performed by the above-mentioned system, the method comprising: identifying skeletons and their location(s) of interest on the first large-size display; allocating distinct awareness cues to distinct identified skeleton; and displaying the awareness cues on the second display, at a location of the second display mapping the location of interest on the first large-size display.Further embodiments

[0032] In some examples, the scene acquiring device comprises an image capturing device configured to capture image frames of the scene and the tracking module is configured to, for each detected skeleton: identify head zones of the image frames which contain a head of a skeleton; crop image frames down to the head zone; and run a recognition algorithm on the head zone to identify a user, wherein the awareness module selects a gaze / hand-pointing awareness cue as a function of the identified user.

[0033] Thus, in combination with a skeleton recognition, a facial recognition can enhance the reliability of the recognition. Cropping the images enables to extract several faces from an original image, and to send these cropped images simultaneously to a dedicated AI for processing to the facial recognition. Advantageously, the skeletons can be used to crop the image appropriately: there is no need to grossly scan the image to detect potential faces before cropping, because the skeletons already provide for the information of the rough location of the face within an image.

[0034] In some examples, the scene acquiring device comprises at least two image capture devices and is configured to merge skeleton data captured by the at least two image capture devices.

[0035] This may be advantageous when the first location is particularly spacious since a single camera may not suffice to capture all of the users. However, when several cameras are used, it is important to merge the skeletons that are captured by different cameras, so that the system does not falsely detect two times a same user. In addition, merging skeletons seen from two points of view enhances the quality of the determination of the position, gaze direction and hand-pointing direction.

[0036] In some examples, the tracking module further comprises a gesture recognition module configured to: crop arm portions of the captured scene, each of the arm portions containing one or both arms of an identified skeleton; and run a recognition algorithm on the arm portions to identify a gesture among prerecorded gestures, the system being further configured to display information related to the recognized gesture on the second large-size display.

[0037] This allows to further increase the amount of information that is transmitted to the second display. As an example, a user who acknowledges or agrees with another user may have a gesture that can be detected and indicated in one way or another on the second display. Another example could be the detection of a user raising his hand to indicate his / her wish to talk. Other gestures may be looked for in a preset or trained database. Gestures are here mostly if not entirely mid-air gestures, in contrast to touch / point gestures which a user would perform with a finger touching a screen.

[0038] In some examples, the awareness cues have a respective size that is determined as a function of the location of the corresponding user.

[0039] To further transmit behavioral information, the size of the cues may be a simple, efficient and universal way to let the users at the second location understand where the respective users are located in the scene of the first location: when a user walks closer to the first display, the size of his awareness cue on the second display may become bigger, and vice versa. In some examples, the detected gesture may also impact the size, shape or color of the awareness cue.

[0040] In some examples, the users interact with the first large-size display exclusively and directly through their movements, the system being void of hand-held token, head-worn, or body-worn tokens, mouse or keyboard.

[0041] The system allows indeed to be run without the users wearing any equipment (motion capture suit, head-worn hat, etc.) or without holding a device (token, mouse, keyboard). Especially when discussing or pointing at a particular element shown on the display, the system does not require any pointer or mouse to be used, as it automatically transmits to the second location the specific areas of interest on the display.

[0042] In some examples, the first and / or second display may comprise a touchscreen. The skeleton detection enables to detect which of the identified skeleton is actually touching a particular portion of the screen, thereby offering the further opportunity for the users at the second location to undoubtedly understand which one of the users of the first location is pointing at which content of the first display.

[0043] In some examples, the second display is a large-size display. As noted above, the second display may then also enable behavioral parameters of multiple users located at the second location to be transmitted to the first display, as the second display may be appropriately equipped to transfer awareness cues to the first location.

[0044] In some examples, the first large-size display is made of at least one display arranged in a vertical plane. This advantageously gives all the freedom for the users to move in the scene of the first location, further enhancing the experience and the amount of behavioral information which can be captured and transmitted to the second location.

[0045] In some examples of the method discussed above, the steps of identifying, allocating and displaying are performed continuously. In other examples, these steps may be performed sequentially or on demand.

[0046] Thus, the various embodiments of the invention enable to use large-size displays in a mixed-presence scenario without losing behavioral information and without the burden of the hardware constraint.Brief description of the figures

[0047] Further benefits may appear in view of the appended drawings wherein:

[0048] shows a general overview of a system;

[0049] represents a schematic view of the functional modules;

[0050] illustrates skeletons;

[0051] shows a method.Detailed description of the figures.

[0052] The figures represent various aspects of the invention in a schematic and illustrative manner. Those skilled in the art will recognize that variations can be performed on the illustrated examples without departing from the scope of the appended claims.

[0053] shows an illustrative example of a system 1 comprising, at a first location 100, a first, large-size display 110. In this example, the large-size display 110 is made of three screens 111, 112, 113, each of a size that is substantially greater than the size of a human being. Each screen 111, 112, 113 is made of a plurality of pixels or similar optical units which can be controlled independently one from the other. A 2D coordinate system (u,v) can be fixed relative to the display 110: each pixel p of the display 110 can be located by a unique pair of values (u(p), v(p)).

[0054] A 3D coordinate system (x, y, z) can be set at the first location 100. Each pixel p can be located in the 3D coordinate system by a unique triplet of respective values (x(p), y(p), z(p)).

[0055] The first display 110 can be simultaneously used by a plurality of users 120, 130 at their respective positions p1, p2.

[0056] A tracking module 140 is used to track the interactions of the users 120, 130 with the display 110. The tracking module 140 may comprise a scene acquiring device 142, foreseen to capture the scene at the first location 100. The scene acquiring device 142 may contain any appropriate detector, sensor, camera, of any appropriate technology. For example, the scene acquiring device 142 may contain a depth camera, an IR camera, and / or a visible color camera. In some examples, the scene acquiring device 142 may contain several (distinct or similar) devices to capture the scene with a wide enough angle, or with a plurality of distinct points of view. In such a case, the content captured by the different devices may be merged as will be explained below.

[0057] At a second location 200, the system 1 further comprises a second display 210. The second display 210 may be a large-size display similar to the first display. The second display 210 may be a display of smaller size. The second display 210 may be provided with a respective 2D coordinate system (u’, v’). Each pixel p’ of the second display can be uniquely located with a pair of coordinates (u’(p’), v’(p’)).

[0058] A portion of the second display 210 may display a content that is similar to a content displayed on the first display 110. Hence a portion of both displays may display common content. A transfer function can be defined, which, for each pixel of the common content, transforms the coordinates (u(p), v(p)) of a given pixel in the first 2D coordinate system (u, v) into coordinates in the second 2D coordinate system (u’(p’), v’(p’)), of a pixel p that displays a content on the second display corresponding to the content displayed on the pixel p’ of the first display. The transfer function may be predetermined and static (i.e., at each moment in time, it is always the same pixel p’ that displays the content corresponding to the content of pixel p in the first display). Alternatively, the users may modify the arrangement of windows, alter the resolution, hide some content, or add a further screen to the first or second display. Hence, it may be relevant to provide for a dynamic transfer function, which evolves dynamically. An external module may keep up to date at each moment in time a correspondence table between the pixels of the first display and the corresponding pixels of the second display. Alternatively, detection means can detect similarities between the two screens.

[0059] The second display 210 may be used by at least one user 220.

[0060] A data link 300 connects the first display 110 to the second display 210. Any one-way or two-ways protocol can be used to establish the communication between the displays 110, 210.

[0061] The tracking module 140 is configured, among others, to distinguish the users 120, 130 and to detect a gaze or hand-point direction.

[0062] For illustration purposes, user 120 points with his / her hand in a direction H. The intersection of the direction H and the display 110 defines a location of interest h, which is a pixel or a group of pixels that constitutes the intersection, in the 3D coordinate system, of the direction H and the display 110.

[0063] Similarly, user 130 looks in a gaze direction G which intersects the display in a location of interest g.

[0064] An awareness module 150 can apply the transfer function to the locations of interest g, h, to transform them into g’ and h’, i.e., a corresponding location in the content displayed by the second display 210. The awareness module 150 selects an awareness cue for each user (and for each activity (gaze, hand-pointing, etc.)). The awareness cues AC120, AC130 may have several parameters: a size, a shape, one or more colors, a brightness, a contrast, etc. The awareness cues AC120, AC130 are distinct through at least one distinct parameter.

[0065] The awareness cues AC120, AC130 are transferred to the second display together with the locations of interest g’, h’. The user 200 is thus able to immediately and clearly see which of the users 120, 130 is looking / pointing at which content.

[0066] The system enables therefore a real-time communication between a large-size display and a remote location. There is no need for the users 120, 130 to hold a respective token, or to pass along one token. The ports of the first display are thus free, and the amount of data that is to be transferred to the second display is reduced. Also, the system of the invention is not bound by a limit in terms of space or number of users: there isn’t any limitation in terms of cable length, or wireless reachability. As the process is performed continuously, the user 220 is continuously guided in his interaction with the display 210.

[0067] shows a diagram representing the flow of information in the system of. The tracking module 140 observes the scene and determines the locations of interest and / or gestures of the users 120, 130.

[0068] To do so, the tracking module 140 comprises a scene acquiring device 142 which outputs a captured scene 143 from a physical analogic scene 141. The scene acquiring device 142 may comprise one more cameras, including a depth camera and a color camera.

[0069] The captured scene 143 may be in any appropriate format (2D, 3D, image frames, video stream, video enriched with depth information, etc.).

[0070] A skeleton identifying module 144 receives the captured scene 143 and identifies skeletons S in these images (see e.g.,). Each skeleton is identified, either by recognizing a known skeleton in a database 160, or by generating a new identifier for a newly discovered skeleton. Other processes may be used, for example, by creating a new identifier even for a previously known skeleton. The skeleton identifying module 144 determines the position of the skeleton. This may be made by the depth information of the captured image 143 or by comparing the captured image 143 with markers positioned in the scene. The position of a skeleton may be a position of a key joint of the skeleton. The gaze direction and hand-pointing direction can also be determined by the skeleton building module.

[0071] Optionally, the skeleton may be used to crop the captured image, for sending cropped portions of the image to a face recognition module 145 (for example a specially trained AI) and for sending cropped arm portions of the image to a gesture recognition module 146 (for example a specially trained AI). Gesture and face recognition may open access to the users to various functionalities or interactions with the system.

[0072] The skeleton building module 144 generates, as output 147, for each skeleton, its identifier, position and gaze and / or hand-pointing direction, as well as potential recognized face or gesture.

[0073] A target locator 148 can then use this information to identify, knowing the set 170 (relative positions of the scene acquiring devices and the displays), locations of interest g, h. Indeed, these locations are geometric intersections of the directions of gaze or hand-pointing (originating from the respective positions of the skeletons) and the display surface 110.

[0074] The locations of interest g, h, are transmitted, with the identifier of each skeleton, 149, to the awareness module 150 which applies the transfer function to the locations of interest g, h, to transform them into g’ and h’. The awareness module 150 also selects awareness modules from a database 180 or generate them. Tuples 151 are then formed, containing the awareness cues AC120, AC130 and their respective locations g’, h’.

[0075] The tuples are then sent to the second display 210, which is configured to display them at the appropriate locations g’, h’.

[0076] Independently from the locations of interest, information related to the face and / or gesture of the users may be transmitted to the second display 210 (see dashed lines).

[0077] It should be understood thatis an illustrative and functional representation of the process performed with the system aiming at facilitating the understanding of the main concepts of the present disclosure. The skilled person will understand that the word “module” can be used here to represent hardware and / or software which may be dedicated to a single task, or which may be part of a more complex element. Some or all of the functions that are carried out by the modules may be integrated into a single hardware and / or software device. Alternatively, some or all of the modules may be separated and may communicated with one another with appropriate means. The computation power may be centralized or distributed. Some or all of the modules may be incorporated into the controller of the first display 110, and / or into the controller of the second display 210, and / or in a separate server, located at the first location or at the second location, or located remotely from both locations.

[0078] shows an example of two skeletons S1, S2 that can be detected by the skeleton building module. A skeleton S is a series of joints or nodes, each one having at a given moment, a distinct location in the 3D coordinate system. The skeleton S is extracted from the captured image by appropriate processing.

[0079] For the optional face and gesture recognition, the skeleton may be used to quickly define areas which are likely to contain a face (head zone, HZ1, HZ2) or areas that are likely to contain one or both arms (arm zones, AZ1a, HZ1b, HZ2). As the image contains several users, cropping the image based on the skeletons enables to feed the cropped head zones and arm zones in parallel to recognition modules. A further advantage of using skeletons for recognizing faces or gestures, is that a traceability can be maintained between the identity of the skeleton and the recognized face or gesture: the system will know that the face recognized from head zone HZ1 belongs to skeleton S1.

[0080] For facial recognition, the system may rely on a face recognition library, which uses 4 steps: (1) face finding, (2) face alignment, (3) facial feature extraction, and (4) facial feature comparison. Step 1 may be based on the Histogram of Oriented Gradients (HOG) method; step 2 may rely on the face landmark estimation algorithm to find landmarks that are used to align faces so that the eyes and mouth are centered properly (the resulting image will be distorted). Then, at step 3, a deep convolutional neural network may be used to “encode” faces into what is referred to as an embedding. Then, at step 4, the embedding for a given face may be compared to the set of known faces via a support vector machine classifier, so that the closest match may be returned.

[0081] In some examples, the scene acquiring device contains at least two capturing devices (cameras), positioned at their respective locations. Hence, more than one captured image frame is generated at each given time. The skeleton building module 144 is then fed with several images depicting co-occurring scenes. The skeletons are identified on each co-occurring scene. The skeletons are then merged: skeletons belonging to distinct users remain distinct, while skeletons belonging to a same user are merged. To recognize similar or distinct skeletons, the system does know the relative positioning of the capture devices and can apply a similarity algorithm. Additionally, or alternatively, the skeleton captured by one capture device can be projected into an expected position of an image captured by another capture device, knowing their relative position and the position (e.g., through depth map) of the skeleton. Merging the skeletons may prevent errors or improve quality when detecting skeletons as two distinct sources of information are merged to build the skeleton.

[0082] The scene capture device may advantageously comprise a depth camera for identifying the position of the joints of the skeletons. A color camera may be advantageous for face recognition.

[0083] summarizes a method 1000 with the three main steps as follows: at step 1100, skeletons are identified and their respective locations of interest in the first large-size display are calculated; at step 1200, awareness cues are allocated to the skeletons; at step 1300, the awareness cues are displayed on the second display, at locations mapping the locations of interest. The method may be carried out with a system as described above. The various steps may be performed with the various modules of the system.

[0084] In some examples, the visual cues can be complemented with an auditory signal (i.e., an auditory awareness cue), which can be provided to the user at the second location. The auditory cue can be localized in 3D (stereo, Dolby Atmos®, etc.) and / or can be of a nature (ring, tone, melody, etc.) that is respective to a particular user present at the first location.

Claims

Multi-user system (1) comprising:at a first location (100), a first large-size display (110) which at least two users (120, 130) may interact with, the first large-size display (110) comprising a first two-dimensional coordinate system (u, v);at at least one second location (200) remote from the first location (100), a second display (210) comprising a second two-dimensional coordinate system (u’, v’);a data link (300) connecting the first large-size display (110) to the second display (210);a tracking module (140) comprising:a scene acquiring device (142), configured to continuously capture a scene (141) at the first location (100);a skeleton identifying module (144), configured to:detect skeletons (S1, S2) in the captured scene (143);distinguish the skeletons (S1, S2) one from another by retrieving a pre-determined identifier of a recognized skeleton or allocating a respective identifier to a non-recognized skeleton;determine a position (p1, p2) of each identified skeleton; anddetermine at least one of:a gaze direction (G) of each identified skeleton; anda hand-pointing direction (H) of each identified skeleton;a target locator (148) configured to continuously determine, for each identified skeleton, based on their position, and the gaze direction (G) and / or the hand-pointing direction (H), and based on a known relative arrangement of the scene acquiring device with respect to the first large-size display, a gaze location of interest (g) in the first large-size display (110) and / or a hand-pointing location of interest (h) in the first large-size display (110), each location of interest (g, h) defining a pixel or a group of pixels (p) of the first large-size display (110), the pixel or group of pixels being identified by their coordinates (u(p), v(p)) in the first coordinate system (u, v); andan awareness module (150) configured to, continuously:allocate to each identified skeleton a gaze awareness cue (AC130) and / or a hand-pointing awareness cue (AC120) or retrieve previously allocated awareness cue(s), the awareness cue(s) allocated to a skeleton being visually distinct from the awareness cue(s) allocated to another skeleton;apply a transfer function to the pixel or group of pixels (p) of each location of interest (g, h) to determine their respective coordinates (u’(p’), v’(p’)) in the second coordinate system (u’, v’);constitute or update, for each identified skeleton, a gaze tuple and / or a hand-pointing tuple, each gaze tuple comprising a gaze awareness cue of an identified skeleton and the coordinates of the gaze location of interest of the same identified skeleton in the second coordinate system, and each hand-pointing tuple comprising a hand-pointing awareness cue of an identified skeleton and the coordinates of the hand-pointing location of interest of the same identified skeleton in the second coordinate system;send, for each identified skeleton (S), the gaze tuple and / or hand-pointing tuple to the second display (210) via the data link (300), the second display (210) being configured to display each received awareness cue (AC120, AC130) of the identified skeleton at the respective coordinates (g’, h’) in the second coordinate system (u’, v’).System according to claim 1, wherein the scene acquiring device (142) comprises an image capturing device configured to capture image frames of the scene and the tracking module is configured to, for each detected skeleton:identify head zones (HZ1, HZ2) of the image frames which contain a head of a skeleton;crop image frames down to the head zone; andrun a recognition algorithm on the head zone to identify a user,wherein the awareness module selects a gaze / hand-pointing awareness cue as a function of the identified user.System according to claim 1 or 2, wherein the scene acquiring device comprises at least two image capture devices and is configured to merge skeleton data captured by the at least two image capture devices.System according to any of the preceding claims, wherein the tracking module further comprises a gesture recognition module (146) configured to:crop arm portions of the captured scene, each of the arm portions containing one or both arms of an identified skeleton; andrun a recognition algorithm on the arm portions to identify a gesture among prerecorded gestures, the system being further configured to display information related to the recognized gesture on the second large-size display.System according to any of the preceding claims, wherein the awareness cues have a respective size that is determined as a function of the location of the corresponding user.System according to any of the preceding claims, wherein the users interact with the first large-size display exclusively and directly through their movements, the system being void of hand-held token, head-worn, or body-worn tokens, mouse or keyboard.System according to any of the preceding claims, wherein the first and / or the second display comprises a touchscreen.System according to any of the preceding claims, wherein the second display is a large-size display.System according to any of the preceding claims, wherein the first large-size display is made of at least one display arranged in a vertical plane.Computer-implemented method (1000) performed by the system according to any of the preceding claims, the method comprising:identifying (1100) skeletons and their location(s) of interest on the first large-size display;allocating (1200) distinct awareness cues to distinct identified skeleton; anddisplaying (1300) the awareness cues on the second display, at a location of the second display mapping the location of interest on the first large-size display.Method according to claim 10, wherein the steps of identifying, allocating and displaying are performed continuously.

Images