Assignment of roles and garments to co-located users

Abstract

A multi-player co-located AR experience is enhanced by assigning roles and garments to respective participants (i.e., "users" of AR-enabled mobile devices) in a multi-player AR session for telling stories, performances, and similar activities. Body tracking techniques and augmented reality (AR) software are used to dress up the body of co-located participants with virtual apparel in the context of a multi-player co-located AR experience. The tracked bodies are distinguished to determine which body belongs to which user and thus which virtual garment belongs to which tracked body, so that the corresponding garment may be assigned for display in augmented reality. A host-guest mechanism is used for networked assignment of roles and corresponding garments in co-located multi-player AR sessions. Body tracking techniques are used to move a garment with the body upon detection of movement of the assigned body.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 17 / 901,518, filed on September 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to augmented reality (AR) applications for mobile devices. More specifically, but not by way of limitation, the present disclosure describes techniques for using body tracking and AR technology to assign roles to co-located participants in an AR session and to decorate the bodies of co-located participants with virtual clothing. Background Art

[0004] Body tracking technology has been adapted to track the body of a user of a mobile device and recognize the user's gestures in order to provide gesture input to a software application running on the mobile device. The camera of the mobile device may be adapted to detect gestures in an image and process the gestures to recognize events that will be triggered in the software application in response to the gestures. For example, the camera may discern that a person in the image is extending a finger, and then select a corresponding option from a menu of the software application. For gesture detection, the biometric component of the mobile device may include a component that detects expressions (e.g., hand expressions, facial expressions, body gestures, eye tracking, and the like) based on images captured by the camera of the mobile device. However, conventional body tracking technology does not provide for tracking different bodies within an image, whereby the tracked body can be identified in the context of a software application running on the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The features of the various embodiments disclosed will be readily understood from the following detailed description, in which reference is made to the accompanying drawings. Reference numerals are used with each element in the specification and throughout the multiple views of the drawings. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements.

[0006] The various elements shown in the drawings are not drawn to scale unless otherwise specified. The sizes of the various elements may be enlarged or reduced for clarity. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0007] Figure 1 is a diagram illustrating multiple participants of a co-located augmented reality (AR) session in a sample configuration;

[0008] Figure 2 is a block diagram of electronic components of a mobile device suitable for implementing a co-located AR session in a sample configuration;

[0009] Figure 3 is used in the sample configuration Figure 2 A block diagram of software components for implementing a co-located AR session on a mobile device of FIG.

[0010] Figure 4 is a flow chart of a method in a sample configuration for detecting a full body gesture, determining which participant performed the full body gesture, and propagating the detected gesture to other co-located participants of a multi-party AR session;

[0011] Figure 5 is a flow chart of a method in a sample configuration for assigning roles to co-located participants in a multi-player software application; and

[0012] Figure 6 is a flow chart of a method in a sample configuration for utilizing AR costumes to dress up co-located participants in a multi-player software application. DETAILED DESCRIPTION

[0013] AR application for assigning characters and costumes in multi-player co-located AR experiences Characters and costumes are assigned to corresponding participants (i.e., "users" of AR-enabled mobile devices) in a multi-player AR session for storytelling, performances, and similar activities. Body tracking technology and augmented reality (AR) software are used to dress up the bodies of co-located participants with virtual costumes in the context of a multi-player co-located AR experience.

[0014] However, there is currently no way to distinguish which tracked body belongs to which user, and thus which virtual clothing belongs to which tracked body. Therefore, in the context of multi-player software applications such as multi-player AR applications, there is no way to distinguish between multiple bodies of co-located participants in a multi-player AR session so that AR overlays such as personalized clothing can be overlaid on the various participants in the multi-player AR session. This limitation is addressed by providing a host-guest mechanism for networked assignment of roles in a co-located multi-player AR session and by further providing a method for associating detected bodies with their corresponding user's clothing in augmented reality. Body tracking technology is used to cause the clothing to move with the body when movement of the assigned body is detected. The resulting method combines body tracking technology with a multi-player co-located AR experience to provide an AR environment in which users can be assigned roles and corresponding clothing.

[0015] As an example of a multi-player co-located AR experience, assume that Alice and Bob have mobile devices suitable for implementing the methods described herein. Alice and Bob are example guests in a multi-player co-located AR session in multiple scenes.

[0016] To set up a multi-player co-located AR experience, roles are first assigned to Alice and Bob. To do this, the host of the AR session points the AR camera of her mobile device toward Alice, who will be assigned the role "Alpha." The host asks Alice to wave or perform some other predetermined gesture to indicate that she is ready for role assignment. The host's mobile device detects Alice's gesture and triggers (e.g., by tapping the screen) the assignment of the role "Alpha" to Alice. The host's mobile device then broadcasts the role assignment event along with Alice's user ID to the mobile devices of all participants in the multi-player co-located AR session. Each mobile device in the multi-player co-located AR session is now able to map Alice's user ID to the character "Alpha." Each participant's mobile device (except Alice's) can now perform a costume dress-up (described below), where Alpha's costume is dressed on Alice's body in augmented reality. The same process can be followed for each other co-located participant in the multi-player co-located AR session.

[0017] Once the role has been assigned, the participants in the next multi-player co-location AR session can be assigned an AR costume corresponding to the assigned role. To do this, each participant's mobile device traverses the list of participants (except this participant) in the multi-player co-location AR session. For each participant (assuming Alice, who is assigned the role Alpha), the screen space position of their mobile device is obtained from the perspective of the current participant. The screen space head position of all detected bodies in the scene is also obtained. The body of the mobile device closest to Alice in terms of screen space distance is selected and identified as Alice's body. The body mesh of Alice's character (Alpha) is retrieved. The parent of the body mesh is set to Alice's tracked body, and the body index of the body mesh is changed to match the object index of Alice's tracked body. The same process can be followed for each other co-location participant in the multi-player co-location AR session. Once all roles and costumes have been assigned, the multi-player co-location AR session can continue, whereby each participant can see each other co-location participant with his or her AR costume overlaid on the participant's body.

[0018] The system described herein therefore provides a method for assigning a role to at least one participant in a multi-party augmented reality (AR) session including at least two co-located session participants. The multi-party session enables all session participants to communicate with each other using their respective mobile devices. In a sample configuration, the method includes receiving an indication from an AR session that a role is to be assigned to at least one participant in the multi-party AR session, receiving a role assignment trigger event for the assignment of the role, obtaining a user ID of the participant in the multi-party AR session who provided the role assignment trigger event, and assigning the role to the participant in the multi-party AR session who provided the role assignment trigger event. The role assignment for the role of the participant in the multi-party AR session who provided the role assignment trigger event can be broadcast to at least two co-located session participants. Each mobile device for each co-located participant can then map the user ID to the assigned role.

[0019] After receiving an indication from the AR session that a role will be assigned to at least one participant in the multi-party AR session, a message can be sent to at least one participant in the multi-party AR session that was not previously assigned a role, stating that the role will be assigned to the participant if the at least one participant provides a predetermined gesture (such as a waving gesture).

[0020] In a sample configuration, obtaining the user ID of the co-located session participant that provided the role assignment trigger event may include capturing the screen space position of the head of the co-located session participant that provided the role assignment trigger event, comparing the captured screen space position of the head of the co-located session participant that provided the role assignment trigger event with the screen space position of each co-located session participant, and determining the user ID of the co-located session participant that is closest to the screen space position of the head of the co-located session participant that provided the role assignment trigger event.

[0021] The method may also include assigning a virtual costume to a participant in the multi-party AR session who provided a role assignment trigger event by the following operations: obtaining a screen space position of a mobile device of the participant in the multi-party AR session who provided a role assignment trigger event, and determining whether a body that is not assigned a virtual costume is detected in the camera's view. For each body in the camera's view that is not assigned a virtual costume, the screen space position of the body's head may be obtained. If the screen space position of the body's head is closest to the screen space position of the mobile device of the participant in the multi-party AR session who provided a role assignment trigger event, the body with the screen space position closest to the mobile device of the participant in the multi-party AR session who provided a role assignment trigger event is assigned as the body of the participant in the multi-party AR session who provided a role assignment trigger event.

[0022] In a sample configuration, the body of a participant in a multi-party AR session who provided a role assignment trigger event may be tracked. A body mesh corresponding to a virtual costume of a character may be retrieved and set to the tracked body of the participant in the multi-party AR session who provided a role assignment trigger event. The role assignment for the virtual costume of the participant in the multi-party AR session who provided a role assignment trigger event may be broadcast to at least two co-located session participants.

[0023] In a sample configuration, whether the screen space position of the head of a body is closest to the screen space position of the mobile device of a participant who provided a role assignment trigger event in a multi-party AR session can be determined by: (a) obtaining session data related to a first participant from a list of co-located participants in the multi-party AR session, (b) determining the screen space position of the first participant, (c) comparing the screen space position of the head of the participant who provided the role assignment trigger event in the multi-party AR session with the screen space position of the first participant, and (d) repeating steps (a) to (c) for each co-located participant in the multi-party AR session. Determining whether the screen space position of the head of the body is closest to the screen space position of the mobile device of the participant who provided the role assignment trigger event in the multi-party AR session can also include determining the user ID of the closest co-located participant.

[0024] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products, which illustrate the examples set forth in the present disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without such details. The aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in a variety of ways. The terms and nomenclature used herein are only used for the purpose of describing specific aspects and are not intended to be restrictive. In general, known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0025] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links or the like, by which an electrical or magnetic signal generated or supplied by one system element is communicated to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communications media, one or more of which may modify, manipulate, or carry electrical signals. The term "on" means supported by an element directly or indirectly supported by an element through another element integrated into or supported by the element.

[0026] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art after examination of the following and the accompanying drawings, or may be learned by production and operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, means and combinations particularly pointed out in the appended claims.

[0027] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below. Figures 1 to 6 Example systems and methods are described for assigning characters and costumes to co-located participants in a multi-party AR session.

[0028] As described in related U.S. patent application Ser. No. 4,876,979 (titled “Co-Located Full-Body Gestures,” filed on the registration date of the present application, which is incorporated herein by reference in its entirety; attorney no. ANON-448us1), in a co-located augmented reality (AR) session, full-body gestures (waving, jumping, kicking, etc.) can be a useful method of interaction. In an example scenario, Alice and Bob are both in a multi-player co-located AR session and point their mobile devices toward each other. Existing body tracking technology is able to detect body gestures, but Alice's mobile device cannot detect her own body gestures (because when Alice holds the mobile device in front of her with her hands, her mobile device does not have a full camera view of her body). In this case, only Bob's camera of Bob's mobile device can detect Alice's body gesture. However, there is no mechanism on Bob's mobile device to recognize that the gesture is Alice's gesture and indicate to Alice's mobile device that she has performed a full-body gesture so that the software on Alice's mobile device can respond accordingly. Therefore, a mechanism is needed for enabling a body gesture detected on an auxiliary mobile device (in this scenario, Bob's mobile device) to be associated back to the mobile device of the user who triggered the gesture (in this scenario, Alice's mobile device). Accordingly, this document describes a process for detecting a full-body gesture corresponding to a gesture event, determining which user triggered the gesture event, and propagating the gesture event among co-located participants. As will be further explained below, this gesture event can be used to identify participants for the purpose of assigning characters and costumes to each participant in a multi-party co-located AR session.

[0029] In a sample configuration, waving may be provided as an example of a full-body gesture. In this sample configuration, the host mobile device detects a tracked body of a co-located participant in a multi-party session. The tracked body of the participant begins waving, and the host's mobile device identifies the presence of a tracked body waving. The host mobile device traverses the list of co-located participants in the multi-party session and finds the participant mobile device that is closest in screen space position relative to the head of the waving body. The host mobile device then obtains the user ID of the closest participant mobile device. The host mobile device can then broadcast the identified full-body gesture event (session participant waving) along with the obtained user ID to all co-located participants in the multi-party session. Each participant's mobile device can then process the gesture event according to the circumstances of the multi-party session. For example, as follows regarding Figure 5 and Figure 6 As described, each participant's mobile device may assign a role to the identified participant, and then assign an AR garment that is overlaid on the participant in the screen view of each other co-located participant's mobile device.

[0030] Figure 1 is a diagram illustrating multiple participants 100, 100', and 100" in a multi-party co-located augmented reality (AR) session in a sample configuration. Although three participants are illustrated, the multi-party co-located AR session can include from two to as many participants as the AR session software can support (e.g., 64 participants). Each participant 100, 100', and 100" has a mobile device 200, 200', and 200" respectively, which is used by the participant 100, 100', and 100" to participate in the co-located AR session. As shown, each mobile device 200, 200', and 200" uses, for example Low energy consumption ( Low Energy, BLE) or any other appropriate local area communication technology to communicate with each other co-located mobile device 200, 200' and 200". During a multi-party co-located AR session, each participant 100, 100' and 100" can share one or more AR applications with each other participant 100, 100' and 100". In addition, each mobile device 200, 200' and 200" can be wirelessly connected to the Internet 110 using a suitable broadband cellular network technology (e.g., 5G) and is enabled to communicate with a remote AR application server 120, which can provide AR applications for the multi-party co-located AR session. For example, the AR application server 120 may provide an AR application to be shared among the mobile devices 200, 200', and 200" during a multi-party co-located AR session, such as a Lens of the type available from Snap Inc. of Santa Monica, California. The following description will utilize Lens as an example of an AR application that may be used in a multi-party co-located session as described herein; however, those skilled in the art will appreciate that other AR applications or other multi-party applications may similarly be shared in a multi-party AR session as described herein.

[0031] The AR application server 120 may include one or more computing devices as part of a service or network computing system, the AR application server 120 including a processor, a memory, and a network communication interface to communicate with the mobile devices 200, 200' and 200" via the Internet 110. In a sample configuration, one or more of the mobile devices 200, 200' and 200" may also be paired with an electronic glasses device (not shown) via a high-speed wireless connection. The AR application server 120 may store a gallery of snapshots and AR objects, which is maintained by the AR application server 120 for access by each participant 100, 100' and 100" and is called via Internet communications that provide links to the snapshots and AR objects stored in the gallery.

[0032] In a sample configuration, a multi-party AR session may be created for co-located participants 100, 100', and 100" by invoking a connection template of the type available, for example, from SnapChat, Inc. of Santa Monica, California. The connection template allows users of a social media network (such as SNAPCHAT, available from SnapChat, Inc. of Santa Monica, California) to join with social media friends to share a common AR application in a common AR space, either in the same place or remotely. The connection template may include a reusable invitation flow that manages how a user joins an AR session with their social media friends, and a relay system that enables participants 100, 100', and 100" in the AR session to pass messages among themselves. An avatar system may also be provided to provide co-presence in a shared AR session when one or more of the participants 100, 100', or 100" are not co-located with the other participants 100, 100', and 100". In the sample configuration, the connection template may also provide examples of synchronizing object instantiation and transformation manipulation within a shared AR session.

[0033] In an example, a shared multi-party AR session may implement a shared AR application, such as Connected Lens available from Snap Inc. of Santa Monica, California. Connected Lens is a type of Lens that allows participants in a shared session to transfer and store data between mobile devices 200, 200', and 200" running the same Lens, whether the mobile devices 200, 200', and 200" are co-located or in different locations. Connected Lens thus allows participants 100, 100', and 100" to create AR experiences that social media friends can experience together at the same time and replay over time. The sample AR experience may permit participants 100, 100', and 100" to instantiate objects, modify the position / rotation / scale of objects that are synchronized across mobile devices 200, 200', and 200", and send messages among participants 100, 100', and 100". When participants 100, 100' and 100" are co-located, participants 100, 100' and 100" can reference the same co-located AR experience on their respective mobile devices 200, 200' and 200". To invite social media friends to such a co-located AR experience, participant 100, 100' or 100" can first map the area, and then in the case of Connected Lens, Lens can display a code (e.g., a Snapcode) that other participants 100, 100' and 100" can scan to join the co-located AR experience. The mapped area does not provide the geometry of the scene but allows participants 100, 100' and 100" to share the same coordinate system with other participants 100, 100' and 100". A single AR application such as Lens can include a single user experience, a remote experience, or a co-located experience, as desired by the participants 100, 100' and 100".

[0034] In the example of Connected Lens, Lens can create a multi-party AR session that can be joined by others using the Connected Lens module. Adding the Connected Lens module transforms Lens into a Connected Lens with a Call to Action button that participant 100, 100', or 100" can click to enter the ConnectedLens State, at which point participant 100, 100', or 100" receives the ability to invite other participants 100, 100', and 100" to enter the connected AR session. When participant 100, 100', or 100" enters Lens via invitation, participant 100, 100', or 100" will enter the Connected Lens State directly. At this point, each connected participant 100, 100', and 100" will not have access to any camera mode, except for changing between the front and rear cameras. In addition, instead of a carousel for selected Lenses, participants 100, 100', and 100" will have an Action bar that allows participants 100, 100', and 100" to see who else is in the connected AR session. Within the connected AR session, each participant 100, 100', and 100" can send messages to and receive messages from each other participant in the connected AR session. To facilitate communication, each participant in the connected AR session is assigned a user ID for the connected AR session.

[0035] When participant 100, 100', or 100" opens Connected Lens for the first time, she is first introduced to a splash screen that introduces participant 100, 100', or 100" to the current Connected Lens experience and provides her with the option to begin interacting with the connected AR session by selecting a Call to Action button to initiate Connected Lens. Once participant 100, 100', or 100" completes the invitation process, she is brought into the connected AR session in which Lens can send data back and forth between each participant 100, 100', and 100".

[0036] It will be appreciated by those skilled in the art that by using the Connected Lenses Module to set up a multi-player session to make Lenses available for multi-party applications, the co-location session can also be used on an electronic eyewear device (such as the SPECTACLES available from Snap Inc. of Santa Monica, California). TM ) on the .

[0037] Figure 2is a block diagram of electronic components of a mobile device 200, 200', or 200" in a sample configuration suitable for implementing a co-located AR session. In the sample configuration, the mobile device 200 can present an interface (e.g., an AR interface) to the participant 100 and can also be used to extend the interface to an associated electronic eyewear device (not shown). The mobile device 200 may include a flash memory 205 that stores programming to be executed by the CPU 210 to perform all or a subset of the functions described herein. The mobile device 200 may further include a camera 225, which includes one or more visible light cameras (visible light cameras with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 205 may further include multiple images or videos generated via the camera 225.

[0038] The mobile device 200 may further include an image display 230, a mobile display driver 235 for controlling the image display 230, and a display controller 240. Figure 2 In the example of , image display 230 may include a user input layer 245 (e.g., a touch screen) layered on top of or otherwise integrated into the screen used by image display 230. Examples of touch screen type mobile devices that may be used include, but are not limited to, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, Figure 2 Thus provided is a block diagram illustration of an example mobile device 200 having a user interface including a touch screen input layer 245 for receiving input (by touch, multi-touch or gestures, etc., by hand, stylus or other implement), and an image display 230 for displaying content.

[0039] like Figure 2 As shown, the mobile device 200 includes at least one digital transceiver (XCVR) 250, shown as a WWAN (Wireless Wide Area Network) XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 200 may also include additional digital or analog transceivers, such as a short-range transceiver (XCVR) 255 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, or For example, the short-range XCVR 255 may employ one or more standard protocols for communicating with wireless local area networks (such as IEEE 802.11). The invention may be in the form of any available two-way wireless local area network (WLAN) transceiver of a type compliant with one of the standards).

[0040] In order to generate location coordinates for locating the mobile device 200, the mobile device 200 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 200 may utilize either or both of the short-range XCVR 255 and the WWAN XCVR 250 to generate location coordinates for positioning. For example, a cellular network, or based on The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the mobile device 200 via the XCVR 250, 255 through one or more network connections.

[0041] The transceivers 250, 255 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 250 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". The transceivers may also be incorporated into broadband cellular network technologies known as "5G". In a sample configuration, the transceivers 250, 255 provide two-way wireless communication of information including digitized audio signals, still images and video signals, web page information for display and input associated with the web page, and various types of mobile message communications to / from the mobile device 200.

[0042] The mobile device 200 may further include a microprocessor that functions as a central processing unit (CPU) 210. The CPU 210 is a processor having elements that are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, the example utilizes components that form a programmable CPU. For example, the microprocessor may include one or more integrated circuit (IC) chips that integrate electronic components to perform the functions of the CPU 210. The CPU 210 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form a CPU 210 or processor hardware in a smart phone, a laptop computer, and a tablet computer.

[0043] The CPU 210 functions as a programmable host controller for the mobile device 200 by configuring the mobile device 200 to perform various operations according to instructions or programming executable by the CPU 210. For example, such operations may include various general operations of the mobile device 200, as well as operations related to programming for a messaging application and an AR camera application on the mobile device 200. Although a processor may be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing programming.

[0044] The mobile device 200 also includes a memory or storage system for storing programming and data. Figure 2 In the example shown in , the memory system may include flash memory 205, random-access memory (RAM) 260, and other memory components 265 as needed. RAM 260 may be used as short-term storage for instructions and data being processed by CPU 210, for example, as working data processing memory. Flash memory 205 typically provides longer-term storage.

[0045] In the example of mobile device 200, flash memory 205 may be used to store programming or instructions for execution by CPU 210. Depending on the type of device, mobile device 200 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPAD devices), Windows Mobile, Amazon Fire OS (Operating System, operating system), RIM BlackBerry OS, or the like.

[0046] Mobile device 200 may also include an audio transceiver 270 that may receive audio signals from the environment via a microphone (not shown) and provide audio output via a speaker (not shown). A messaging application or social media application implemented on mobile device 200 may couple audio signals with video signals and other messages.

[0047] In a sample configuration, the mobile device 200 may further include a biometric component 275 to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measure bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), and similar operations. The motion component may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), and the like. The position component may include a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a position sensor component for generating positioning system coordinates, and a position sensor component for generating a positioning system coordinate. or transceivers, altitude sensor components (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), orientation sensor components (e.g., a magnetometer), and the like. Such positioning system coordinates may also be received from other devices via wireless connections.

[0048] In a sample configuration, the mobile device 200 can execute mobile augmented reality (AR) application software 220 loaded into the flash memory 205, such as the one available from Snap Inc. of Santa Monica, California. As will be discussed below Figure 3 to Figure 4 As described in more detail, the mobile AR application software 220 can be executed to create a multi-party session, detect full body gestures of participants, determine which session participant triggered a gesture event, and propagate the gesture event among the co-located participants in the multi-party session. In a sample configuration, as described below with respect to Figure 5 to Figure 6 As described, gesture events may be used to assign a role to an identified participant and a corresponding AR garment to that participant, whereby other co-located participants in a multi-party session may see the corresponding AR garment overlaid on each other co-located participant in her camera view.

[0049] Figure 3 is used in the sample configuration Figure 2 2 is a block diagram of software components for implementing an AR session on a mobile device 200. As shown, the mobile device 200 executes an AR application 220 to implement an AR session 310 rendered by an AR renderer 320 to a display 230 of the mobile device 200. The AR session 310 can be implemented as a connected AR application session 330, such as a Connected Lens session as described above. The connected AR application session 330 controls the AR application session 310 with respect to the mobile device 200. Figure 1The connected AR application session 330 may communicate with other co-located AR devices (including mobile devices 200' and 200") in the co-located session. The Connected Lens in the connected AR application session 330 may function to overlay one or more shared Lenses on images received from the cameras 225 of the respective mobile devices 200, 200' and 200" of each participant 100, 100' and 100" in the connected AR application session 330.

[0050] In a sample configuration, the AR application 220 further implements full body tracking software 340 and gesture / participant recognition software 350 during an AR session. The full body tracking software 340 may implement a three-dimensional (3D) body template that allows the participant 100, 100', or 100" to track the position and rotation of the joints of the body in 3D in the image captured by the camera 225. One or more bodies and parts of the body may be tracked simultaneously. In a sample configuration, the 3D model may be assembled to a skeleton that matches what the full body tracking software 340 sees in the image from the camera 225. This allows an AR application (e.g., Lens) to map the body's movements to the model's movements.

[0051] The 3D rigged model may be based on a stick figure with joints corresponding to the person's feet, knees, hips, shoulders, spine, neck, head, elbows, hands, and the like. The 3D rigged model assigns all of the joints of the person in the image from the camera 225 to the 3D rigged model for use by the 3D body tracking software 340. The 3D body tracking software 340 then tracks the movement of the person in the image from the camera by causing the 3D rigged model to move with the person as the person moves. The scale of the 3D rigged model may be adjusted as appropriate to fit the person in the image from the camera 225.

[0052] When the camera 225 captures overlapping images of more than one person, the location of the invisible parts of the body may not be known. In this case, the 3D body tracking software 340 can disable or hide the joints that are not automatically tracked. Independent objects, such as AR objects in the form of Lenses, can also be attached to the tracked body by attaching the AR objects to the independent joints of the people in the image.

[0053] In order to recognize gestures performed by people in images, a full-body trigger template can be implemented, which allows different screen effects to be triggered by different body movements, such as custom body gestures (e.g., waving), touch screen triggers, and the distance between body joints, or the distance between body joints and screen elements identified in screen transformations. The full-body trigger template can implement a full-body tracking system for tracking full-body gestures, an upper-body tracking system for tracking upper-body gestures, or a lower-body tracking system for tracking lower-body gestures, depending on the situation. The full-body tracking system can also track facial features by providing access to facial points such as the nose, left and right eyes, and left and right ears. The trigger template further specifies what should happen in the corresponding AR application (e.g., Connected Lens) when a trigger event is identified. The triggered event can be a full-screen effect, a visual effect attached to the body, a switching of an object, the playing of an animation, or any other content that a trigger event may trigger within an AR application.

[0054] In a sample configuration, the whole body tracking software 340 provides 2D screen points representing the joints or body parts of the people identified in the image and being the basis of the whole body tracking template. Sample body movements can include T-poses, right foot clicks, waving and similar actions. Each body movement can have a motion trigger script attached thereto, which allows different types of motion to be attached, including gestures. Each gesture can be composed of a list of gesture triggers (e.g., for right foot click triggers, right leg extends or right leg clicks). When a gesture matches the people in the camera image, a list of behavior triggers is called and matched with the gesture. When a gesture matches a behavior trigger, a corresponding gesture event is activated. The gesture can be a motion, such as waving, which triggers a motion trigger response script to implement a gesture event.

[0055] For example, specific body movements of a person in an image can trigger role assignment and clothing assignment. To this end, the AR session can initiate a process of assigning roles and clothing to participants in the current AR session. A first role can be assigned to one or more participants who provide a predetermined gesture (e.g., waving) for assignment of a specific role. The full body tracking software 340 of the host's mobile device tracks the co-located participants in the camera view and the gesture / participant recognition software 350 determines that one or more co-located participants have provided the predetermined gesture. The gesture event and the user ID of the participant performing the predetermined gesture are provided to the AR session 310 for the assignment of roles. The user ID is then provided to the role / clothing assignment software 360 ​​for the assignment of roles and corresponding clothing. As described below with respect to Figure 5 to Figure 6 As described above, this process can be repeated until all characters and costumes have been assigned.

[0056] Figure 4The operation of gesture / participant identification software 350 for identifying a full body gesture and the participant performing the gesture is shown in FIG. Figure 4 Flowchart of a method for detecting a full-body gesture in a sample configuration, determining which participant performed the full-body gesture, and propagating the detected gesture to other co-located participants of a co-located AR session.

[0057] In the sample configuration, Figure 4 The method is Figure 3 The gesture / participant recognition software 350 shown in FIG. Figure 4 As shown, the method starts at 400 by determining at 402 whether there is a tracked body detected by the full-body tracking software 340 in the image received from the camera 225 of the mobile device 200. If the tracked body is not detected at 402, the process proceeds to 404 to wait for the tracked body to be detected. Once the tracked body is detected at 402, a determination is made at 406 as to whether the tracked body is performing a full-body gesture. As noted above, a full-body gesture such as waving, jumping, or kicking can be detected from the movement of the body model paired with the received image. If the full-body gesture is not detected at 406, the process proceeds to 408 to wait for the full-body gesture to be detected. Once the full-body gesture is detected at 406, the screen space position of the head of the body making the gesture is captured at 410.

[0058] Once the screen space position of the head of the gesturing body has been captured at 410, the captured screen space position of the head of the gesturing body is compared to the screen space position of each co-located participant in the multi-party AR session. In particular, the first available participant is obtained from the list of co-located participants in the multi-party AR session at 412. At 414, the screen space position of the participant's mobile device 200, 200', or 200" is obtained from the session data and compared to the screen space position of the head of the gesturing body. If the screen space position of the current participant's mobile device 200, 200', or 200" is determined at 416 to be closer than the screen space position of any other participant in the AR session, then at 418 the variable "ClosestParticipant" is set to the user ID of the current participant. If it is determined at 420 that there are other participants in the AR session whose proximity to the screen space position of the head of the gesturing body has not yet been evaluated, then data for the next available participant is obtained at 422, and steps 414 to 422 are repeated until all participants in the AR session have had their proximity to the screen space position of the head of the gesturing body evaluated.

[0059] Once all participants in the AR session have been evaluated, the user ID of the participant closest to the head of the gesturing body is obtained at 424. The captured gesture event (e.g., waving, jumping, or kicking) is then broadcast to other participants in the co-located AR session at 426 along with the user ID of the participant that generated the gesture event. At 428, each AR device 200, 200', and 200" for each participant 100, 100', and 100" in the AR session processes the gesture event based on the nature of the given gesture event and the user ID of the person who generated the gesture event. The process then ends at 430.

[0060] Once the gesture event and the user ID of the person who generated the gesture event are so determined, AR session 310 can invoke character / costume assignment software 360 ​​to assign characters and costumes to the corresponding co-located participants of the multi-party AR session. Figure 5 It is provided by the character / costume distribution software 360 ​​( Figure 3 ) is a flowchart of a method implemented by FIG. 1 for assigning roles to co-located participants in a multi-party AR session in a sample configuration.

[0061] like Figure 5 As shown, the role / costume assignment process begins at 500 by determining at 510 that the AR conversation software 310 intends to assign a role (e.g., the role "Alpha"). For example, the role / costume assignment software 360 ​​may receive an ID for the role "Alpha". The AR conversation software 310 may also send a message to each co-located participant in the multi-party conversation, asking each participant to perform a predetermined gesture (e.g., waving) in order to be assigned a designated role. The conversation software 310 may then wait for an indication from the gesture / participant recognition software 350 that the co-located participant has provided a predetermined gesture for role assignment. Upon receiving the user ID of each co-located participant who has provided the predetermined gesture, the role / costume assignment software 360 ​​is notified that a role assignment trigger event has been received. When the role assignment trigger event has been received at 520, the user ID of the participant who provided the predetermined gesture and thus provided the assignment trigger event is obtained at 540. Otherwise, the role / costume assignment software 360 ​​waits for the assignment trigger event at 530.

[0062] Once the user ID of the participant who provided the assignment trigger event is obtained at 540, the role assignment trigger event is broadcast to all co-located participants in the multi-party AR session at 550. For example, if Alice raises her hand and is assigned the role "Alpha", Alice's user ID for the role "Alpha" is obtained at 540, and Alice's user ID is broadcast to all participants in the multi-party AR session at 550 by providing Alice's user ID associated with the ID of the role "Alpha", indicating that Alice has been assigned the role Alpha. At 560, each mobile device of each co-located participant in the multi-party AR session stores a mapping of Alice's user ID to the ID for the role "Alpha". Once Alice has been associated with the role "Alpha", at 570, when Alice is recognized in the camera view of the AR camera of any other co-located participant in the multi-party AR session, Alice's body is covered with the corresponding role costume for the role "Alpha" (described below with respect to role costume). Figure 6 ). As long as Alice's body remains in the camera view of the AR cameras of any other co-located participants in the multi-party session, the character costume is updated for each frame. Figure 5 The process can be repeated for each participant in the multi-party AR session to be assigned a role. The process ends at 580.

[0063] It will be understood by those skilled in the art that Figure 5 The described character / costume assignment process associates characters and costumes with the mobile devices of participants in a multi-party session and assumes the user ID of the participant associated with the mobile device. Figure 4 As is apparent from the gesture / participant recognition process of the presenter, the gesture for the receiving character is associated with the body closest from the perspective of the host's mobile device, and thus the character and costume are associated with the user ID of the mobile device closest to the body of the gesturing participant from the perspective of the host's mobile device. Any body gesture event (such as waving) detected by the gesture / participant recognition software 350 can therefore be associated with the mobile device of the gesturing participant, and the host can broadcast the user ID of the gesturing participant's mobile device to all other mobile devices of other co-located participants of the multi-party AR session.

[0064] Figure 6 is a flow chart of a method for utilizing AR clothing to dress up co-located participants in a multi-player software application in a sample configuration. Figure 6As shown, the method begins at 600 by executing on the host's mobile device a process including determining if there are any participants in the co-located AR session at 602. If not, the process proceeds to 604 to wait for a participant. Once a participant is present in the co-located AR session, a first participant of the AR session is obtained at 606. If it is assumed that the first participant is Alice, a screen space position of Alice's mobile device is obtained at 608.

[0065] At 610, the host mobile device determines whether a body is detected in the camera view. If not, the process proceeds to 612 to wait for a body to enter the camera view. Once a body is detected at 610, a determination is made at 614 as to whether a body that has not been assigned a garment has been detected. If all bodies have been assigned a garment, the process ends at 616. Otherwise, the first available body is obtained at 618, and at 620, the body is assigned a garment using the method described above. Figure 4 The described technique obtains the screen space position of the body's head.

[0066] At 622, the screen space position of the head of the body currently being examined is compared to the screen space positions of the heads of other bodies that have been examined so far to determine whether the head of the body currently being examined is closest to the screen space position of Alice's mobile device. If at 622 it is determined that the screen space position of the current participant's mobile device 200, 200' or 200" is closer than the screen space position of any other participant in the AR session evaluated so far, then at 624 the variable "closest body" is set to the user ID of the body currently being examined. If at 626 it is determined that there are other participants in the AR session who have not yet been examined for proximity to the screen space position of the head of the body currently being examined, then at 628 data for the next available participant is obtained, and steps 620 to 626 are repeated until all participants in the AR session have been evaluated for proximity to the screen space position of the head of the body currently being examined.

[0067] Once all participants in the AR session have been evaluated, the user ID of the participant closest to the head of Alice's body is obtained at 630 (i.e., the variable "closest body" is assigned the user ID of the closest body in the AR session that has been examined). In other words, Alice's body has been identified. The body mesh of Alice's character's clothing is then retrieved at 632, and the parent of the retrieved body mesh is set to Alice's tracked body at 634. The body index (Body Index) of the body mesh is changed to match the object index (ObjectIndex) of Alice's tracked body. The process then determines at 636 whether there are any other participants in the AR session who have not yet assigned their clothing to the tracked body. If there are other participants who have not yet assigned their clothing to the tracked body, the next available participant of the AR session is obtained at 638, and steps 620 to 636 are repeated. Once all participants have assigned their clothing to the tracked body, the process ends at 616. The character and clothing assignments can be broadcast to all co-located participants in the AR session along with the user ID of the mobile device to which the character and clothing are assigned.

[0068] Each mobile device in a multi-player co-located AR session is now able to map each user ID to an assigned character. Each participant's mobile device can now perform costume dressing by overlaying the assigned AR clothing over the body of the corresponding co-located user in the camera view during the course of the AR session. The same process can be followed for each other co-located participant in the multi-player co-located AR session. Therefore, in the disclosed example, participants in a multi-party co-located AR session can be assigned characters and clothing during the AR session for participating in character-driven stories, performances, and similar activities.

[0069] By way of example, an AR session can assign roles in a story to participants in the AR session. One of the participants in the AR session can choose to be the host to set roles for the story. To this end, the host's mobile device broadcasts a message to the mobile devices of other co-located participants in the AR session (e.g., Alice and Bob), similar to: "If you want to play the role of Alpha, please wave." Alice starts waving. The host's mobile device identifies the waving gesture, determines that Alice's mobile device is closest to the body making the waving gesture, and assigns the role of Alpha to Alice. The role assignment for Alpha can be broadcast to the mobile devices of other co-located participants in the AR session. The role assignment process can be repeated for each role in the story. For example, the host's mobile device can broadcast a message to the mobile devices of other co-located participants in the AR session, similar to: "If you want to play the role of Beta, please wave." Bob starts waving. The host's mobile device identifies the waving gesture, determines that Bob's mobile device is closest to the body making the waving gesture, and assigns the role of Beta to Bob. The role assignment for Beta can be broadcast to the mobile devices of other co-located participants in the AR session. This process can be repeated until all roles have been assigned and / or roles have been assigned to all co-located participants in the AR session. Once all roles have been assigned, the AR session can continue with the AR story.

[0070] The mobile devices of participants who are not the host will see a message indicating that they are waiting for the host to join. Once the host joins, the participants of the AR session (except the host) will see a message indicating that roles will be assigned to the story. Each such participant will then see the message "If you want to play the role Alpha, please wave." Once the host has identified that Alice is waving, Alice's mobile device will be updated with a message indicating that the host knows that she is waving. Alice's mobile device will also be updated with a message indicating that she has been assigned the role Alpha. Alice will also see the message "If you want to play the role Beta, please wave," and, once the host identifies that Bob is waving, Alice's mobile device will be updated with a message indicating that the host knows that Bob is waving. Alice's mobile device will also be updated with a message indicating that Bob has been assigned the role Beta. Corresponding messages will also be sent to Bob's mobile device and the mobile devices of other co-located participants as appropriate. This process repeats until all roles have been assigned and / or all co-located participants in the AR session have been assigned roles. The mobile devices of all co-located participants except the host indicate that the mobile device is waiting for the host to start the AR story. Once all roles have been assigned, the AR session can continue as the host initiates the AR story.

[0071] The host can of course assign characters and costumes to herself by selecting them directly on her mobile device within her AR session. The host's character assignments can be broadcast to other co-located participants in the AR session. The host can also select AR stories to broadcast to other co-located participants for role playing and similar activities.

[0072] In a sample configuration, the clothing assigned according to the techniques described herein can be a full-body model clothing that is embedded in an AR object (such as a Lens available from Snap Inc. of Santa Monica, California). Roles and clothing are assigned to mobile devices, which in turn are associated with the body of one of the co-located participants in the AR session. The location of the mobile device is in turn known through the co-located session, thereby facilitating role and clothing assignment. Therefore, the system tracks the mobile devices in the co-located environment to obtain the three-dimensional coordinates of each mobile device and each participant in the AR session. The screen space coordinates of each participant's head are used to assign the participant to his or her mobile device (also in screen coordinates) based on the proximity of the participant's head to a specific mobile device with a user ID known from the session data. It will be understood that since the role and clothing are assigned by the user ID of the mobile device, the role and clothing travel with the mobile device and are assigned to the participant closest to the mobile device. In other words, Alice and Bob can switch roles by exchanging mobile devices. Once matched to the body, the joints of the clothing can be mapped to move with the body of the participant to whom the clothing is assigned.

[0073] Those skilled in the art will further appreciate that the technology described herein can be further extended to SPECTACLES, such as those available from Snapshot Corporation of Santa Monica, California. TM In such cases, the message is presented to the display of the eyeglass device, and the camera of the eyeglass device is used to capture images of other co-located participants in the AR session. In such cases, the position of the eyeglasses will very closely match the position of the participant's head, thus simplifying the matching of the mobile device to the corresponding participant.

[0074] The technology described herein can also be used with one or more computer systems described herein or with one or more other systems. For example, the various processes described herein can be implemented with hardware or software, or a combination of the two. At least one of the processors, memories, storage, output devices, input devices, or communication connections discussed below can each be at least a portion of one or more hardware components. A dedicated hardware logic component can be constructed to implement at least a portion of one or more of the technologies described herein. For example and without limitation, such hardware logic components can include field programmable gate arrays (Field-programmable Gate Array, FPGA), application-specific integrated circuits (Application-specific Integrated Circuit, ASIC), application-specific standard products (Application-specific Standard Product, ASSP), systems on chip (System-on-a-chip system, SOC), complex programmable logic devices (Complex Programmable Logic Device, CPLD), etc. The applications of the devices and systems that can include various aspects can widely include various electronic and computer systems. The technology can be implemented using two or more specific interconnected hardware modules or devices with related control and data signals, which can be transmitted between modules and through modules, or as part of an ASIC. In addition, the technology described herein can be implemented by a software program executable by a computer system. As examples, implementations may include distributed processing, component / object distributed processing, and parallel processing. Furthermore, virtual computer system processing may be constructed to implement one or more techniques or functions as described herein.

[0075] As described herein, examples may include or operate on a processor, logic or multiple components, modules or mechanisms ("modules" herein). A module is a tangible entity (e.g., hardware) that can perform a specified operation and can be configured or arranged in some way. In an example, a circuit can be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuits) as a module. In an example, one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors in their entirety or in part can be configured as a module by firmware or software (e.g., instructions, application parts or applications), and the module operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. The software causes the hardware to perform the specified operation when executed by the underlying hardware of the module.

[0076] Thus, the term "module" is understood to encompass at least one of a tangible hardware or software entity, which refers to an entity that is physically constructed, specially configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a specified manner or to perform part or all of any operations described herein. Considering examples in which modules are temporarily configured, each of the modules need not be instantiated at any one time. For example, in the case where the module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, constituting a particular module at one time instance and constituting different modules at different time instances.

[0077] Although a machine-readable medium may be a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., at least one of a centralized or distributed database or associated cache or server) configured to store one or more instructions. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by a processor such as CPU 210 and which causes CPU 210 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory, and optical and magnetic media. Specific examples of machine-readable media may include nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and digital video disk (DVD)-ROM disks. In some examples, the machine-readable medium includes non-transitory machine-readable media. In some examples, the machine-readable medium may include a machine-readable medium that is not a transitory propagating signal.

[0078] Example communication networks that may be implemented by a mobile device may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a Plain Old Telephone Service (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as ), IEEE 802.15.4 series standards, Long Term Evolution (LTE) series standards, Universal Mobile Telecommunications System (UMTS) series standards, peer-to-peer (P2P) networks, etc. In an example, the short-range transceiver 255 and the WWAN transceiver 250 may include multiple antennas 1030 to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology for wireless communication. In some examples, the network interface device may use multi-user MIMO technology for wireless communication.

[0079] The features and flow charts described herein may be embodied in one or more methods as method steps or in one or more applications as described above. According to some configurations, an "application" or "applications" is a program that performs the functions defined in a program. Various programming languages ​​may be used to generate one or more applications that are structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system of a mobile phone or other mobile operating system. In this example, a third-party application can call an API (Application Programming Interface) provided by the operating system to facilitate the functionality described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in specialized computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0080] The program aspects of the technology may be considered to be a "product" or "article of manufacture" typically in the form of at least one of executable code or associated data (these executable code or associated data are carried on or embodied in a type of machine-readable medium). For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which can provide non-transitory storage for software programming at any time. All or part of the software may sometimes be transmitted over the Internet or various other telecommunications networks. Such communications may enable software to be loaded from one computer or processor to another. Thus, another type of medium that can carry programming, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline telephone networks, and through various air links. Physical elements that carry such waves (such as wired or wireless links, optical links, or the like) may also be considered as media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0081] Thus, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as can be used to implement the client devices, media gateways, transcoders, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including the wires that make up the bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media thus include: a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, a punched card tape, any other physical storage medium with a pattern of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read at least one of a programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0082] The scope of protection is limited only by the claims that now follow. When interpreted in light of this specification and the subsequent prosecution history, the scope is intended and should be interpreted to be broad and encompass all structural and functional equivalents consistent with the ordinary meaning of the language used in the claims. Nevertheless, none of the claims is intended to include subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0083] Except as stated immediately above, nothing stated or illustrated is intended or should be construed as causing any element, step, feature, object, benefit, advantage, or equivalent to be dedicated to the public regardless of whether it is recited in the claims.

[0084] It should be understood that the terms and expressions used in this article have the ordinary meanings assigned to the corresponding fields of inquiry and research corresponding to such terms and expressions, unless the specific meanings are otherwise set forth herein. Relational terms such as first and second can be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", "includes", "including" or any other variations thereof are intended to cover non-exclusive inclusions, so that the process, method, article or device including a list of elements or steps includes not only those elements or steps, but also includes other elements or steps inherent to such processes, methods, articles or devices that are not explicitly listed. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0085] In addition, in the foregoing detailed description, various features are grouped together in various examples for the purpose of simplifying the present disclosure. The disclosure method should not be interpreted as reflecting the intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, and each claim exists independently as a separately claimed subject matter.

[0086] Although the best mode and other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and may be applied to many applications, only some of which have been described herein. It is intended that all modifications and variations that fall within the true scope of the present concept be claimed by the following claims.

Claims

1. A mobile device, comprising: camera; a memory for storing instructions; as well as a processor coupled to the memory, wherein the processor executes the instructions to assign a role to at least one participant in a multi-party augmented reality (AR) session by executing the instructions to perform operations comprising: receiving, from an AR session, an indication that a role is to be assigned to at least one participant in a multi-party AR session, the AR session including at least two co-located session participants, the multi-party session enabling all session participants to communicate with each other using respective mobile devices; receiving a role assignment triggering event for assignment of said role; Obtaining a user ID of a participant in the multi-party AR session who provided the role assignment triggering event; Allocating the role to a participant in the multi-party AR session who has provided the role allocation triggering event; broadcasting the role assignment of the role for the participant in the multi-party session who provided the role assignment triggering event to the at least two co-located session participants; and Maps user IDs to assigned roles.

2. The mobile device according to claim 1 further includes the processor executing instructions to, after receiving an indication from the AR session that the role will be assigned to at least one participant in the multi-party AR session, send a message to at least one participant in the multi-party AR session who was not previously assigned a role, stating that the role can be assigned to the participant if at least one participant provides a predetermined gesture.

3. The mobile device according to claim 2, wherein: Receiving the role assignment triggering event for assignment of the role includes the processor executing instructions to detect a waving gesture from a co-located session participant in the multi-party AR session.

4. The mobile device according to claim 3, wherein: Obtaining the user ID of the co-located session participant who provided the role assignment trigger event includes the processor executing instructions to capture the screen space position of the head of the co-located session participant who provided the role assignment trigger event, comparing the captured screen space position of the head of the co-located session participant who provided the role assignment trigger event with the screen space position of each co-located session participant, and determining the user ID of the co-located session participant that is closest to the screen space position of the head of the co-located session participant who provided the role assignment trigger event.

5. The mobile device according to claim 1, wherein: The processor also executes instructions to assign a virtual costume to a participant in the multi-party AR session who provides the role assignment trigger event by: Obtaining a screen space position of a mobile device of a participant in the multi-party AR session who provided the role assignment triggering event; determining whether a body that has not been assigned a virtual garment is detected in the view of the camera; as well as For each body in the camera's view that has not yet been assigned a virtual costume, obtain the screen space position of the body's head, determine whether the screen space position of the body's head is closest to the screen space position of the mobile device of the participant in the multi-party AR session that provided the role assignment trigger event, and assign the body with the screen space position closest to the mobile device of the participant in the multi-party AR session that provided the role assignment trigger event as the body of the participant in the multi-party AR session that provided the role assignment trigger event.

6. The mobile device of claim 5, further comprising the processor executing instructions to track a body of a participant in the multi-party AR session who provided the role assignment trigger event, retrieve a body mesh corresponding to a virtual costume of the character, and set the body mesh to the tracked body of the participant in the multi-party AR session who provided the role assignment trigger event.

7. The mobile device of claim 6, further comprising the processor executing instructions to broadcast the role assignment of the virtual costume for the participant who provided the role assignment triggering event in the multi-party AR session to the at least two co-located session participants.

8. The mobile device according to claim 5, wherein: The processor executing instructions to determine whether the screen space position of the head of the body is closest to the screen space position of the mobile device of the participant in the multi-party AR session who provided the role assignment trigger event includes the processor executing instructions to: (a) obtaining session data related to a first participant from a list of co-located participants in the multi-party AR session; (b) determining the screen space position of the first participant; (c) comparing the screen space position of the head of the participant who provided the role assignment trigger event in the multi-party AR session with the screen space position of the first participant; as well as (d) Repeat steps (a) to (c) for each co-located participant in the multi-party AR session.

9. The mobile device according to claim 8, wherein: The processor executes instructions to determine whether the screen space position of the head of the body is closest to the screen space position of the mobile device of the participant who provided the role assignment trigger event in the multi-party AR session, including: the processor executes instructions to determine the user ID of the closest co-located participant.

10. A method for assigning a role to at least one participant in a multi-party augmented reality (AR) session, comprising: receiving, from an AR session, an indication that a role is to be assigned to at least one participant in a multi-party AR session, the AR session including at least two co-located session participants, the multi-party session enabling all session participants to communicate with each other using respective mobile devices; receiving a role assignment triggering event for assignment of said role; Obtaining a user ID of a participant in the multi-party AR session who provided the role assignment triggering event; Allocating the role to a participant in the multi-party AR session who has provided the role allocation triggering event; broadcasting the role assignment of the role for the participant in the multi-party session who provided the role assignment triggering event to the at least two co-located session participants; as well as Maps user IDs to assigned roles.

11. The method according to claim 10 further includes, after receiving an indication from the AR session that a role will be assigned to at least one participant in the multi-party AR session, sending a message to at least one participant in the multi-party AR session who has not been previously assigned a role, stating that the role can be assigned to the participant if at least one participant provides a predetermined gesture.

12. The method according to claim 11, wherein: Receiving the role assignment triggering event for assignment of the role includes detecting a waving gesture from a co-located session participant in the multi-party AR session.

13. The method according to claim 12, wherein: Obtaining the user ID of the co-location session participant who provided the role assignment trigger event includes capturing the screen space position of the head of the co-location session participant who provided the role assignment trigger event, comparing the captured screen space position of the head of the co-location session participant who provided the role assignment trigger event with the screen space position of each co-location session participant, and determining the user ID of the co-location session participant that is closest to the screen space position of the head of the co-location session participant who provided the role assignment trigger event.

14. The method according to claim 10, further comprising assigning a virtual costume to a participant in the multi-party AR session who provided the role assignment triggering event by: Obtaining a screen space position of a mobile device of a participant in the multi-party AR session who provided the role assignment triggering event; determining whether a body that has not been assigned a virtual garment is detected in a view of a camera of the mobile device; as well as For each body in the view of the camera of the mobile device that has not yet been assigned a virtual costume, the screen space position of the head of the body is obtained, it is determined whether the screen space position of the head of the body is closest to the screen space position of the mobile device of the participant in the multi-party AR session who provided the role assignment trigger event, and the body with the screen space position closest to the mobile device of the participant in the multi-party AR session who provided the role assignment trigger event is assigned as the body of the participant in the multi-party AR session who provided the role assignment trigger event.

15. The method according to claim 14 also includes tracking the body of the participant in the multi-party AR session who provided the role assignment trigger event, retrieving the body mesh corresponding to the virtual clothing of the character, and setting the body mesh to the tracked body of the participant in the multi-party AR session who provided the role assignment trigger event. 16 . The method of claim 15 , further comprising broadcasting the role assignment of the virtual costume for the participant in the multi-party AR session who provided the role assignment triggering event to the at least two co-located session participants.

17. The method of claim 14, further comprising determining whether the screen space position of the head of the body is closest to the screen space position of a mobile device of a participant in the multi-party AR session who provided the role assignment trigger event by: (a) obtaining session data related to a first participant from a list of co-located participants in the multi-party AR session; (b) determining the screen space position of the first participant; (c) comparing the screen space position of the head of the participant who provided the role assignment trigger event in the multi-party AR session with the screen space position of the first participant; as well as (d) Repeat steps (a) to (c) for each co-located participant in the multi-party AR session.

18. The method according to claim 17, wherein: Determining whether the screen space position of the head of the body is closest to the screen space position of the mobile device of the participant who provided the role assignment trigger event in the multi-party AR session also includes: determining the user ID of the closest co-located participant.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to assign a role to at least one participant in a multi-party augmented reality (AR) session by performing operations comprising: receiving, from an AR session, an indication that a role is to be assigned to at least one participant in a multi-party AR session, the AR session including at least two co-located session participants, the multi-party session enabling all session participants to communicate with each other using respective mobile devices; receiving a role assignment triggering event for assignment of said role; Obtaining a user ID of a participant in the multi-party AR session who provided the role assignment triggering event; Allocating the role to a participant in the multi-party AR session who has provided the role allocation triggering event; broadcasting the role assignment of the role for the participant in the multi-party session who provided the role assignment triggering event to the at least two co-located session participants; as well as Maps user IDs to assigned roles.

20. The non-transitory computer-readable storage medium of claim 19, further comprising instructions that, when executed by the at least one processor, cause the at least one processor to assign a virtual costume to a participant in the multi-party AR session who provided the role assignment triggering event by: Obtaining a screen space position of a mobile device of a participant in the multi-party AR session who provided the role assignment triggering event; determining whether a body that has not been assigned a virtual garment is detected in a view of a camera of the mobile device; as well as For each body in the view of the camera of the mobile device that has not yet been assigned a virtual costume, the screen space position of the head of the body is obtained, it is determined whether the screen space position of the head of the body is closest to the screen space position of the mobile device of the participant in the multi-party AR session who provided the role assignment trigger event, and the body with the screen space position closest to the mobile device of the participant in the multi-party AR session who provided the role assignment trigger event is assigned as the body of the participant in the multi-party AR session who provided the role assignment trigger event.

Images