Same-address whole-body gestures

Abstract

A method for detecting a whole body gesture by a mobile device that includes a host mobile device that detects a tracked body of a co-located participant in a multi-party session. When the tracked body of the participant provides a full-body gesture, the host's mobile device discriminates the tracked body. The host mobile device traverses the list of participants in the multi-party session and finds the closest participant mobile device relative to the screen space location of the head of the participant making the gesture. The host mobile device then retrieves the user ID of the closest participant mobile device and broadcasts the recognized whole body gesture event along with the retrieved user ID to all co-located participants in the multi-party session. The mobile device of each participant may then handle the gesture event according to the situation of the multi-party session. For example, a character or garment may be assigned to a participant making a gesture.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 901,483, filed on September 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to mobile devices. More specifically, but not by way of limitation, the present disclosure describes techniques for detecting full-body gestures from co-located participants using a mobile device's camera in a multi-party conversation, determining which participant triggered the gesture, and propagating the detected gesture to the co-located conversation participants. Background Art

[0004] Mobile devices such as smart phones can be adapted to discern gestures for triggering events in software applications executed on the mobile device. The camera of the mobile device can be adapted to detect gestures in an image and process the gesture to identify events to be triggered in response to the gesture. For example, the camera can discern that the person in the image is extending a finger, and then select the corresponding option from the menu of the software application. For gesture detection, the biometric component of the mobile device can include a component for detecting expressions (e.g., hand expressions, facial expressions, body gestures, eye tracking and the like) based on images captured by the camera of the mobile device. Existing body tracking technology can be adapted to detect body gestures in images captured by the camera. However, the user's mobile device cannot detect many of her own body gestures, because when the user's mobile device is held in front of her by her hand, her mobile device does not have a complete camera view of her body. Therefore, detectable body gestures have generally been limited to hand expressions and facial expressions and a limited number of gestures in the field of view when the user holds the camera in front of her body. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The features of the various embodiments disclosed will be readily understood from the following detailed description, in which reference is made to the accompanying drawings. Reference numerals are used with each element in the specification and throughout the multiple views of the drawings. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements.

[0006] The various elements shown in the drawings are not drawn to scale unless otherwise specified. The sizes of the various elements may be enlarged or reduced for clarity. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0007] Figure 1 is a diagram illustrating multiple participants of a co-located augmented reality (AR) session in a sample configuration;

[0008] Figure 2 is a block diagram of electronic components of a mobile device suitable for implementing a co-located AR session in a sample configuration;

[0009] Figure 3 is used in the sample configuration Figure 2 A block diagram of software components for implementing a co-located AR session on a mobile device of

[0010] Figure 4 is a flow chart of a method in a sample configuration for detecting a full body gesture, determining which participant performed the full body gesture, and propagating the detected gesture to other co-located participants of a multi-party AR session. DETAILED DESCRIPTION

[0011] In a co-located augmented reality (AR) session, full-body gestures (waving, jumping, kicking, etc.) can be a useful method of interaction. As an example scenario, Alice and Bob, both in a co-located AR session, hold their mobile devices toward each other. Existing body tracking technology is able to detect body gestures, but Alice's mobile device cannot detect her own body gestures (because when Alice holds the mobile device in front of her with her hands, her mobile device does not have a complete camera view of her body). In this case, only Bob's camera of Bob's mobile device can detect Alice's body gestures. However, there is no mechanism on Bob's mobile device to distinguish that the gesture is Alice's gesture and indicate to Alice's mobile device that she has performed a full-body gesture so that the software on Alice's mobile device can respond accordingly. Therefore, a mechanism is needed to enable a body gesture detected on an auxiliary mobile device (in this scenario, Bob's mobile device) to be associated back to the mobile device of the user who triggered the gesture (in this scenario, Alice's mobile device).

[0012] Thus, this document describes a process for detecting a full body gesture corresponding to a gesture event, determining which user triggered the gesture event, and propagating the gesture event among co-located participants.

[0013] In a sample configuration, waving can be provided as an example of a full-body gesture. In the sample configuration, the host mobile device detects the tracked body of a co-located participant in a multi-party session. The tracked body of the participant begins to wave, and the host's mobile device identifies that there is a tracked body waving. The host mobile device traverses the list of co-located participants in the multi-party session and finds the participant mobile device closest to the screen space position of the head of the waving body. The host mobile device then obtains the user ID of the closest participant mobile device. The host mobile device can then broadcast the identified full-body gesture event (session participant waving) together with the obtained user ID to all co-located participants in the multi-party session. Each participant's mobile device can then process the gesture event according to the situation of the multi-party session. The mobile device of the waving participant can identify that its user is a participant who has performed a waving gesture and respond to the waving gesture event according to the situation of the co-located session. For example, a character or costume can be assigned to a waving participant in an AR session or the AR environment can be changed in response to a waving gesture event.

[0014] The system described herein provides a method for implementing a gesture event detected by another co-located party in a multi-party session. The method includes generating a multi-party session (e.g., a multi-party augmented reality (AR) session) including at least two co-located session participants. The multi-party session can enable all session participants to communicate with each other using their respective mobile devices. The camera of the mobile device is used to detect the gesture event of the co-located session participant, and the screen space position of the head of the co-located session participant that generated the gesture event is captured. The detected gesture event can be a full-body gesture (e.g., T-pose), an upper body gesture (e.g., waving) or a lower body gesture (e.g., foot click) of the co-located session participant. The captured screen space position of the head of the co-located session participant that generated the gesture event is compared with the screen space position of each co-located participant in the multi-party session, and the user identification (ID) of the co-located participant that is closest to the screen space position of the head of the co-located session participant that generated the gesture event is determined. The detected gesture event and the determined user ID can then be broadcast to at least the co-located session participant in the multi-party session. If the mobile device corresponds to the user ID or can respond to the gesture event from the determined user ID, an action corresponding to the detected gesture event received from another mobile device can be performed in the multi-party session. For example, the detected gesture event can trigger a full screen effect on the mobile device of at least one session participant, a visual effect attached to the body of at least one session participant, a switching of objects in the session, or the playing of an animation in the session.

[0015] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products, which illustrate the examples set forth in the present disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without such details. The aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in a variety of ways. The terms and nomenclature used herein are only used for the purpose of describing specific aspects and are not intended to be restrictive. In general, known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0016] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links or the like, by which an electrical or magnetic signal generated or supplied by one system element is communicated to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communications media, one or more of which may modify, manipulate, or carry electrical signals. The term "on" means supported by an element directly or indirectly supported by an element through another element integrated into or supported by the element.

[0017] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art after examination of the following and the accompanying drawings, or may be learned by production and operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, means and combinations particularly pointed out in the appended claims.

[0018] Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below. Sample systems and methods for detecting full body gestures, determining which co-located participant in a multi-party session triggered a gesture event, and propagating gesture events among co-located participants will be described with respect to Figures 1 to 4 Give a description.

[0019] Figure 1 is a diagram illustrating multiple participants 100, 100', and 100" in a co-located augmented reality (AR) session in a sample configuration. Although three participants are illustrated, the co-located AR session can include from two to as many participants as the AR session software can support (e.g., 64 participants). Each participant 100, 100', and 100" has a mobile device 200, 200', and 200" respectively that is used by the participant 100, 100', and 100" to participate in the co-located AR session. As shown, each mobile device 200, 200', and 200" uses, for example Low energy consumption ( Low Energy (BLE) or any other appropriate local area communication technology to communicate with each other co-located mobile device 200, 200' and 200". During the co-located AR session, each participant 100, 100' and 100" can share one or more AR applications with each other participant 100, 100' and 100". In addition, each mobile device 200, 200' and 200" can be wirelessly connected to the Internet 110 using a suitable broadband cellular network technology (e.g., 5G) and is enabled to communicate with a remote AR application server 120, which can provide AR applications for the co-located AR session. For example, the AR application server 120 may provide an AR application to be shared among the mobile devices 200, 200', and 200" during a co-located AR session, such as a Lens of the type available from Snap Inc. of Santa Monica, California. The following description will utilize Lens as an example of an AR application that may be used in a co-located session as described herein; however, those skilled in the art will appreciate that other AR applications or other multi-party applications may similarly be shared in a multi-party session as described herein.

[0020] The AR application server 120 may include one or more computing devices as part of a service or network computing system, the AR application server 120 including a processor, a memory, and a network communication interface to communicate with the mobile devices 200, 200' and 200" via the Internet 110. In a sample configuration, one or more of the mobile devices 200, 200' and 200" may also be paired with an electronic glasses device (not shown) via a high-speed wireless connection. The AR application server 120 may store a gallery of snapshots and AR objects, which is maintained by the AR application server 120 for access by each participant 100, 100' and 100" and is called via Internet communications that provide links to the snapshots and AR objects stored in the gallery.

[0021] In a sample configuration, an AR session may be created for co-located participants 100, 100', and 100" by invoking a connection template of the type available, for example, from SnapChat, Inc. of Santa Monica, California. The connection template allows users of a social media network (such as SNAPCHAT, available from SnapChat, Inc. of Santa Monica, California) to join with social media friends to share a common AR application in a common AR space, either in the same place or remotely. The connection template may include a reusable invitation flow that manages how users join an AR session with their social media friends and a relay system that enables participants 100, 100', and 100" in the AR session to pass messages among themselves. An avatar system may also be provided to provide co-presence in a shared AR session when one or more of the participants 100, 100', or 100" are not co-located with the other participants 100, 100', and 100". In the sample configuration, the connection template may also provide examples of synchronizing object instantiation and transformation manipulation within a shared AR session.

[0022] In an example, a shared AR session may implement a shared AR application, such as the Connected Lens available from Snap Inc. of Santa Monica, California. The Connected Lens is a type of Lens that allows participants in a shared session to transfer and store data between mobile devices 200, 200', and 200" running the same Lens, whether the mobile devices 200, 200', and 200" are co-located or in different locations. The Connected Lens thus allows participants 100, 100', and 100" to create AR experiences that social media friends can experience together at the same time and replay over time. The sample AR experience may permit participants 100, 100', and 100" to instantiate objects, modify the position / rotation / scale of objects that are synchronized across mobile devices 200, 200', and 200", and send messages among participants 100, 100', and 100". When participants 100, 100' and 100" are co-located, participants 100, 100' and 100" can reference the same co-located AR experience on their respective mobile devices 200, 200' and 200". To invite social media friends to such a co-located AR experience, participant 100, 100' or 100" can first map the area, and then in the case of Connected Lens, Lens can display a code (e.g., a Snapcode) that other participants 100, 100' and 100" can scan to join the co-located AR experience. The mapped area does not provide the geometry of the scene but allows participants 100, 100' and 100" to share the same coordinate system with other participants 100, 100' and 100". A single AR application such as Lens can include a single user experience, a remote experience, or a co-located experience, as desired by the participants 100, 100' and 100".

[0023] In the example of Connected Lens, Lens can create an AR session that can be joined by others using the Connected Lens module. Adding the Connected Lens module transforms the Lens into a Connected Lens with a Call to Action button that participant 100, 100', or 100" can click to enter the ConnectedLens State, at which point participant 100, 100', or 100" receives the ability to invite other participants 100, 100', and 100" to enter the connected AR session. When participant 100, 100', or 100" enters Lens via invitation, participant 100, 100', or 100" will enter the Connected Lens State directly. At this point, each connected participant 100, 100', and 100" will not have access to any camera modes, except for changing between the front and rear cameras. In addition, instead of a carousel for selected Lenses, participants 100, 100', and 100" will have an Action bar that allows participants 100, 100', and 100" to see who else is in the connected AR session. Within the connected AR session, each participant 100, 100', and 100" can send messages to and receive messages from each other participant in the connected AR session. To facilitate communication, each participant in the connected AR session is assigned a user ID for the connected AR session.

[0024] When participant 100, 100', or 100" opens Connected Lens for the first time, she is first introduced to a splash screen that introduces participant 100, 100', or 100" to the current Connected Lens experience and provides her with the option to begin interacting with the connected AR session by selecting a Call to Action button to initiate Connected Lens. Once participant 100, 100', or 100" completes the invitation process, she is brought into the connected AR session in which Lens can send data back and forth between each participant 100, 100', and 100".

[0025] It will be appreciated by those skilled in the art that by using the Connected Lenses Module to set up a multi-player session to make Lenses available for multi-party applications, the co-location session can also be used on an electronic eyewear device (such as the SPECTACLES available from Snap Inc. of Santa Monica, California). TM ) on the .

[0026] Figure 2is a block diagram of electronic components of a mobile device 200, 200', or 200" in a sample configuration suitable for implementing a co-located AR session. In the sample configuration, the mobile device 200 can present an interface (e.g., an AR interface) to the participant 100 and can also be used to extend the interface to an associated electronic eyewear device (not shown). The mobile device 200 may include a flash memory 205 that stores programming to be executed by the CPU 210 to perform all or a subset of the functions described herein. The mobile device 200 may further include a camera 225, which includes one or more visible light cameras (visible light cameras with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 205 may further include multiple images or videos generated via the camera 225.

[0027] The mobile device 200 may further include an image display 230, a mobile display driver 235 for controlling the image display 230, and a display controller 240. Figure 2 In the example of , image display 230 may include a user input layer 245 (e.g., a touch screen) layered on top of or otherwise integrated into the screen used by image display 230. Examples of touch screen type mobile devices that may be used include, but are not limited to, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, Figure 2 Thus provided is a block diagram illustration of an example mobile device 200 having a user interface including a touch screen input layer 245 for receiving input (by touch, multi-touch or gestures, etc., by hand, stylus or other implement), and an image display 230 for displaying content.

[0028] like Figure 2 As shown, the mobile device 200 includes at least one digital transceiver (XCVR) 250, shown as a WWAN (Wireless Wide Area Network) XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 200 may also include additional digital or analog transceivers, such as a short-range transceiver (XCVR) 255 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, or For example, the short-range XCVR 255 may employ one or more standard protocols for communicating with wireless local area networks (such as IEEE 802.11). The invention may be in the form of any available two-way wireless local area network (WLAN) transceiver of a type compliant with one of the standards).

[0029] In order to generate location coordinates for locating the mobile device 200, the mobile device 200 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 200 may utilize either or both of the short-range XCVR 255 and the WWAN XCVR 250 to generate location coordinates for positioning. For example, a cellular network, or based on The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the mobile device 200 via the XCVR 250, 255 through one or more network connections.

[0030] The transceivers 250, 255 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 250 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". The transceivers may also be incorporated into broadband cellular network technologies known as "5G". In a sample configuration, the transceivers 250, 255 provide two-way wireless communication of information including digitized audio signals, still images and video signals, web page information for display and input associated with the web page, and various types of mobile message communications to / from the mobile device 200.

[0031] The mobile device 200 may further include a microprocessor that functions as a central processing unit (CPU) 210. The CPU 210 is a processor having elements that are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, the example utilizes components that form a programmable CPU. For example, the microprocessor may include one or more integrated circuit (IC) chips that integrate electronic components to perform the functions of the CPU 210. The CPU 210 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form a CPU 210 or processor hardware in a smart phone, a laptop computer, and a tablet computer.

[0032] The CPU 210 functions as a programmable host controller for the mobile device 200 by configuring the mobile device 200 to perform various operations according to instructions or programming executable by the CPU 210. For example, such operations may include various general operations of the mobile device 200, as well as operations related to programming for a messaging application and an AR camera application on the mobile device 200. Although a processor may be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing programming.

[0033] The mobile device 200 also includes a memory or storage system for storing programming and data. Figure 2 In the example shown in , the memory system may include flash memory 205, random-access memory (RAM) 260, and other memory components 265 as needed. RAM 260 may be used as short-term storage for instructions and data being processed by CPU 210, for example, as working data processing memory. Flash memory 205 typically provides longer-term storage.

[0034] In the example of mobile device 200, flash memory 205 may be used to store programming or instructions for execution by CPU 210. Depending on the type of device, mobile device 200 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPAD devices), Windows Mobile, Amazon Fire OS (Operating System, operating system), RIM BlackBerry OS, or the like.

[0035] Mobile device 200 may also include an audio transceiver 270 that may receive audio signals from the environment via a microphone (not shown) and provide audio output via a speaker (not shown). A messaging application or social media application implemented on mobile device 200 may couple audio signals with video signals and other messages.

[0036] In a sample configuration, the mobile device 200 may further include a biometric component 275 to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measure bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), and similar operations. The motion component may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), and the like. The position component may include a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a position sensor component for generating positioning system coordinates, and a position sensor component for generating a positioning system coordinate. or transceivers, altitude sensor components (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), orientation sensor components (e.g., a magnetometer), and the like. Such positioning system coordinates may also be received from other devices via wireless connections.

[0037] In a sample configuration, the mobile device 200 can execute mobile augmented reality (AR) application software 220 loaded into the flash memory 205, such as the one available from Snap Inc. of Santa Monica, California. As will be discussed below Figure 3 to Figure 4 Described in more detail, the mobile AR application 220 may be executed to create a multi-party session, detect full body gestures of participants, determine which session participant triggered a gesture event, and propagate the gesture event among co-located participants in the multi-party session.

[0038] Figure 3 is used in the sample configuration Figure 2 2 is a block diagram of software components for implementing an AR session on a mobile device 200. As shown, the mobile device 200 executes an AR application 220 to implement an AR session 310 rendered by an AR renderer 320 to a display 230 of the mobile device 200. The AR session can be implemented as a connected AR application session 330, such as a ConnectedLens session as described above. The connected AR application session 330 controls the AR session with respect to the mobile device 200. Figure 1 The connected AR application session 330 may communicate with other co-located AR devices (including mobile devices 200' and 200") in the co-located session. The Connected Lens in the connected AR application session 330 may function to overlay one or more shared Lenses on images received from the cameras 225 of the respective mobile devices 200, 200' and 200" of each participant 100, 100' and 100" in the connected AR application session 330.

[0039] In a sample configuration, the AR application 220 further implements full body tracking software 340 and gesture / participant recognition software 350 during an AR session. The full body tracking software 340 may implement a three-dimensional (3D) body template that allows the participant 100, 100', or 100" to track the position and rotation of the joints of the body in 3D in the image captured by the camera 225. One or more bodies and parts of the body may be tracked simultaneously. In a sample configuration, the 3D model may be assembled to a skeleton that matches what the full body tracking software 340 sees in the image from the camera 225. This allows an AR application (e.g., Lens) to map the body's movements to the model's movements.

[0040] The 3D rigged model may be based on a stick figure with joints corresponding to the person's feet, knees, hips, shoulders, spine, neck, head, elbows, hands, and the like. The 3D rigged model assigns all of the joints of the person in the image from the camera 225 to the 3D rigged model for use by the 3D body tracking software 340. The 3D body tracking software 340 then tracks the movement of the person in the image from the camera by causing the 3D rigged model to move with the person as the person moves. The scale of the 3D rigged model may be adjusted as appropriate to fit the person in the image from the camera 225.

[0041] When the camera 225 captures overlapping images of more than one person, the location of the invisible parts of the body may not be known. In this case, the 3D body tracking software 340 can disable or hide the joints that are not automatically tracked. Independent objects, such as AR objects in the form of Lenses, can also be attached to the tracked body by attaching the AR objects to the independent joints of the people in the image.

[0042] In order to recognize gestures performed by people in images, a full-body trigger template can be implemented, which allows different screen effects to be triggered by different body movements, such as custom body gestures (e.g., waving), touch screen triggers, and the distance between body joints, or the distance between body joints and screen elements identified in screen transformations. The full-body trigger template can implement a full-body tracking system for tracking full-body gestures, an upper-body tracking system for tracking upper-body gestures, or a lower-body tracking system for tracking lower-body gestures, depending on the situation. The full-body tracking system can also track facial features by providing access to facial points such as the nose, left and right eyes, and left and right ears. The trigger template further specifies what should happen in the corresponding AR application (e.g., Connected Lens) when a trigger event is identified. The triggered event can be a full-screen effect, a visual effect attached to the body, a switching of an object, the playing of an animation, or any other content that a trigger event may trigger within an AR application.

[0043] In a sample configuration, the full-body tracking software 340 provides 2D screen points that represent the joints or body parts of the person identified in the image and are the basis of the full-body tracking template. Sample body movements can include T-poses, right foot clicks, waving, and similar actions. Each body movement can have a motion trigger script attached to it, which allows different types of movements to be attached, including gestures. Each gesture can be composed of a list of gesture triggers (e.g., for a right foot click trigger, right leg extends or right leg clicks). When a gesture matches a person in a camera image, a list of behavior triggers is called and matched with the gesture. When a gesture matches a behavior trigger, a corresponding gesture event is activated. The gesture can be a motion, such as waving, which triggers a motion trigger response script to implement a gesture event. For example, a specific body movement of a person in an image can trigger animations and full-screen effects associated with a behavior script, which is associated with an AR object in an AR session.

[0044] Figure 4 The operation of gesture / participant identification software 350 for identifying a full body gesture and the participant performing the gesture is shown in FIG. Figure 4 Flowchart of a method for detecting a full-body gesture in a sample configuration, determining which participant performed the full-body gesture, and propagating the detected gesture to other co-located participants of a co-located AR session.

[0045] In the sample configuration, Figure 4 The method is Figure 3 The gesture / participant recognition software 350 shown in FIG. Figure 4 As shown, the method starts at 400 by determining at 402 whether there is a tracked body detected by the full-body tracking software 340 in the image received from the camera 225 of the mobile device 200. If the tracked body is not detected at 402, the process proceeds to 404 to wait for the tracked body to be detected. Once the tracked body is detected at 402, a determination is made at 406 as to whether the tracked body is performing a full-body gesture. As noted above, a full-body gesture such as waving, jumping, or kicking can be detected from the movement of the body model paired with the received image. If the full-body gesture is not detected at 406, the process proceeds to 408 to wait for the full-body gesture to be detected. Once the full-body gesture is detected at 406, the screen space position of the head of the body making the gesture is captured at 410.

[0046] Once the screen space position of the head of the gesturing body has been captured at 410, the captured screen space position of the head of the gesturing body is compared to the screen space position of each co-located participant in the multi-party AR session. In particular, the first available participant is obtained from the list of co-located participants in the multi-party AR session at 412. At 414, the screen space position of the participant's mobile device 200, 200', or 200" is obtained from the session data and compared to the screen space position of the head of the gesturing body. If the screen space position of the current participant's mobile device 200, 200', or 200" is determined at 416 to be closer than the screen space position of any other participant in the AR session, then at 418 the variable "ClosestParticipant" is set to the user ID of the current participant. If it is determined at 420 that there are other participants in the AR session whose proximity to the screen space position of the head of the gesturing body has not yet been evaluated, then data for the next available participant is obtained at 422, and steps 414 to 422 are repeated until all participants in the AR session have had their proximity to the screen space position of the head of the gesturing body evaluated.

[0047] Once all participants in the AR session have been evaluated, the user ID of the participant closest to the head of the gesturing body is obtained at 424. The captured gesture event (e.g., waving, jumping, or kicking) is then broadcast to other participants in the co-located AR session at 426 along with the user ID of the participant that generated the gesture event. At 428, each AR device 200, 200', and 200" for each participant 100, 100', and 100" in the AR session processes the gesture event based on the nature of the given gesture event and the user ID of the person who generated the gesture event. The process then ends at 430.

[0048] Thus, in the disclosed examples, a participant in a multi-party co-located session can generate a gesture that is not recognizable by the participant's mobile device but is recognizable by the mobile devices of other co-located participants and used to trigger a gesture event that would otherwise not be recognizable.

[0049] Although described in the context of an AR application that shares AR objects among session participants, those skilled in the art will appreciate that the techniques described herein may also be applied to other types of multi-player applications that include co-located players that can capture the full body gestures of other co-located players. Thus, the processes described herein for detecting full body gestures corresponding to gesture events, determining which participant triggered the gesture event, and propagating the gesture event among co-located participants are not limited to AR applications, but may be applied to any such multi-player application that includes co-located players.

[0050] The technology described herein can also be used with one or more computer systems described herein or with one or more other systems. For example, the various processes described herein can be implemented with hardware or software, or a combination of the two. At least one of the processors, memories, storage, output devices, input devices, or communication connections discussed below can each be at least a portion of one or more hardware components. A dedicated hardware logic component can be constructed to implement at least a portion of one or more of the technologies described herein. For example and without limitation, such hardware logic components can include field programmable gate arrays (Field-programmable Gate Array, FPGA), application-specific integrated circuits (Application-specific Integrated Circuit, ASIC), application-specific standard products (Application-specific Standard Product, ASSP), systems on chip (System-on-a-chip system, SOC), complex programmable logic devices (Complex Programmable Logic Device, CPLD), etc. The applications of the devices and systems that can include various aspects can widely include various electronic and computer systems. The technology can be implemented using two or more specific interconnected hardware modules or devices with related control and data signals, which can be transmitted between modules and through modules, or as part of an ASIC. In addition, the technology described herein can be implemented by a software program executable by a computer system. As examples, implementations may include distributed processing, component / object distributed processing, and parallel processing. Furthermore, virtual computer system processing may be constructed to implement one or more techniques or functions as described herein.

[0051] As described herein, examples may include or operate on a processor, logic or multiple components, modules or mechanisms ("modules" herein). A module is a tangible entity (e.g., hardware) that can perform a specified operation and can be configured or arranged in some way. In an example, a circuit can be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuits) as a module. In an example, one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors in their entirety or in part can be configured as a module by firmware or software (e.g., instructions, application parts or applications), and the module operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. The software causes the hardware to perform the specified operation when executed by the underlying hardware of the module.

[0052] Thus, the term "module" is understood to encompass at least one of a tangible hardware or software entity, which refers to an entity that is physically constructed, specially configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a specified manner or to perform part or all of any operations described herein. Considering examples in which modules are temporarily configured, each of the modules need not be instantiated at any one time. For example, in the case where the module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, constituting a particular module at one time instance and constituting different modules at different time instances.

[0053] Although a machine-readable medium may be a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., at least one of a centralized or distributed database or associated cache or server) configured to store one or more instructions. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by a processor such as CPU 210 and which causes CPU 210 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory, and optical and magnetic media. Specific examples of machine-readable media may include nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and digital video disk (DVD)-ROM disks. In some examples, the machine-readable medium includes non-transitory machine-readable media. In some examples, the machine-readable medium may include a machine-readable medium that is not a transitory propagating signal.

[0054] Example communication networks that may be implemented by a mobile device may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a Plain Old Telephone Service (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as ), IEEE 802.15.4 series standards, Long Term Evolution (LTE) series standards, Universal Mobile Telecommunications System (UMTS) series standards, peer-to-peer (P2P) networks, etc. In an example, the short-range transceiver 255 and the WWAN transceiver 250 may include multiple antennas 1030 to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology for wireless communication. In some examples, the network interface device may use multi-user MIMO technology for wireless communication.

[0055] The features and flow charts described herein may be embodied in one or more methods as method steps or in one or more applications as described above. According to some configurations, an "application" or "applications" is a program that performs the functions defined in a program. Various programming languages ​​may be used to generate one or more applications that are structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system of a mobile phone or other mobile operating system. In this example, a third-party application can call an API (Application Programming Interface) provided by the operating system to facilitate the functionality described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in specialized computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0056] The program aspects of the technology may be considered to be a "product" or "article of manufacture" typically in the form of at least one of executable code or associated data (these executable code or associated data are carried on or embodied in a type of machine-readable medium). For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which can provide non-transitory storage for software programming at any time. All or part of the software may sometimes be transmitted over the Internet or various other telecommunications networks. Such communications may enable software to be loaded from one computer or processor to another. Thus, another type of medium that can carry programming, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline telephone networks, and through various air links. Physical elements that carry such waves (such as wired or wireless links, optical links, or the like) may also be considered as media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0057] Thus, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as can be used to implement the client devices, media gateways, transcoders, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including the wires that make up the bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media thus include: a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, a punched card tape, any other physical storage medium with a pattern of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read at least one of a programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0058] The scope of protection is limited only by the claims that now follow. When interpreted in light of this specification and the subsequent prosecution history, the scope is intended and should be interpreted to be broad and encompass all structural and functional equivalents consistent with the ordinary meaning of the language used in the claims. Nevertheless, none of the claims is intended to include subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0059] Except as stated immediately above, nothing stated or illustrated is intended or should be construed as causing any element, step, feature, object, benefit, advantage, or equivalent to be dedicated to the public regardless of whether it is recited in the claims.

[0060] It should be understood that the terms and expressions used in this article have the ordinary meanings assigned to the corresponding fields of inquiry and research corresponding to such terms and expressions, unless the specific meanings are otherwise set forth herein. Relational terms such as first and second can be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", "includes", "including" or any other variations thereof are intended to cover non-exclusive inclusions, so that the process, method, article or device including a list of elements or steps includes not only those elements or steps, but also includes other elements or steps inherent to such processes, methods, articles or devices that are not explicitly listed. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0061] In addition, in the foregoing detailed description, various features are grouped together in various examples for the purpose of simplifying the present disclosure. The disclosure method should not be interpreted as reflecting the intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, and each claim exists independently as a separately claimed subject matter.

[0062] Although the best mode and other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and may be applied to many applications, only some of which have been described herein. It is intended that all modifications and variations that fall within the true scope of the present concept be claimed by the following claims.

Claims

1. A mobile device, comprising: camera; a memory storing instructions; as well as a processor coupled to the memory, wherein the processor executes the instructions to implement a gesture event corresponding to the detected gesture by executing instructions that implement operations including: generating a multi-party session including at least two co-located session participants, the multi-party session enabling all session participants to communicate with each other using respective mobile devices; The camera detects gesture events of co-located conversation participants; capturing a screen space position of a head of a co-located session participant that generated the gesture event; comparing the captured screen-space position of the head of the co-located session participant that generated the gesture event to the screen-space position of each co-located participant in the multi-party session; determining a user identification (ID) of a co-located participant that is closest to the screen space location of the head of the co-located conversation participant that generated the gesture event; and The detected gesture event and the determined user ID are broadcast to at least the co-located session participants in the multi-party session.

2. The mobile device according to claim 1, wherein: The detected gesture event is a full body gesture, an upper body gesture, or a lower body gesture of the co-located session participant.

3. The mobile device according to claim 2, wherein: Detected gesture events include T-gestures, hand waves, or foot taps.

4. The mobile device according to claim 1, wherein: The processor also executes instructions for determining whether the mobile device corresponds to the user ID or is capable of responding to gesture events from the determined user ID, and performing an action corresponding to a detected gesture event received from another mobile device in the multi-party session when the mobile device corresponds to the user ID or the mobile device is capable of responding to gesture events from the determined user ID.

5. The mobile device according to claim 1, wherein: The detected gesture event triggers a full screen effect on a mobile device of at least one session participant, a visual effect attached to a body of at least one session participant, a switching of objects in the session, or the playing of an animation in the session.

6. The mobile device according to claim 1, wherein: The processor executes instructions to generate a multi-party session including at least two co-located session participants, including the processor executing instructions to generate a multi-party augmented reality (AR) session for sharing at least one AR application among the at least two co-located session participants.

7. The mobile device according to claim 1, wherein: The processor executes instructions for detecting gesture events of the co-located session participant, including the processor executing instructions for determining whether a detected tracked body is present in an image received from the camera, determining whether the detected tracked body is performing a gesture, and determining the gesture event from the gesture performed by the detected tracked body.

8. The mobile device according to claim 1, wherein: The processor executes instructions for comparing the captured screen space position of the head of the co-located session participant that generated the gesture event with the screen space position of each co-located participant in the multi-party session, including: the processor executes instructions for (a) obtaining session data related to a first participant from a list of co-located participants in the multi-party session, (b) determining the screen space position of the first participant, (c) comparing the screen space position of the head of the co-located session participant that generated the gesture event with the screen space position of the first participant, and (d) repeating steps (a) to (c) for each co-located participant in the multi-party session.

9. The mobile device according to claim 8, wherein: The processor executes instructions for determining the user ID of the co-located participant that is closest to the screen space position of the head of the co-located session participant that generated the gesture event, including: the processor executes instructions for determining the closest co-located participant in the multi-party session that has a screen space position that is closest to the screen space position of the co-located session participant that generated the gesture event, and determining the user ID of the closest co-located participant.

10. A method for implementing a gesture event corresponding to a detected whole body gesture, comprising: generating a multi-party session including at least two co-located session participants, the multi-party session enabling all session participants to communicate with each other using respective mobile devices; Using a mobile device's camera to detect gesture events of co-located conversation participants; capturing a screen space position of a head of a co-located session participant that generated the gesture event; comparing the captured screen-space position of the head of the co-located session participant that generated the gesture event to the screen-space position of each co-located participant in the multi-party session; determining a user identification (ID) of a co-located participant that is closest to a screen space location of a head of the co-located conversation participant that generated the gesture event; as well as The detected gesture event and the determined user ID are broadcast to at least the co-located session participants in the multi-party session.

11. The method according to claim 10, wherein: The detected gesture event is a full body gesture, an upper body gesture, or a lower body gesture of the co-located session participant.

12. The method according to claim 11, wherein: Detected gesture events include hand waving, foot tapping, or T-gestures.

13. The method according to claim 10, further comprising: Determine whether the mobile device corresponds to the user ID or is capable of responding to gesture events from the determined user ID, and when the mobile device corresponds to the user ID or the mobile device is capable of responding to gesture events from the determined user ID, perform an action corresponding to a detected gesture event received from another mobile device in the multi-party session.

14. The method according to claim 10, further comprising: A full screen effect on a mobile device of at least one session participant, a visual effect attached to a body of at least one session participant, a switching of an object in the session, or a playing of an animation in the session is triggered in response to the detected gesture event.

15. The method according to claim 10, wherein: Generating a multi-party session including at least two co-located session participants includes generating a multi-party augmented reality (AR) session for sharing at least one AR application among the at least two co-located session participants.

16. The method according to claim 10, wherein: Detecting a gesture event of the co-located session participant includes determining whether a detected tracked body is present in an image received from a camera of the mobile device, determining whether the detected tracked body is performing a gesture, and determining the gesture event from the gesture performed by the detected tracked body.

17. The method according to claim 10, wherein: Comparing the captured screen space position of the head of the co-located session participant that generated the gesture event with the screen space position of each co-located participant in the multi-party session includes: (a) obtaining session data related to a first participant from a list of co-located participants in the multi-party session, (b) determining the screen space position of the first participant, (c) comparing the screen space position of the head of the co-located session participant that generated the gesture event with the screen space position of the first participant, and (d) repeating steps (a) to (c) for each co-located participant in the multi-party session.

18. The method according to claim 17, wherein: Determining the user ID of the co-located participant that is closest to the screen space position of the head of the co-located session participant that generated the gesture event includes: determining the closest co-located participant in the multi-party session that has a screen space position that is closest to the screen space position of the co-located session participant that generated the gesture event, and determining the user ID of the closest co-located participant.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to implement a gesture event corresponding to a detected gesture by performing operations comprising: generating a multi-party session including at least two co-located session participants, the multi-party session enabling all session participants to communicate with each other using respective mobile devices; Using a mobile device's camera to detect gesture events of co-located conversation participants; capturing a screen space position of a head of a co-located session participant that generated the gesture event; comparing the captured screen-space position of the head of the co-located session participant that generated the gesture event to the screen-space position of each co-located participant in the multi-party session; determining a user identification (ID) of a co-located participant that is closest to the screen space location of the head of the co-located conversation participant that generated the gesture event; as well as The detected gesture event and the determined user ID are broadcast to at least the co-located session participants in the multi-party session.

20. The non-transitory computer-readable storage medium of claim 19 further comprises instructions that, when executed by the at least one processor, cause the at least one processor to generate the multi-party augmented reality (AR) session by generating a multi-party AR session for sharing at least one augmented reality application among the at least two co-located session participants.

Images