Dance score generator

Abstract

A method of generating a real-time avatar animation using a danceability score begins with a processor receiving a real-time acoustic signal including an acoustic segment. The processor generates a danceability score for each of the acoustic segments using a danceability neural network. The processor generates the first avatar and a real-time animation of the first avatar based on the danceability score and avatar characteristics associated with the first avatar and the second avatar. The processor causes real-time animations of the first avatar and the second avatar to be displayed on the first client device. Other embodiments are described herein.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims the benefit of priority of U.S. Application No. 18 / 491,563, filed on October 20, 2023, which claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 381,757, entitled "DANCEABILITY SCORE GENERATOR", filed on October 31, 2022, which is hereby incorporated by reference in its entirety. Background of the Invention

[0003] The popularity of electronic messaging, augmented reality, and virtual reality continues to grow. Users are increasingly using customized avatars within different platforms, which reflects a global demand for more intuitive communication.

[0004] These customized avatars can be personalized by users to represent the user in various applications, video games, messaging services, etc. Since customized avatars can be generated in a range of different situations, display various emotions, or even be animated, users are able to use customized avatars to more accurately convey their feelings in messages and on different platforms, and can thus be more fully represented by agents using their customized avatars. Brief Description of the Drawings

[0005] In the drawings (which are not necessarily drawn to scale), like reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the most significant digits in the reference numeral refers to the figure number in which the element was first introduced. Some non - limiting examples are shown in the figures of the drawings, in which:

[0006] Figure 1 is a graphical representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0007] Figure 2 is a graphical representation of a messaging system having both client - side functionality and server - side functionality, according to some examples.

[0008] Figure 3 is a graphical representation of a data structure maintained in a database, according to some examples.

[0009] Figure 4 is a graphical representation of a message, according to some examples.

[0010] Figure 5 shows details of an avatar animation system 232, according to one embodiment.

[0011] Figure 6Illustrated is a process 600 for generating real-time avatar animations using danceability scores according to one embodiment.

[0012] Figure 7 Illustrated are systems including head-mounted devices according to some examples.

[0013] Figure 8 Is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein.

[0014] Figure 9 Is a block diagram showing a software architecture in which examples can be implemented. Detailed Description

[0015] Current avatar animation systems can generate dance animations for an avatar to perform based on the music being played. Most of these dance animations are hard-coded and do not take into account the musical characteristics of the song that should drive the avatar's dance. Additionally, different parts of a given song can vary significantly in musical characteristics, making more realistic dance animations require considering these variations.

[0016] Embodiments of the present disclosure improve the functionality of avatar animation systems by incorporating a danceability score generator configured to generate a danceability score for each segment of the song being played, which is further used to inform the animation system when generating real-time avatar dance animations. Danceability is defined as the quality or state of being able to be used for dancing. A song with high danceability is one that has musical characteristics that people can dance to in a fast-paced or energetic manner.

[0017] Musical characteristics can include, for example, rhythm, melody, harmony, timbre, dynamics, texture, and form. Using any musical characteristic or combination of these musical characteristics, the danceability and energy of the music can be determined and used to drive corresponding dance animations. For example, a song with high danceability may have high-paced animations, a song with low danceability may have very few animations, and finally, a song that does not have an associated dance (or "non-danceable song"), such as a song that only includes noise or talking, may not drive any animations.

[0018] Using a danceability score generator to improve avatar dance animations to correspond to the music being played will increase the use of 2D or 3D avatars in, for example, different platforms, various applications, augmented reality (AR), virtual reality (VR), video games, and messaging services, and further drive user engagement with these systems.

[0019] Networked computing environment

[0020] Figure 1 is a block diagram showing an example interactive system 100 for facilitating interactions on a network (e.g., exchanging text messages, making text, audio, and video calls, or playing games). The interactive system 100 includes a plurality of client systems 102, and each client system of the plurality of client systems 102 hosts a plurality of applications including an interactive client 104 and other applications 106. Each interactive client 104 is communicatively coupled via one or more communication networks including a network 108 (e.g., the Internet) to other instances of the interactive client 104 (e.g., hosted on corresponding other user systems 102), an interactive server system 110, and a third-party server 112. The interactive client 104 can also communicate with the locally hosted applications 106 using an application programming interface (API).

[0021] Each user system 102 can include a plurality of user devices, such as a mobile device 114, a head-mounted device 116, and a computer client device 118, which are communicatively connected to exchange data and messages.

[0022] The interactive client 104 interacts via the network 108 with other interactive clients 104 and with the interactive server system 110. The data exchanged between the interactive clients 104 (e.g., interaction 120) and between the interactive client 104 and the interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0023] The interactive server system 110 provides server-side functions to the interactive clients 104 via the network 108. Although certain functions of the interactive system 100 are described herein as being performed by the interactive client 104 or by the interactive server system 110, the location of certain functions within the interactive client 104 or within the interactive server system 110 can be a design choice. For example, it may be technically preferable to initially deploy a particular technology and function within the interactive server system 110, but later migrate the technology and function to the interactive client 104 where the user system 102 has sufficient processing power.

[0024] The interaction server system 110 supports various services and operations provided to the interaction client 104. Such operations include sending data to the interaction client 104, receiving data from the interaction client 104, and processing data generated by the interaction client 104. The data may include message content, client device information, geographical location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the interaction system 100 is activated and controlled through functions available via the user interface (UI) of the interaction client 104.

[0025] Turning now specifically to the interaction server system 110, the application programming interface (API) server 122 is coupled to and provides a programming interface for the interaction server 124, making the functionality of the interaction server 124 accessible to the interaction client 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to the database server 126 to facilitate access to the database 128, which stores data associated with interactions processed by the interaction server 124. Similarly, the web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. To this end, the web server 130 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0026] The application programming interface (API) server 122 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 124 and the client system 102 (and, for example, the interaction client 104 and other applications 106) as well as third-party servers 112. Specifically, the application programming interface (API) server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction client 104 and other applications 106 can call or query to activate the functionality of the interaction server 124. The application programming interface (API) server 122 exposes various functions supported by the interaction server 124, including: account registration; login functionality; sending interaction data from a specific interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting a collection of media data (e.g., a story); retrieving a friend list of a user of the user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) to and from an entity graph (e.g., a social graph); locating friends in the social graph; and opening application events (e.g., related to the interaction client 104).

[0027] The interaction server 124 hosts multiple systems and subsystems, which are described below with reference to Figure 2 and

[0028] Application of Link

[0029] Returning to the interactive client 104, the features and functions of an external resource (e.g., the linked application 106 or applet) are made available to the user via the interface of the interactive client 104. In this context, "external" refers to the fact that the application 106 or applet is external to the interactive client 104. External resources are typically provided by a third party, but can also be provided by the creator or provider of the interactive client 104. The interactive client 104 receives a user selection of an option for initiating or accessing the features of such an external resource. The external resource can be an application 106 (e.g., a "local app") installed on the user system 102, or a scaled-down version of an application (e.g., an "applet") hosted on or remote from the user system 102 (e.g., on a third-party server 112). The scaled-down version of the application includes a subset of the features and functions of the application (e.g., the full-scale local version of the application) and is implemented using a markup language document. In some examples, the scaled-down version of the application (e.g., an "applet") is a web-based markup language version of the application and is embedded within the interactive client 104. In addition to using a markup language document (e.g., a.*ml file), the applet can include a scripting language (e.g., a.*js file or a.json file) and a style sheet (e.g., a.*ss file).

[0030] In response to receiving a user selection of an option for initiating or accessing the features of an external resource, the interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 locally installed on the user system 102 can be launched independently of and separately from the interactive client 104, e.g., by selecting an icon corresponding to the application 106 on the home screen of the user system 102. A scaled-down version of such an application can be launched or accessed via the interactive client 104, and in some examples, no part or only a limited part of the scaled-down application can be accessed outside of the interactive client 104. The scaled-down application can be launched by receiving, e.g., a markup language document associated with the scaled-down application from a third-party server 112 via the interactive client 104 and processing such a document.

[0031] In response to determining that the external resource is a locally installed application 106, the interaction client 104 instructs the user system 102 to start the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client 104 communicates with a third-party server 112 (for example) to obtain a markup language document corresponding to the selected external resource. The interaction client 104 then processes the obtained markup language document to render the web-based external resource within the user interface of the interaction client 104.

[0032] The interaction client 104 may notify a user of the user system 102 or other users associated with such a user (e.g., “friends”) of an activity occurring in one or more external resources. For example, the interaction client 104 may provide a notification to participants in a conversation (e.g., a chat conversation) in the interaction client 104 regarding an external resource currently or recently used by one or more members of a user group. One or more users may be invited to join an active external resource or start an external resource that was recently used but is currently inactive (within a group of friends). The external resource may provide the ability to share items, conditions, statuses, or locations within the external resource with one or more members of the user group in a chat session to the participants in the conversation who each use a corresponding interaction client 104. The shared item may be an interactive chat card that members of the chat can interact with to, for example, start the corresponding external resource, view specific information within the external resource, or take the members of the chat to a specific location or status within the external resource. Within a given external resource, a response message may be sent to a user on the interaction client 104. The external resource may selectively include different media items in the response based on the current context of the external resource.

[0033] The interaction client 104 may present a list of available external resources (e.g., applications 106 or applets) to start or access a given external resource. The list may be presented in the form of a context-sensitive menu. For example, the icons representing different applications (or applets) of the application 106 (or applet) may vary based on how the menu is launched (e.g., from a conversation interface or from a non-conversation interface).

[0034] System Architecture

[0035] Figure 2 is a block diagram showing additional details regarding the interaction system 100 according to some examples. Specifically, the interaction system 100 is shown to include an interaction client 104 and an interaction server 124. The interaction system 100 includes multiple subsystems that are supported by the interaction client 104 on the client side and by the interaction server 124 on the server side.

[0036] The image processing system 202 provides various functions that enable a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0037] The camera device system 204 includes control software (e.g., in a camera device application) that interacts with and controls the hardware camera device of the user system 102 (e.g., directly or via an operating system) to modify and enhance real-time images captured and displayed via the interaction client 104.

[0038] The enhancement system 206 provides functions related to the generation and publication of enhancements (e.g., media overlays) for images captured in real time by the camera device of the user system 102 or retrieved from the memory of the user system 102. For example, the enhancement system 206 is operable to select, present, and display a media overlay (e.g., an image filter or an image lens) to the interaction client 104 for enhancing a real-time image received via the camera device system 204 or a stored image retrieved from the memory 702 of the user system 102. These enhancements are selected by the enhancement system 206 based on some input and data, such as:

[0039] · The geographical location of the user system 102; and

[0040] · The social network information of the user of the user system 102.

[0041] Enhancements can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at the user system 102 for transmission in a message, or to video content such as a video content stream or feed sent from the interaction client 104. Thus, the image processing system 202 can interact with and support various subsystems of the communication system 208, such as the messaging system 210 and the video communication system 212.

[0042] Media overlays can include text or image data that can be overlaid on a photo taken by user system 102 or a video stream produced by user system 102. In some examples, the media overlay can be a location overlay (e.g., Venice Beach), a live event name, or a business name overlay (e.g., Beach Café). In other examples, the image processing system 202 uses the geographical location of user system 102 to identify a media overlay that includes the name of a business at the geographical location of user system 102. The media overlay can include other markers associated with the business. The media overlay can be stored in database 128 and accessed via database server 126.

[0043] The image processing system 202 provides a user-based publishing platform that enables a user to select a geographical location on a map and upload content associated with the selected geographical location. The user can also specify the circumstances under which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographical location.

[0044] The enhancement creation system 214 supports an augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish enhancements (e.g., augmented reality experiences) for the interactive client 104. The enhancement creation system 214 provides a library of built-in features and tools for content creators, which includes, for example, custom shaders, tracking techniques, and templates.

[0045] In some examples, the enhancement creation system 214 provides a business-based publishing platform that enables a business to select a specific enhancement associated with a geographical location via a bidding process. For example, the enhancement creation system 214 associates the media overlay of the highest bidding business with the corresponding geographical location for a predefined amount of time.

[0046] Communication system 208 is responsible for enabling and handling various forms of communication and interaction within interactive system 100, and includes messaging system 210, audio communication system 216, and video communication system 212. Messaging system 210 is responsible for effectuating temporary or time-limited access to content by interactive client 104. Messaging system 210 includes multiple timers (e.g., within short-lived timer system 218) that selectively enable access (e.g., for presentation and display) to messages and associated content via interactive client 104 based on durations and display parameters associated with a message or a set of messages (e.g., a story). Additional details regarding the operation of short-lived timer system 218 are provided below. Audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0047] User management system 220 is operationally responsible for managing user data and profiles, and includes social networking system 222, which maintains information regarding relationships between users of interactive system 100.

[0048] Collection management system 224 is operationally responsible for managing collections or sets of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories". Such collections can be made available for a specified period of time (e.g., the duration of the event to which the content pertains). For example, content related to a concert can be available as a "story" for the duration of that concert. Collection management system 224 is also responsible for posting an icon that provides a notification of a particular collection to the user interface of interactive client 104. Collection management system 224 includes curation functions that enable collection managers to manage and curate particular collections of content. For example, a curation interface enables event organizers to curate a collection of content related to a particular event (e.g., delete inappropriate content or redundant messages). Additionally, collection management system 224 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such cases, collection management system 224 operates to automatically pay such users for the use of their content.

[0049] The map system 226 provides various geolocation functions and supports the presentation of map-based media content and messages by the interactive client 104. For example, the map system 226 enables the display on the map of user icons or avatars (e.g., stored in the profile data 302) to indicate the current or past locations of a user's "friends" within the context of the map, as well as the media content (e.g., a collection of messages including photos and videos) generated by these friends. For example, on the map interface of the interactive client 104, messages posted by a user from a specific geographical location to the interactive system 100 can be displayed to the "friends" of the specific user within the context of the map at that specific location. A user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), and the location and status information is similarly displayed to the selected users within the context of the map interface of the interactive client 104.

[0050] The game system 228 provides various game functions within the context of the interactive client 104. The interactive client 104 provides a game interface that presents a list of available games that can be launched by the user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a specific user to invite such other users to participate in playing a specific game by sending an invitation from the interactive client 104 to the other users. The interactive client 104 also supports voice, video, and text messaging (e.g., chat) within the context of playing a game, provides a leaderboard for the game, and also supports the provision of in-game rewards (e.g., game currency and items).

[0051] The external resource system 230 provides an interface for the interactive client 104 to communicate with remote servers (e.g., third-party servers 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts an application or a scaled-down version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on, for example, a markup language (e.g., HTML5). The interactive client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by the interactive server 124. The SDK includes application programming interfaces (APIs) with functions that can be called or activated by the web-based application. The interactive server 124 hosts a JavaScript library that provides access to the specific user data of the interactive client 104 for a given external resource. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0052] To integrate the functionality of the SDK into a web-based resource, the SDK is downloaded by a third-party server 112 from an interaction server 124, or the third-party server 112 receives the SDK in some other way. Once downloaded or received, the SDK is included as part of the application code of a web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interaction client 104 into the web-based resource.

[0053] The SDK stored on the interaction server system 110 effectively provides a bridge between an external resource (e.g., an application 106 or a mini-program) and the interaction client 104. This gives the user a seamless experience of communicating with other users on the interaction client 104 while also preserving the appearance of the interaction client 104. To bridge the communication between the external resource and the interaction client 104, the SDK facilitates the communication between the third-party server 112 and the interaction client 104. The WebView JavaScript Bridge running on the user system 102 establishes two one-way communication channels between the external resource and the interaction client 104. Messages are sent asynchronously between the external resource and the interaction client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0054] By using the SDK, not all information from the interaction client 104 is shared with the third-party server 112. The SDK restricts which information is shared based on the needs of the external resource. Each third-party server 112 provides an HTML5 file corresponding to the web-based external resource to the interaction server 124. The interaction server 124 can add a visual representation (e.g., a box design or other graphics) of the web-based external resource to the interaction client 104. Once the user selects the visual representation or instructs the interaction client 104 to access the features of the web-based external resource through the GUI of the interaction client 104, the interaction client 104 obtains the HTML5 file and instantiates the resources for accessing the features of the web-based external resource.

[0055] The interaction client 104 presents a graphical user interface for an external resource (e.g., a landing page or a title screen). During, before, or after presenting the landing page or the title screen, the interaction client 104 determines whether the launched external resource has been previously authorized to access the user data of the interaction client 104. In response to determining that the launched external resource has been previously authorized to access the user data of the interaction client 104, the interaction client 104 presents another graphical user interface of the external resource that includes the functions and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access the user data of the interaction client 104, after a display threshold time period (e.g., 3 seconds) of the landing page or the title screen of the external resource, the interaction client 104 slides up a menu (e.g., animates the menu to emerge from the bottom of the screen to the middle or other part of the screen) for authorizing the external resource to access the user data. The menu identifies the types of user data that the external resource will be authorized to use. In response to receiving a user selection of an accept option, the interaction client 104 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interaction client 104. The external resource is authorized by the interaction client 104 to access the user data under the OAuth 2 framework.

[0056] The interaction client 104 controls the types of user data shared with the external resource based on the type of the authorized external resource. For example, access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics) is provided to an external resource that includes a full-scale application (e.g., application 106). As another example, access to a second type of user data (e.g., payment information, a two-dimensional avatar of the user, a three-dimensional avatar of the user, and avatars with various avatar characteristics) is provided to an external resource that includes a scaled-down version of the application (e.g., a web-based version of the application). Avatar characteristics include different ways to customize the appearance of the avatar (e.g., different poses, facial features, clothing, etc.).

[0057] The avatar animation system 232 controls the animation to be generated in real time based on the received acoustic signals (e.g., songs, music, etc.). The avatar animation system 232 can generate dance animations for the avatar associated with the user of the user system 102. The avatar animation system 232 can also cause the avatar dance animation to be displayed by the user system 102 (or the client system 102).

[0058] Data Architecture

[0059] Figure 3FIG. is a schematic diagram showing a data structure 300 that can be stored in a database 304 of an interactive server system 110 according to certain examples. Although the contents of the database 304 are shown as including a plurality of tables, it should be understood that data can be stored in other types of data structures (e.g., object-oriented databases).

[0060] The database 304 includes message data stored within a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Details regarding other information that can be included in a message and that is included within the message data stored in the message table 306 are described below with reference to Figure 3 Describe other details regarding information that can be included in a message and that is included within the message data stored in the message table 306.

[0061] The entity table 308 stores entity data and is linked (e.g., by reference) to an entity graph 310 and profile data 302. Entities for which records are maintained within the entity table 308 can include individuals, corporate entities, organizations, objects, locations, events, etc. Any entity for which the interactive server system 110 stores data regarding it can be an identified entity, regardless of the entity type. Each entity is set with a unique identifier and an entity type identifier (not shown).

[0062] The entity graph 310 stores information regarding relationships and associations between entities. By way of example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as a personal user's subscription to digital content of a commercial or publishing user (e.g., a newspaper or other digital media channel or brand). Other relationships can be two-way, such as the "friend" relationship between personal users of the interactive system 100.

[0063] Certain permissions and relationships can be attached to each relationship and can also be attached to each direction of the relationship. For example, a two-way relationship (e.g., the friend relationship between personal users) can include authorization for the publication of digital content items between personal users, but certain restrictions or filters (e.g., based on content characteristics, location data, or time-of-day data) can be imposed on the publication of such digital content items. Similarly, the subscription relationship between a personal user and a commercial user can impose different degrees of restrictions on the publication of digital content from the commercial user to the personal user and can significantly limit or prevent the publication of digital content from the personal user to the commercial user. As an example of an entity, a particular user can record certain restrictions (e.g., through privacy settings) in the record for that entity within the entity table 308. Such privacy settings can apply to all types of relationships within the context of the interactive system 100 or can be selectively applied to certain types of relationships.

[0064] Profile data 302 stores multiple types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case where the entity is an individual, the profile data 302 includes, for example, a user name, a phone number, an address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. Then, a particular user can selectively include one or more of these avatar representations within the content of a message transmitted via the interaction system 100 and on a map interface displayed by the interaction client 104 to other users. The collection of avatar representations can include "status avatars" that present graphical representations of statuses or activities that the user can choose to transmit at a particular time.

[0065] In the case where the entity is a group, in addition to the group name, members, and various settings (e.g., notifications) for the relevant group, the profile data 302 for the group can similarly include one or more avatar representations associated with the group.

[0066] The database 304 also stores enhancement data, such as overlays or filters, in an enhancement table 312. The enhancement data is associated with videos (the data of the videos is stored in a video table 314) and images (the data of the images is stored in an image table 316) and is applied to the videos and images.

[0067] In some examples, a filter is an overlay that is displayed as being superimposed on an image or video during presentation to a receiving user. Filters can be of various types, including filters selected by a user from a set of filters presented by the interaction client 104 to a sending user while the sending user is composing a message. Other types of filters include geographic location filters (also known as geo-filters), which can be presented to a sending user based on a geographic location. For example, based on geographic location information determined by a global positioning system (GPS) unit of the user system 102, the interaction client 104 can present geographic location filters specific to nearby or special locations within the user interface.

[0068] Another type of filter is a data filter, which can be selectively presented to a sending user by the interaction client 104 based on other input or information collected by the user system 102 during a message creation process. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user system 102, or the current time.

[0069] Other enhancement data that can be stored within the image table 316 includes augmented reality content items (e.g., corresponding to app "lenses" or augmented reality experiences). Augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0070] The story table 318 stores data about a collection of messages and associated image, video, or audio data, where the messages and associated image, video, or audio data are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the interaction client 104 can include user-selectable icons to enable the sending user to add specific content to his or her personal story.

[0071] The collection can also constitute a "Live Story", which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" can constitute a curated stream of content submitted by users from different locations and events. Options to contribute content to a particular Live Story can be presented, for example, via the user interface of the interaction client 104 to users whose client devices have location services enabled and are at a common location event at a particular time. The interaction client 104 can identify Live Stories to a user based on the user's location. The end result is a "Live Story" told from a group perspective.

[0072] Another type of content collection is called a "Location Story", which enables users whose user system 102 is located within a particular geographical location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributing to a Location Story may employ secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0073] As mentioned above, the video table 314 stores video data, which in some examples is associated with messages for which records are maintained in the message table 306. Similarly, the image table 316 stores image data associated with messages whose message data is stored in the entity table 308. The entity table 308 can associate various enhancements from the enhancement table 312 with the various images and videos stored in the image table 316 and the video table 314.

[0074] The database 304 also includes an animation table 320 that stores animation data, which includes a danceability score generated by the danceability neural network 504, a test acoustic signal, an encoded test acoustic segment, a test quantization score for the test acoustic segment, and music features (e.g., frequency response, chromagram, rhythmogram, etc.). The animation data may also include a test video containing a dancer performing dance movements and the test acoustic signal, body postures determined using skeletal approximation, momentum scores associated with body parts, and a test danceability score.

[0075] Data communication architecture

[0076] Figure 4 is a schematic diagram showing the structure of a message 400 according to some examples. The message 400 is generated by the interactive client 104 for transmission via the interactive server 124 to another interactive client 104. The content of a particular message 400 is used to populate a message table 306 stored in a database 304 accessible by the interactive server 124. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data of the user system 102 or the interactive server 124. The message 400 is shown to include the following example components:

[0077] · Message identifier 402: A unique identifier that identifies the message 400.

[0078] · Message text payload 404: Text to be generated by the user via the user interface of the user system 102 and included in the message 400.

[0079] · Message image payload 406: Image data captured by a camera device component of the user system 102 or retrieved from a memory component of the user system 102 and included in the message 400. The image data for a sent or received message 400 may be stored in an image table 316.

[0080] · Message video payload 408: Video data captured by a camera device component or retrieved from a memory component of the user system 102 and included in the message 400. The video data for a sent or received message 400 may be stored in an image table 316.

[0081] · Message audio payload 410: Audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 400.

[0082] · Message enhancement data 412: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of message 400. Enhancement data for sent or received message 400 can be stored in enhancement table 312.

[0083] · Message duration parameter 414: A parameter value indicating, in seconds, the amount of time that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) is to be presented to or made accessible to the user via interactive client 104.

[0084] · Message geographic location parameter 416: Geographic location data (e.g., latitude coordinates and longitude coordinates) associated with the content payload of the message. Multiple message geographic location parameter 416 values can be included in the payload, each of these parameter values being associated with a content item included in the content (e.g., a specific image within message image payload 406 or a specific video within message video payload 408).

[0085] · Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "stories" identified in story table 318) associated with a specific content item within message image payload 406 of message 400. For example, identifier values can be used to associate each of multiple images within message image payload 406 with multiple content collections.

[0086] · Message tag 420: Each message 400 can be tagged with multiple tags, each of the multiple tags indicating a theme of the content included in the message payload. For example, in the case where a specific image included in message image payload 406 depicts an animal (e.g., a lion), a tag value can be included in message tag 420 indicating the relevant animal. Tag values can be generated manually based on user input or can be generated automatically using, for example, image recognition.

[0087] · Message sender identifier 422: An identifier (e.g., a message transceiver system identifier, an email address, or a device identifier) of the user of user system 102 on which message 400 was generated and from which message 400 was sent.

[0088] · Message recipient identifier 424: An identifier (e.g., a message transceiver system identifier, an email address, or a device identifier) of the user of user system 102 to which message 400 is addressed.

[0089] The content (e.g., value) of each component of message 400 can be a pointer to a location in a table where content data values are stored. For example, the image value in message image payload 406 can be a pointer to a location (or its address) within image table 316. Similarly, the values within message video payload 408 can point to data stored within image table 316, the values stored within message enhancement data 412 can point to data stored within enhancement table 312, the values stored within message story identifier 418 can point to data stored within story table 318, and the values stored within message sender identifier 422 and message receiver identifier 424 can point to user records stored within entity table 308.

[0090] Avatar animation system

[0091] Figure 5 Details of an avatar animation system 232 according to one embodiment are shown. The avatar animation system 232 controls animations to be generated in real time based on received acoustic signals (e.g., songs, music, etc.). The avatar animation system 232 includes a danceability score generator 502 and a dance animation controller 506.

[0092] The danceability score generator 502 includes a danceability neural network 504 to receive a real-time music stream (e.g., an acoustic signal). The real-time music stream can be a song or music that is playing in the background and captured by a microphone of one of the client systems 102. In another example, the real-time music stream is received in real time by the interaction server system 110.

[0093] The danceability neural network 504 generates a danceability score for each segment of the song being played. The dance animation controller 506 receives the danceability score and avatar characteristics to generate a dance animation for the avatar associated with the user of the user system 102. The avatar animation system 232 can also cause the avatar dance animation to be displayed by the user system 102 (or client system 102).

[0094] Since a danceability score is generated for each segment of the song and the dance animation controller 506 generates an avatar dance animation based on the danceability score of each segment, the avatar dance animation is a real-time dance animation that matches the real-time music stream.

[0095] Training the danceability neural network

[0096] Using videos of expert dancers dancing to music (e.g., songs, audio, acoustic signals), the danceability neural network 504 is a deep learning neural network that is trained to predict the danceability of songs that have not been heard.

[0097] In one example, training the danceability neural network 504 involves a processor training the danceability neural network 504 receiving a test acoustic signal, which may include all or part of a song, audio, sound, speech, etc. The test acoustic signal may also be an acoustic signal included in a test video of an expert dancer. The test acoustic signal includes a plurality of test acoustic segments, which are shorter parts (e.g., 3 seconds) of the test acoustic signal, for example. The danceability neural network 504 (or the processor) encodes the test acoustic segments to generate an internal representation of the test acoustic segments (e.g., latent variables in the danceability neural network 504).

[0098] Using the internal representation of the test acoustic segments, the danceability neural network 504 (or the processor) generates a test quantization score for each test acoustic segment in the test acoustic segments. The test quantization score is based on the musical features present in the test acoustic segments. Musical features include frequency response, chromagram, tempogram, or any combination thereof. For the encoded test acoustic segments, the test quantization score can be a score from 0 to 1, where 0 is low danceability and 1 is high danceability.

[0099] In one example, the danceability neural network 504 can be a transformer network that performs sequence-to-sequence modeling. In one example, the danceability neural network 504 transforms a sequence of musical features into a sequence of danceability scores (or test quantization scores, or test danceability scores). It should be understood that the danceability neural network 504 can be implemented using any type of neural network (e.g., perceptron, feedforward neural network, multi-layer perceptron, convolutional neural network, radial basis function neural network, recurrent neural network, LSTM - long short-term memory, sequence-to-sequence model, or modular neural network) or a combination thereof.

[0100] Training the danceability neural network 504 also involves receiving a test video that includes a dancer performing dance movements and the test acoustic signal. The test video includes test video segments, which can be shorter parts (e.g., 3 seconds) of the test video, for example. The test video segments also include a plurality of test video frames. For example, a test video segment can include 10 test video frames.

[0101] The processor determines the body pose for each test video frame in the test video using the dancer's skeletal approximation. The skeletal approximation can be a two-dimensional or three-dimensional skeletal approximation. For each test video segment in the test video clip, the processor generates a momentum score associated with the dancer's body part. The momentum score represents the energy or effort exerted by the dancer on that body part in the video frames of that test video segment. Then, the processor generates a test danceability score for each test video segment in the test video clip based on the momentum scores. And the momentum scores are for each body part (such as hips, fingers, knees, etc.). The test danceability score is a score assigned to the dancer's entire body or skeleton. The test danceability score of a test video segment can be the average or sum of the momentum scores in that test video segment. The test danceability score of a test video segment can also be the weighted sum of the momentum scores in that test video segment. For example, a higher weight can be assigned to the momentum score of the body core, and a lower weight can be assigned to the momentum score of the fingers. In one example, higher weights are assigned to the momentum scores of larger body parts, and lower weights are assigned to the momentum scores of smaller body parts. In this example, a higher energy or effort is reflected in a higher momentum score, which corresponds to a higher danceability of that test video segment. Thus, based on the movement or motion of each body part in the dancer's body parts, the danceability neural network 504 generates a test danceability score for each test video segment in the test video clip.

[0102] For each test video segment in the test video clip, the danceability neural network 504 (or the processor) associates the test danceability score with the test quantization score of the test acoustic segment. In one example, the test video segment corresponds in time to the test acoustic segment in the test video, such that the danceability neural network 504 is trained to generate a danceability score based on the music or song in the test video (e.g., the test quantization score) and the danceability score of the skeleton (e.g., the test danceability score).

[0103] Process of generating real-time avatar animation using the danceability score

[0104] Although the described flowcharts may show the operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of the operations can be rearranged. The process terminates when its operations are completed. The process can correspond to a method, procedure, algorithm, etc. The operations of the method can be performed in whole or in part, can be combined with some or all of the operations in other methods, and can be performed by any number of different systems (e.g., the systems described herein) or any part thereof (e.g., a processor included in any of these systems).

[0105] Figure 6Processing 600 for generating real-time avatar animation using a danceability score according to an example is shown. In one example, the processor in the avatar animation system 232, the processor in the client system 102, the processor in the interaction server system 110, or any combination thereof may perform the operations in processing 600.

[0106] In operation 602, the processor receives a real-time acoustic signal including a plurality of acoustic segments.

[0107] In operation 604, the processor generates a danceability score for each acoustic segment in the acoustic segments using a danceability neural network. The processor may generate a danceability score for each acoustic segment in the acoustic segments using the danceability neural network based on an associated test danceability score and a test quantization score.

[0108] In operation 606, the processor generates real-time animations of a first avatar and a second avatar based on the danceability score and avatar characteristics associated with the first avatar and the second avatar. The avatar characteristics of the first user are associated with the first avatar, and the avatar characteristics of the second user are associated with the second avatar.

[0109] In operation 608, the processor causes the real-time animations of the first avatar and the second avatar to be displayed on a first client device. The processor may also generate a real-time animation based on the position of the first avatar displayed on the first client device and the position of the second avatar displayed on the first client device to prevent overlapping display of the first avatar and the second avatar. In one example, the processor also causes the real-time animations of the first avatar and the second avatar to be displayed on a second client device.

[0110] In one example, training the danceability neural network 504 by the processor includes receiving a plurality of test acoustic signals including a plurality of test acoustic segments. The processor encodes the plurality of test acoustic segments and generates a test quantization score for each test acoustic segment in the test acoustic segments. The test quantization score is based on music features such as frequency response, chromagram, rhythmogram, or any combination thereof.

[0111] The processor may also train the danceability neural network 504 by receiving a plurality of test videos including a dancer performing dance movements and a test acoustic signal. The test videos include a plurality of test video segments, and each test video segment in the test video segments includes a plurality of test video frames. The processor may determine a body pose for each test video frame in the test video frames using a skeletal approximation of the dancer. For each test video segment in the test video segments, the processor generates a plurality of momentum scores associated with a plurality of body parts of the dancer and generates a test danceability score for each test video segment in the test video segments based on the momentum scores. The test danceability score may be generated based on a weighted average of the momentum scores.

[0112] In one example, for each test video segment in a test video, the processor correlates a test danceability score with a test quantization score of a test acoustic segment. The test video segment corresponds in time to the test acoustic segment in the test video.

[0113] The improved avatar dance animation is presented to correspond to the music being played in real time to show a more realistic reaction of the avatar to the music. Thus, embodiments of the present disclosure improve the functionality of the avatar animation system by combining the danceability scores of each segment of the song being played, which are further used to inform the animation system when generating real-time avatar dance animations. This improvement will further increase the use of the avatar and drive user engagement on each platform in the platform where the avatar is displayed.

[0114] System with a head-mounted device

[0115] Figure 7 System 700 is shown, and system 700 includes a head-mounted device 116 according to some examples. Figure 7 is a high-level functional block diagram of an example head-mounted device 116 communicatively coupled to a mobile device 114 and various server systems 704 (e.g., an interaction server system 110) via various networks 108.

[0116] The head-mounted device 116 includes one or more camera devices, and each of the one or more camera devices can be, for example, a visible light camera device 706, an infrared emitter 708, and an infrared camera device 710.

[0117] The mobile device 114 is connected to the head-mounted device 116 using both a low-power wireless connection 712 and a high-speed wireless connection 714. The mobile device 114 is also connected to the server system 704 and the network 716.

[0118] The head-mounted device 116 also includes two image displays in an image display 718 of an optical component. The two image displays 718 of the optical component include an image display associated with the left lateral side of the head-mounted device 116 and an image display associated with the right lateral side of the head-mounted device 116. The head-mounted device 116 also includes an image display driver 720, an image processor 722, a low-power circuit 724, and a high-speed circuit 726. The image displays 718 of the optical component are used to present images and videos to the user of the head-mounted device 116, including images that may include a graphical user interface.

[0119] The image display driver 720 commands and controls the image display 718 of the optical component. The image display driver 720 can deliver image data directly to the image display 718 of the optical component for presentation or can convert the image data into a signal or data format suitable for delivery to an image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, etc., while the still image data can be formatted according to a compression format such as Portable Network Graphics (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (Exif), etc.

[0120] The head-mounted device 116 includes a frame and a stem (or temple) extending from a lateral side of the frame. The head-mounted device 116 also includes a user input device 728 (e.g., a touch sensor or a push button) including an input surface on the head-mounted device 116. The user input device 728 (e.g., a touch sensor or a push button) is used to receive input selections from a user for manipulating a graphical user interface of the presented image.

[0121] Figure 7 The components of the head-mounted device 116 shown in are located on one or more circuit boards (e.g., a PCB or a flexible PCB) in the rim or the temples. Alternatively or additionally, the depicted components can be located in chunks, frames, hinges, or nose bridges of the head-mounted device 116. The left visible light camera device and the right visible light camera device 706 can include digital camera device elements, e.g., complementary metal oxide semiconductor (CMOS) image sensors, charge-coupled devices, camera device lenses, or any other corresponding visible light or light capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0122] The head-mounted device 116 includes a memory 702 that stores instructions for performing a subset or all of the functions described herein. The memory 702 can also include a storage device.

[0123] As Figure 7As shown, the high-speed circuit 726 includes a high-speed processor 730, a memory 702, and a high-speed wireless circuit 732. In some examples, the image display driver 720 is coupled to the high-speed circuit 726 and is operated by the high-speed processor 730 to drive the left and right image displays in the image display 718 of the optical component. The high-speed processor 730 can be any processor capable of managing the high-speed communication and operation of any general computing system required for the head-mounted device 116. The high-speed processor 730 includes the processing resources required to manage high-speed data transmission over the high-speed wireless connection 714 to a wireless local area network (WLAN) using the high-speed wireless circuit 732. In certain examples, the high-speed processor 730 executes the operating system of the head-mounted device 116 (e.g., the LINUX operating system) or other such operating systems, and the operating system is stored in the memory 702 for execution. In addition to any other duties, the high-speed processor 730 that executes the software architecture of the head-mounted device 116 manages data transmission with the high-speed wireless circuit 732. In certain examples, the high-speed wireless circuit 732 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as WiFi. In some examples, the high-speed wireless circuit 732 may implement other high-speed communication standards.

[0124] The low-power wireless circuit 734 and the high-speed wireless circuit 732 of the head-mounted device 116 may include a short-range transceiver (Bluetooth TM ) and a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or WiFi). The mobile device 114, including transceivers that communicate via the low-power wireless connection 712 and the high-speed wireless connection 714, may be implemented using the details of the architecture of the head-mounted device 116, and so may the other elements of the network 716.

[0125] Memory 702 includes any storage device capable of storing various data and applications, the data including camera device data generated by the left visible light camera device and the right visible light camera device 706, the infrared camera device 710, and the image processor 722, and images generated for display on the image display of the optical component via the image display driver 720. Although memory 702 is shown integrated with the high-speed circuitry 726, in some examples, memory 702 may be a separate stand-alone component of the head-mounted device 116. In certain such examples, electrical wiring may provide a connection from the image processor 722 or the low-power processor 736 to memory 702 through a chip including the high-speed processor 730. In some examples, the high-speed processor 730 may manage the addressing of memory 702 such that the low-power processor 736 will initiate the high-speed processor 730 whenever a read or write operation involving memory 702 is needed.

[0126] As Figure 7 shown, the low-power processor 736 or the high-speed processor 730 of the head-mounted device 116 may be coupled to the camera devices (visible light camera device 706, infrared emitter 708, or infrared camera device 710), the image display driver 720, the user input device 728 (e.g., touch sensor or push button), and the memory 702.

[0127] The head-mounted device 116 is connected to a host computer. For example, the head-mounted device 116 is paired with the mobile device 114 via the high-speed wireless connection 714 or connected to the server system 704 via the network 716. The server system 704 may be one or more computing devices that are part of a service or network computing system, e.g., including a processor, a memory, and a network communication interface to communicate with the mobile device 114 and the head-mounted device 116 via the network 716.

[0128] The mobile device 114 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via the network 716, the low-power wireless connection 712, or the high-speed wireless connection 714. The mobile device 114 may also store at least part of the instructions for generating binaural audio content in the memory of the mobile device 114 to implement the functions described herein.

[0129] The output components of the head-mounted device 116 include visual components such as a display (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light-emitting diode (LED) display, a projector, or a waveguide). The image display of the optical component is driven by an image display driver 720. The output components of the head-mounted device 116 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, and the like. The input components (e.g., user input device 728) of the head-mounted device 116, the mobile device 114, and the server system 704 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., physical buttons, a touch screen that provides the position and force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0130] The head-mounted device 116 may also include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-mounted device 116. For example, the peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such elements described herein.

[0131] For example, biometric components include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. Motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes), and the like. Positioning components include position sensor components for generating position coordinates (e.g., a global positioning system (GPS) receiver component), Wi-Fi or Bluetooth TM transceivers for generating positioning system coordinates, altitude sensor components (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), orientation sensor components (e.g., magnetometers), and the like. Such positioning system coordinates may also be received from the mobile device 114 via a low-power wireless circuit 734 or a high-speed wireless circuit 732 through a low-power wireless connection 712 and a high-speed wireless connection 714.

[0132] Machine architecture

[0133] Figure 8is an illustrative representation of a machine 800 within which instructions 802 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 800 to perform any one or more of the methods discussed herein. For example, the instructions 802 can cause the machine 800 to perform any one or more of the methods described herein. The instructions 802 transform a general, unprogrammed machine 800 into a particular machine 800 programmed to perform the described and illustrated functions in the described manner. The machine 800 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 can include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 802 specifying actions to be taken by the machine 800. Further, although only a single machine 800 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 802 to perform any one or more of the methods discussed herein. For example, the machine 800 can include the user system 102 or any one of the multiple server devices that form part of the interactive server system 110. In some examples, the machine 800 can also include both a client system and a server system, where certain operations of a particular method or algorithm are executed on the server side and certain operations of the particular method or algorithm are executed on the client side.

[0134] The machine 800 can include a processor 804, a memory 806, and an input / output I / O component 808 that can be configured to communicate with each other via a bus 810. In an example, the processor 804 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 812 and a processor 814 that execute the instructions 802. The term "processor" is intended to include multi-core processors that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. AlthoughFigure 8 A number of processors 804 are shown, but machine 800 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0135] Memory 806 includes main memory 816, static memory 818, and storage unit 820, all of which are accessible by processor 804 via bus 810. Main memory 806, static memory 818, and storage unit 820 store instructions 802 that implement any one or more of the methods or functions described herein. The instructions 802 may also reside, completely or partially, within machine-readable medium 822 within main memory 816, static memory 818, storage unit 820, within at least one of the processors 804 (e.g., within a cache memory of the processor), or any suitable combination thereof, during execution by machine 800.

[0136] I / O components 808 can include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 808 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will unlikely include such a touch input device. It should be appreciated that I / O components 808 can include Figure 8 many other components not shown in the figure. In various examples, I / O components 808 can include user output components 824 and user input components 826. User output components 824 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. User input components 826 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), haptic input components (e.g., a physical button, a touch screen that provides the location and force of a touch or touch gesture, or other haptic input components), audio input components (e.g., a microphone), etc.

[0137] In other examples, I / O component 808 can include biometric component 828, motion component 830, environmental component 832, or location component 834, as well as a wide array of other components. For example, biometric component 828 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so on. Motion component 830 includes acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes).

[0138] Environmental component 832 includes, for example, one or more camera devices (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety or for measuring pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment.

[0139] Regarding the camera device, user system 102 can have a camera device system that includes, for example, a front camera device on the front surface of user system 102 and a rear camera device on the rear surface of user system 102. The front camera device can be used, for example, to capture still images and videos of the user of user system 102 (e.g., “selfies”), which can then be enhanced with the aforementioned enhancement data (e.g., filters). The rear camera device can be used, for example, to capture still images and videos in a more conventional camera device mode, and these images are similarly enhanced with the enhancement data. In addition to the front camera device and the rear camera device, user system 102 can also include a 360° camera device for capturing 360° photos and videos.

[0140] Furthermore, the camera device system of user system 102 can include a dual rear camera device (e.g., a main camera device and a depth sensing camera device), or even a triple, quadruple, or quintuple rear camera device configuration on the front and rear sides of user system 102. For example, these multi-camera device systems can include wide-angle camera devices, ultra-wide-angle camera devices, telephoto camera devices, macro camera devices, and depth sensors.

[0141] The positioning component 834 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0142] Various techniques can be used to implement communication. The I / O component 808 also includes a communication component 836 that is operable to couple the machine 800 to a network 838 or a device 840 via corresponding couplings or connections. For example, the communication component 836 can include a network interface component that interfaces with the network 838 or other suitable devices. In other examples, the communication component 836 can include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power consumption), components and other communication components for providing communication via other forms. The device 840 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0143] In addition, the communication component 836 can detect an identifier or include components operable to detect an identifier. For example, the communication component 836 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as universal product code (UPC) barcodes, and multi-dimensional barcodes such as quick response (QR) codes, Aztec codes, data matrix, DataGlyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D barcodes, and other optical codes, as well as other optical sensors), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). Additionally, various information can be obtained via the communication component 836, such as a location via Internet protocol (IP) geolocation, a location via signal triangulation, a location obtained via detecting an NFC beacon signal that can indicate a specific location, and the like.

[0144] Various memories (e.g., main memory 816, static memory 818, and the memory of the processor 804) and the storage unit 820 can store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 802), when executed by the processor 804, cause various operations to implement the disclosed examples.

[0145] Instructions 802 can be sent or received over network 838 via a network interface device (e.g., the network interface component included in communication component 836) using a transmission medium and any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 802 can be sent or received using a transmission medium via a coupling (e.g., a peer-to-peer coupling) to device 840.

[0146] Software architecture

[0147] Figure 9 is a block diagram 900 showing a software architecture 902 that can be installed on any one or more of the devices described herein. The software architecture 902 is supported by hardware, such as a machine 904 that includes a processor 906, a memory 908, and I / O components 910. In this example, the software architecture 902 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 902 includes the following layers, such as an operating system 912, libraries 914, frameworks 916, and applications 918. In operation, an application 918 activates an API call 920 through the software stack and receives a message 922 in response to the API call 920.

[0148] The operating system 912 manages hardware resources and provides common services. The operating system 912 includes, for example: a kernel 924, services 926, and drivers 928. The kernel 924 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 924 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. The services 926 can provide other common services for other software layers. The drivers 928 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 928 can include a display driver, a camera device driver, or a low-power driver, a flash driver, a serial communication driver (e.g., a USB driver), drivers, an audio driver, a power management driver, etc.

[0149] Library 914 provides common low-level infrastructure used by application 918. Library 914 may include system libraries 930 (e.g., C standard library), and system libraries 930 provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, library 914 may include API libraries 932, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), High Efficiency Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., OpenGL framework for 2D and 3D presentation in graphical content on a display), database libraries (e.g., SQLite that provides various relational database functions), web libraries (e.g., WebKit that provides web browsing functions), etc. Library 914 may also include a variety of other libraries 934 to provide many other APIs to application 918.

[0150] Framework 916 provides common high-level infrastructure used by application 918. For example, framework 916 provides various Graphical User Interface (GUI) functions, high-level resource management, and high-level location services. Framework 916 may provide a wide range of other APIs that can be used by application 918, and some of these APIs may be specific to a particular operating system or platform.

[0151] In an example, application 918 may include a home application 936, a contacts application 938, a browser application 940, a book reader application 942, a location application 944, a media application 946, a messaging application 948, a gaming application 950, and a variety of other applications such as third-party applications 952. Application 918 is a program that executes functions defined in the program. One or more applications in application 918 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C language or assembly language). In a specific example, a third-party application 952 (e.g., an application developed using an ANDROID TM or IOS TM Software Development Kit (SDK)) can be an application on platforms such as IOS TM 、ANDROID TM 、 Mobile software running on the mobile operating system of a Phone or other mobile operating systems. In this example, a third-party application 952 can activate an API call 920 provided by the operating system 912 to facilitate the functions described herein.

[0152] Glossary

[0153] "Carrier signal" means any intangible medium that can store, encode, or carry instructions executed by a machine and includes digital or analog communication signals, or other intangible media that facilitate the transmission of such instructions. Instructions can be sent or received over a network via a network interface device using a transmission medium.

[0154] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistant (PDA), smartphone, tablet computer, ultrabook, netbook, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user can use to access a network.

[0155] "Communication network" means one or more portions of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a network, other types of networks, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless network or a cellular network, and the coupling can be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling can implement any data transfer technology among various types of data transfer technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, the 4th Generation Wireless (4G) network, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other data transfer technologies defined by various standards-setting organizations, other long-distance protocols, or other data transfer technologies.

[0156] "Component" refers to, for example, a logical or physical entity having boundaries defined by, for example, function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization of a particular processing or control function. A component can be combined with other components via its interfaces to perform machine processing. A component can be a packaged functional hardware unit designed to work with other components, and can also be part of a program with specific functions that generally perform related functions. A component can be a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or a part of an application) to operate to perform certain operations as described herein as a hardware component. A hardware component can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) that is uniquely customized to perform the configured function and is no longer a general purpose processor. It will be appreciated that the decision of whether to implement a hardware component mechanically in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry can be made for cost and time considerations. Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity physically constructed, permanently configured (e.g., hard-wired) or temporarily configured (e.g., programmed) to operate in some manner or to perform certain operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate every hardware component at any given time. For example, in a case where a hardware component includes a general purpose processor that is configured by software to become a dedicated processor, the general purpose processor can be configured at different times to be respective different dedicated processors (e.g., including different hardware components). The software accordingly configures a particular one or more processors to, for example, constitute a particular hardware component at one moment and different hardware components at different moments. A hardware component can provide information to and receive information from other hardware components. Thus, the described hardware components can be considered to be communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving the information from the memory structure. For example, one hardware component can perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Then, another hardware component can access the memory device at a subsequent time to retrieve the stored output and process it. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors temporarily configured (e.g., via software) or permanently configured to perform the associated operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least in part processor-implemented, where one or more particular processors are examples of hardware. For example, at least some of the various operations of the method can be performed by one or more processors or processor-implemented components. Additionally, one or more processors can also operate to support the execution of associated operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some of the operations can be performed by a group of computers (as an example of a machine including processors), where the operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of certain operations can be distributed among processors, not residing only within a single machine but deployed across multiple machines. In some examples, the processor or processor-implemented components can be located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processor or processor-implemented components can be distributed across multiple geographical locations.

[0157] "Computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium", and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0158] "Ephemeral message" means a message that is accessible, for example, within a time-limited duration. The ephemeral message can be text, image, video, etc. The access time of the ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0159] "Machine storage medium" means, for example, a single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be regarded as including, but not limited to, solid-state memory as well as optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and can be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are subsumed under the term "signal medium".

[0160] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions executable by a machine.

[0161] "Signal medium" means, for example, any intangible medium capable of storing, encoding, or carrying instructions executable by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be regarded as including any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose one or more characteristics are set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

[0162] "User equipment" means, for example, equipment that a user accesses, controls, or owns and with which the user interacts to perform an action or interact with other users or computer systems.

Claims

1. A method, comprising: receiving, by a processor, a real-time acoustic signal including a plurality of acoustic segments; generating, for each of the acoustic segments in the acoustic signal, a danceability score using a danceability neural network; generating real-time animations of a first avatar and a second avatar based on the danceability score and avatar characteristics associated with the first avatar and the second avatar; and causing the real-time animations of the first avatar and the second avatar to be displayed on a first client device.

2. The method according to claim 1, further comprising: training the danceability neural network, wherein training the danceability neural network includes: receiving a plurality of test acoustic signals including a plurality of test acoustic segments; encoding the plurality of test acoustic segments; and generating, for each of the test acoustic segments, a test quantization score, wherein the test quantization score is based on music features.

3. The method according to claim 2, wherein the music features include frequency response, chromagram, rhythmogram, or any combination thereof.

4. The method according to claim 2, wherein training the danceability neural network further includes: receiving a plurality of test videos including a dancer performing dance movements and the test acoustic signals, the test videos including a plurality of test video segments, wherein each of the test video segments includes a plurality of test video frames; determining body postures for each of the test video frames using a skeletal approximation of the dancer; generating, for each of the plurality of test video segments, a plurality of momentum scores associated with a plurality of body parts of the dancer; and generating, based on the momentum scores, a test danceability score for each of the test video segments.

5. The method according to claim 4, wherein generating the test danceability score includes generating a weighted average of the momentum scores.

6. The method according to claim 4, further comprising: for each of the test video segments, associating the test danceability score with the test quantization score of the test acoustic segment, wherein the test video segment corresponds in time to the test acoustic segment in the test video.

7. The method according to claim 6, wherein generating, using the danceability neural network, the danceability score for each of the acoustic segments further includes: generating the danceability score for each of the acoustic segments based on the associated test danceability score and test quantization score.

8. The method according to claim 1, wherein generating the real-time animations of the first avatar and the second avatar further includes: generating the real-time animations based on the position of the first avatar displayed on the first client device and the position of the second avatar displayed on the first client device to prevent overlapping display of the first avatar and the second avatar.

9. The method according to claim 1, further comprising: causing the real-time animations of the first avatar and the second avatar to be displayed on a second client device.

10. The method according to claim 9, wherein, the first client device is associated with a first user, and the second client device is associated with a second user, wherein the first user is associated with the first avatar, and the second user is associated with the second avatar.

11. A system, comprising: a processor; and a memory storing instructions that, when executed by the processor, cause the system to perform operations, the operations including: receiving a real-time acoustic signal including a plurality of acoustic segments; generating a danceability score for each of the acoustic segments using a danceability neural network; generating real-time animations of the first avatar and the second avatar based on the danceability scores and avatar characteristics associated with the first avatar and the second avatar; and causing the real-time animations of the first avatar and the second avatar to be displayed on a first client device.

12. The system according to claim 11, wherein, causing the system to perform operations further includes: training the danceability neural network, wherein training the danceability neural network includes: receiving a plurality of test acoustic signals including a plurality of test acoustic segments; encoding the plurality of test acoustic segments; and generating a test quantization score for each of the test acoustic segments, wherein the test quantization score is based on music features.

13. The system according to claim 12, wherein, the music features include frequency response, chromagram, rhythmogram, or any combination thereof.

14. The system according to claim 12, wherein, training the danceability neural network further includes: receiving a plurality of test videos including a dancer performing dance movements and the test acoustic signals, the test videos including a plurality of test video segments, wherein each of the test video segments includes a plurality of test video frames; determining body postures for each of the test video frames using the skeletal approximation of the dancer; generating a plurality of momentum scores associated with a plurality of body parts of the dancer for each of the plurality of test video segments; and generating a test danceability score for each of the test video segments based on the momentum scores.

15. The system according to claim 14, wherein, generating the test danceability score includes generating a weighted average of the momentum scores.

16. The system according to claim 14, wherein, causing the system to perform operations further includes: for each of the test video segments, associating the test danceability score with the test quantization score of the test acoustic segment, wherein the test video segment corresponds in time to the test acoustic segment in the test video.

17. The system according to claim 16, wherein, generating the danceability score for each of the acoustic segments using the danceability neural network further includes: Generate the danceability score for each acoustic segment in the acoustic segment based on the associated test danceability score and test quantization score.

18. The system according to claim 11, wherein, generating the real-time animations of the first avatar and the second avatar further includes: generating the real-time animations based on the position of the first avatar displayed on the first client device and the position of the second avatar displayed on the first client device to prevent overlapping display of the first avatar and the second avatar.

19. The system according to claim 11, wherein, causing the system to perform operations further includes: displaying the real-time animations of the first avatar and the second avatar on a second client device, wherein the first client device is associated with the first user, and the second client device is associated with the second user, wherein the first user is associated with the first avatar, and the second user is associated with the second avatar.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform operations, the operations including: receiving a real-time acoustic signal including a plurality of acoustic segments; generating a danceability score for each acoustic segment in the acoustic segment using a danceability neural network; generating real-time animations of the first avatar and the second avatar based on the danceability score and avatar features associated with the first avatar and the second avatar; and displaying the real-time animations of the first avatar and the second avatar on a first client device.

Images