Methods, apparatuses, and media for generating and displaying animations

Abstract

Methods, devices, media, and other implementations are described for generating, modifying, and outputting pseudo-random animations that can be synchronized with audio data. In one implementation, a computer animation model comprising one or more control points is accessed by one or more processors, the one or more processors associate a motion pattern with a first control point of the one or more control points and associate one or more velocity harmonics with the first control point. A set of motion states is identified using the motion pattern for a probabilistic combination, and a probability value is assigned to each motion state in the set of motion states. The probability values can be used to probabilistically determine a particular motion state as part of a display animation of the computer animation model.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application Serial No. 16 / 588,446, filed September 30, 2019; U.S. Patent Application Serial No. 16 / 588,412, filed September 30, 2019; U.S. Patent Application Serial No. 16 / 588,373, filed September 30, 2019; and U.S. Patent Application Serial No. 16 / 588,329, filed September 30, 2019, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of this disclosure generally relate to computer animation and graphical user interfaces (GUIs), including generating and using animated structures within a messaging system capable of accessing audio data. Embodiments of this disclosure relate to automated dance animation. Background Technology

[0004] Computer animation involves adding movement to structures within a computer model output to a device's display. Augmented reality is the display of the physical world and / or physical objects within it, overlaid with computer-generated perceptual information (e.g., an animated computer model). The overlaid information can be constructive (added to the display) and / or destructive (masked from the display). In either case, the computer-generated perceptual information can be animated to modify the information presented on the device's display. Attached Figure Description

[0005] To facilitate identification of any discussion of a particular element or action, one or more of the highest significant digits in the figure labels refer to the figure number in which the element was first introduced.

[0006] Figure 1 This is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and associated content) over a network according to some implementations. The messaging system may include models and data for animation.

[0007] Figure 2 This is a block diagram illustrating further details of a messaging system having elements for creating and implementing animations, according to an example implementation.

[0008] Figure 3 This is a block diagram illustrating further details of a messaging system having elements for creating and implementing animations, according to an example implementation.

[0009] Figure 4A It is an interface diagram depicting aspects of display and image data that can be animated according to certain example implementations.

[0010] Figure 4B A device display showing aspects of an overlay that can be implemented to generate animation, according to some embodiments.

[0011] Figure 4C Aspects of a system for generating and displaying animations, according to some embodiments, are shown.

[0012] Figure 4D Aspects of a system for generating and displaying animations, according to some embodiments, are shown.

[0013] Figure 4E Aspects of a system for generating and displaying animations, according to some embodiments, are shown.

[0014] Figure 5A This illustrates aspects of audio data that can be used with systems for generating and displaying animations, according to some implementations.

[0015] Figure 5B This illustrates aspects of audio data that can be used with systems for generating and displaying animations, according to some implementations.

[0016] Figure 6A This illustrates aspects of motion patterns that, according to some implementations, can be used as part of an animation state space.

[0017] Figure 6B Aspects of a system for generating and displaying animations, according to some embodiments, are shown.

[0018] Figure 6C Aspects of a system for generating and displaying animations, according to some embodiments, are shown.

[0019] Figure 7A Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0020] Figure 7B Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0021] Figure 7C This illustrates aspects of motion patterns that, according to some implementations, can be used as part of an animation state space.

[0022] Figure 7D Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0023] Figure 7EAspects of a system for generating and displaying animations, according to some embodiments, are shown.

[0024] Figure 8 Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0025] Figure 9A Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0026] Figure 9B Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0027] Figure 9C Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations.

[0028] Figure 10 Example methods according to some implementations described herein are shown.

[0029] Figure 11 Example methods according to some implementations described herein are shown.

[0030] Figure 12 Example methods according to some implementations described herein are shown.

[0031] Figure 13 Example methods according to some implementations described herein are shown.

[0032] Figure 14 This is a block diagram illustrating a representative software architecture that can be used in conjunction with the various hardware architectures described herein to implement various implementations.

[0033] Figure 15 This is a block diagram illustrating components of a machine, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any or more of the methods discussed herein. Detailed Implementation

[0034] Systems, methods, user interfaces, instructions stored in a medium, computing devices, and various other implementations associated with configuring and generating animations are described. In particular, some implementations include a structure for identifying the possible animations of a model. The model's state space can be described by the possible movements of independent control points within the model and the animation velocity of each action and each control point. Each element of the state space can have a probability assigned to configure pseudo-random animations, where the probability of certain movements is configurable. Furthermore, some implementations include aspects of matching pseudo-random animations with audio signals and audio thresholds used to initiate animations, as well as systems for matching the pseudo-random motions of a computer model with characteristics of the audio signals.

[0035] For example, a designer can use animation or overlay creation tools to generate a penguin model with certain control points in the model. The designer can choose certain movements of the control points mapped to certain audio characteristics, such as those determined by the designer, while preserving the random element of the animated model's movement. Control points can have all possible movements assigned to describe all possible movements of all parts of the model. The designer can then assign a probability to each movement, such that the likelihood of certain movements occurring during the displayed animation is set probabilistically by the designer. Movements that the designer does not want to occur are assigned a probability of zero or removed from the state space. In addition to selecting a movement for each control point, the display velocity can also be selected as a harmonic of the audio rhythm to allow pseudo-random movement to automatically match the rhythm of the music. In other implementations, other characteristics can be set as part of the model's state space. When the computer animation runs on the device, the designer's choice of probabilities in the model's state space affects the animated movement of the model. By matching actions to the rhythm of the audio stream, a computer model running on the device can display animations with random elements that match audio data on or around the device. This can create "dancing" animations, which include randomness in the selection of a large number of possible movements, but which are filtered by the designer from all possible movements to emphasize the movement that illustrates the characteristics chosen by the designer.

[0036] In addition to pseudo-random animations that can match audio, various properties can be used as thresholds for when to execute certain animations. For example, certain beat elements can be used to trigger specific sets of probabilities, and different beat elements or the absence of different beats can be used to select a default animation or different sets of probabilities for the state space of all possible motions of the model.

[0037] Such a system can be integrated with a messaging system to automatically analyze message data to apply a model to image data that is part of the system message. The recipient of such a message can then display an image with pseudo-random motion applied to the model within the message. Audio data detected at the recipient's device can influence the displayed animation. For example, an image of the sender's face can be sent along with a model applied to cause animation of facial features in the image, such as hair, ears, eyebrows, eyes, etc. The animation displayed at the recipient's device will use probabilities applied by the designer, along with audio from the recipient's device, to create an actual animation output at the recipient's device. Other examples may use a 3D model applied to the image, or an overlay applied to the image that manipulates the image in the message or adds augmented reality animation to the image in the message. Examples of various such implementations are described in detail below.

[0038] Figure 1 This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content, including data for modifying animated images or creating animations from models) over a network. Messaging system 100 includes multiple client devices 102, each hosting multiple applications, including messaging client applications 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client applications 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0039] According to the implementation described herein, client device 102 can use an application such as messaging client application 104 to implement a system for generating pseudo-random animations synchronized with audio data received at client device 102. Data for the system can be managed by animation system 124 of application server 112. Part of the management performed by application server 112 may involve accepting data created by a designer for a specific animation as image modification data (e.g., overlays, image transformations, filters, etc., to be implemented using model animation within the messaging system, and managing the availability of such image modification data).

[0040] Therefore, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and with the messaging server system 108 via network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands for invoking functions) and payload data (e.g., text, audio, video, or other multimedia data, including image modification data for implementing pseudo-random animations as described herein).

[0041] Messaging server system 108 provides server-side functionality to specific messaging client applications 104 via network 106. While some functions of messaging system 100 are described herein as being performed by messaging client application 104 or by messaging server system 108, it should be understood that the location of certain functions within messaging client application 104 or messaging server system 108 is a design choice. For example, it may be technically more resource-efficient to initially deploy certain technologies and functions within messaging server system 108, but later migrate those technologies and functions to messaging client application 104, where client device 102 has sufficient processing power.

[0042] The messaging server system 108 supports various services and operations provided to the messaging client application 104. Such operations include sending data to and receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. In some implementations, as an example, this data includes: message content, client device information, geolocation information, media comments and overlays, message content persistence conditions, social network information, and live event information. In other implementations, other data is used. Data exchange within the messaging system 100 is activated and controlled via functions available through the GUI of the messaging client application 104.

[0043] Now, specifically to the messaging server system 108, the application programming interface (API) server 110 is coupled to the application server 112 and provides a programming interface to the application server 112. The application server 112 is communicatively coupled to the database server 118, which provides easy access to the database 120, in which data associated with the messages processed by the application server 112 is stored.

[0044] Specifically, an application programming interface (API) server 110 is described, which receives and transmits message data (e.g., commands and message payloads) between client device 102 and application server 112. Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that messaging client application 104 can invoke or query to activate functionality of application server 112. Application Programming Interface (API) server 110 exposes various functions supported by application server 112, including account registration, login functionality, sending messages from one messaging client application 104 to another messaging client application 104 via application server 112, sending media files (e.g., images or videos) from messaging client application 104 to messaging server application 114 and making them accessible by another messaging client application 104, setting up media data (e.g., stories) collections, retrieving the friend list of the user of client device 102, retrieving such collections, retrieving messages and content, adding and deleting friends from the social graph, locating friends within the social graph, and opening application events (e.g., related to messaging client application 104). In some implementations, aspects of a system for generating pseudo-random animations and synchronizing animations with audio data can be accessed through such API server 100.

[0045] Application server 112 hosts multiple applications and subsystems, including messaging server application 114, image processing system 116, social networking system 122, and animation system 124. Messaging server application 114 implements multiple messaging techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories, galleries, or sets). These collections are then made available to messaging client application 104 by messaging server application 114. Given the hardware resources available for additional processor and memory-intensive processing of data, such processing can also be performed on the server side by messaging server application 114.

[0046] Application server 112 also includes an image processing system 116, which is dedicated to performing various image processing operations, typically performing various image processing operations with respect to images or videos received in the payload of messages at message server application 114.

[0047] Social networking system 122 supports various social networking functions and services and makes these functions and services available to messaging server application 114. To this end, social networking system 122 maintains and accesses an entity graph within database 120. Examples of functions and services supported by social networking system 122 include identifying other users of messaging system 100 with whom a particular user has a relationship or who a particular user "follows," as well as identifying a particular user's interests and other entities.

[0048] Application server 112 is communicatively coupled to database server 118, which provides easy access to database 120, where data associated with messages processed by message server application 114 is stored.

[0049] Database 120 also stores image modification data, which may include computer models for implementing animations as described herein. In some embodiments, such image modification data may be used to implement filters or other such transformations or AR images.

[0050] As mentioned above, filters, overlays, image transformations, AR images, and similar terms refer to modifications that can be made to videos or images. This includes real-time modifications, modifying images as they are captured using the device's sensors, and then displaying the modified image on the device's screen. It also includes modifications to the stored content of video clips in, for example, compilations that can be modified. For instance, in a device that can access multiple filters, a user can use a single video clip with multiple filters to see how different filters will modify the stored clip. For example, multiple filters applying different pseudo-random motion models can be applied to the same content by selecting different filters for the content. Similarly, real-time video capture can be used in conjunction with the shown modifications to show how the video image currently captured by the device's sensors will modify the captured data. Such data can simply be displayed on the screen without being stored in memory, or the content captured by the device's sensors can be recorded and stored in memory with or without modification (or both). In some systems, preview functionality can simultaneously show how different filters look in different windows of the display. For example, this can make it possible to view multiple windows with different pseudo-random animations on the display simultaneously.

[0051] Therefore, the data, and the various systems used to modify content using filters or other such transformation systems, can involve: detecting objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking such objects as they leave, enter, and move around the field of view in video frames, and modifying or transforming them while tracking. In various implementations, different methods can be used to achieve such transformations. For example, some implementations may involve generating a three-dimensional mesh model of one or more objects and using transformations of the model and animated textures in the video to achieve the transformation. In other implementations, tracking points on objects can be used to place images or textures (which can be two-dimensional or three-dimensional) at the tracked locations. In yet another implementation, neural network analysis of video frames can be used to place images, models, or textures into content (e.g., images or video frames). Therefore, filter data refers both to the images, models, and textures used to create content transformations and to the additional modeling and analysis information required to achieve such transformations through object detection, tracking, and placement.

[0052] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and store it in the device's memory, or a video stream can be generated using the device's sensors. Furthermore, computer-animated models can be used to process any object, such as a human face and parts of a human body, animals, or inanimate objects such as chairs, cars, or other objects.

[0053] In some implementations, when a specific modification is selected along with the content to be converted, a computing device identifies the element to be converted and then detects and tracks the element if it exists in a frame of the video. The elements of the object are modified according to the modification request, thus converting the frames of the video stream. The conversion of video stream frames can be performed using different methods for different types of conversions. For example, for a conversion that primarily refers to changing the form of object elements, feature points of each of the object's elements are calculated (e.g., using an Active Shape Model (ASM) or other known methods). A feature-point-based mesh is then generated for each of at least one element of the object. This mesh is used for subsequent stages of tracking object elements in the video stream. During tracking, the mesh for each mentioned element is aligned with the position of each element. Additional points are then generated on the mesh. A first set of first points is generated for each element based on the modification request, and a second set of points is generated for each element based on the first set of points and the modification request. The frames of the video stream can then be converted by modifying the elements of the object based on the first and second sets of points and the mesh. In such a method, the background of the modified object can also be changed or deformed by tracking and modifying the background.

[0054] In one or more embodiments, the transformation of some regions of an object by altering its elements can be performed by calculating feature points for each element of the object and generating a mesh based on the calculated feature points. Points are generated on the mesh, and then various regions based on these points are generated. The elements of the object are then tracked by aligning the region of each element with the position of each of at least one element, and the properties of the regions can be modified based on modification requests, thus transforming frames of the video stream. Depending on the specific modification request, the properties of the mentioned regions can be transformed in different ways. Such modifications may involve: changing the color of the region; removing at least some portions of the region from the frames of the video stream; including one or more new objects in the region based on the modification request; and modifying or deforming the elements of the region or object. In various embodiments, any combination of such modifications or other similar modifications may be used. For some models to be animated, some features can be selected as control points to determine the entire state space of options for model animation.

[0055] In some implementations of computer animation models used to transform image data using face detection, faces are detected on the image using a specific face detection algorithm (e.g., Viola-Jones). The Active Shape Model (ASM) algorithm is then applied to the facial regions of the image to detect facial feature reference points.

[0056] In other implementations, other methods and algorithms suitable for face detection can be used. For example, in some implementations, markers are used to locate features, representing distinguishable points present in most of the images considered. For example, for facial markers, the location of the left pupil could be used. Secondary markers can be used when initial markers are unrecognizable (e.g., if the person is wearing an eye patch). Such marker recognition processes can be used for any such object. In some implementations, the set of markers forms a shape. The shape can be represented as a vector using the coordinates of its midpoints. One shape is aligned with another shape through a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the points of the shapes. The average shape is the mean of the aligned training shapes.

[0057] In some implementations, the search begins by searching for marker points from an average shape aligned with the position and size of a face determined by a global face detector. This search is then repeated, adjusting the position of the shape points to suggest an initial shape using template matching with the image texture around each point, and then conforming the initial shape to a global shape model, until convergence occurs. In some systems, individual template matching is unreliable, and the shape model pools the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid, from coarse to fine resolution.

[0058] The implementation of the conversion system can capture image or video streams on a client device and perform complex image manipulations locally on a client device such as client device 102, while maintaining a suitable user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion shifting (e.g., changing a frowning face to a smiling face), state shifting (e.g., aging a subject, reducing apparent age, changing gender), style shifting, application of graphic elements, and any other suitable image or video manipulation implemented by a convolutional neural network that has been configured to execute efficiently on the client device.

[0059] In some example implementations, a computer animation model for transforming image data can be used by the system, where a user can capture an image or video stream (e.g., a selfie) using a client device 102, which has a neural network running as part of a messaging application 104 running on the client device 102. The transformation system running within the messaging application 104 determines the presence of faces in the image or video stream and provides modification icons associated with the computer animation model to transform the image data, or the computer animation model can be presented as associated with the interface described herein. Modification icons include changes that can serve as the basis for modifying the user's face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates processing to transform the user's image to reflect the selected modification icon (e.g., generating a smiley face for the user). In some implementations, once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the mobile client device. The transformation system can implement a sophisticated convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. In other words, once the edit icon is selected, the user can capture an image or video stream and the results of the edits can be presented in real-time or near real-time. Furthermore, the edits can be continuous while the video stream is captured and the selected edit icon remains active. Machine-trained neural networks can be used to achieve such edits.

[0060] In some implementations, the graphical user interface (GUI) presenting the modifications performed by the conversion system can provide the user with additional interactive options. Such options may be based on the interface used to initiate content capture and the selection of a specific computer animation model (e.g., initiated from a content creator user interface). In various implementations, modifications can persist after the initial selection of the modification icon. The user can turn modifications on or off by tapping or otherwise selecting a face modified by the conversion system and save it for later viewing or browsing to other areas of the imaging application. In cases where multiple faces are modified by the conversion system, the user can globally turn modifications on or off by tapping or selecting a single face modified and displayed within the GUI. In some implementations, individual faces in a set of multiple faces can be modified or individually turned on by tapping or selecting a single face or a series of individual faces displayed within the GUI.

[0061] In the various embodiments described herein, any of the modifications described above can be integrated with a system used to generate a model, or with a model having state-space options associated with matching audio data (e.g., setting a velocity harmonic value for the animation speed in conjunction with a rhythm value from detected audio data). In other embodiments, such modifications, including filters or any other such overlay changes to the content, can be integrated with the pseudo-random animation system described herein in a variety of different ways.

[0062] Figure 2 This is a block diagram illustrating further details of a messaging system 100 according to an example implementation. Specifically, the messaging system 100 is shown as including a messaging client application 104 and an application server 112, which in turn include several subsystems, namely a short timer system 202, a collection management system 204, and an annotation system 206.

[0063] Annotation system 206 includes animation system 207, which can be used to implement some or all aspects of the pseudo-random animation system as described herein as part of messaging client application 104 on client device 102. In other embodiments, such a system may be divided into different parts running on client device 102 and server system.

[0064] Content system 204 can be used to store content (e.g., images and videos) that can be used in animation models to generate pseudo-random animations as described herein. In some systems, such content may be real-time content (e.g., for AR content) or stored (e.g., for compilations of previously captured content). If a user is generating a message, management interface 208 can be used to select modification information (e.g., image transformation or AR modification) for a previously configured element to be included in the message. If a user is designing modification information (e.g., by selecting animation probabilities, designing animation models, or selecting other such options, such as creating filters, AR modifications, or other such overlays or image and video transformations), management interface 208 can be used in a designer device with such a system.

[0065] The short timer system 202 is responsible for implementing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the short timer system 202 incorporates multiple timers that selectively display messages and associated content based on duration and display parameters associated with messages, message sets, or graphical elements, enabling access to messages and associated content via the messaging client application 104. In some implementations, this can restrict access to specific configurations of pseudo-random animations as described herein.

[0066] The collection management system 204 is responsible for managing media collections (e.g., media collections that include collections of text, images, video, and audio data). In some examples, collections of content (e.g., messages that include images, videos, text, and audio) can be organized into "event highlights" or "event stories." Such collections can be made available for a specified time period, such as the duration of an event related to the content. For example, content related to a concert can be made available as a "story" for the duration of a concert. The collection management system 204 can also be responsible for publishing icons that notify the user interface of the messaging client application 104 of the existence of a specific collection.

[0067] In some implementations, the management interface 208 of the collection management system 204 also includes interface options that allow collection managers to manage and curate specific collections of content. For example, the management interface 208 enables event organizers to curate collections of content related to a specific event within such a system (e.g., removing inappropriate content or redundant messages). Furthermore, the collection management system 204 can employ machine vision (or image recognition technology) and content rules to automatically curate content collections. In some implementations, users may be compensated for including user-generated content (e.g., a specific model with associated pseudo-random animation elements) in the collection. In such cases, the management interface 208 operates to automatically pay such users for using their content.

[0068] Figure 3 This is a block diagram illustrating the components of a messaging system 100 that enable the system to perform operations including transmitting content modified using a pseudo-random animation system and associated modification data between accounts. The animation system 124 is shown as including a presentation module 302, a user profile module 304, a media module 306, and a communication module 308, all configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules can be implemented using one or more processors 310 (e.g., by configuring one or more such processors 310 to perform the functions described for that module) and therefore may include one or more processors 310. While these modules are described in the context of an animation system to implement pseudo-random animation in a messaging system, such elements can be integrated with larger profile and data management systems in various implementations.

[0069] Any one or more of the described modules can be implemented individually using hardware (e.g., one or more processors 310 of a machine) or a combination of hardware and software. For example, any module of the described animation system 124 may physically comprise an arrangement of one or more processors 310 (e.g., a subset of one or more processors of a machine or one or more processors of a machine) configured to perform the operations described herein for that module. As another example, any module of the animation system 124 may comprise software, hardware, or both software and hardware configured to perform the operations described herein for that module (e.g., one or more processors of a machine) of an arrangement of one or more processors 310. Thus, different modules of the animation system 124 may include or configure different arrangements of such processors 310 or a single arrangement of such processors 310 at different points in time. Furthermore, any two or more modules of the animation system 124 may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Moreover, according to various example embodiments, modules described herein as being implemented within a single machine, database 120, or device may be distributed across multiple machines, databases 120, or devices.

[0070] Various modules within the animation system 124 or other management systems of the messaging system 100 (e.g., systems using messaging server system 108) can operate in multiple ways to improve device performance by managing the system communications and interfaces described herein. The state-space structure, as part of the pseudo-random animation, allows processing and memory resources to be used more efficiently to generate complex animations than other systems that simply predetermine such complex animations or use other resource-intensive options to generate them. In some systems according to the embodiments described herein, the use of the structure for pseudo-random animations is both outside the standard structure of the messaging system 100 and outside the drivers used by the system; therefore, the overall system performance is improved by reducing interface operations (especially for the creator account) while also providing increased creator control and options for creator attributes. Thus, the various modules and systems described above and below enable the implementation of complex pseudo-random animations within a messaging system using resource-constrained user devices for animation implementation.

[0071] Then, Figures 4A to 4E Aspects of a system for displaying pseudo-random animation are shown. As described above and... Figure 4A As shown, the user's client device 102 can be used to display an image, such as image 410, on the display 402 of the client device 102. When an implementation of pseudo-random animation is configured on the client device 102, a computer animation model is applied to the image data to achieve pseudo-random animation.

[0072] Figure 4B An aspect of a potential computer animation model is shown, which acquires image data and segments the data into multiple frames, wherein animation is applied to the transitions between frames. Figure 4B An animation region 420 is included in the display 402 of the client device 102. The animation region includes three-frame animation regions 422, 424, and 426 with boundaries 428. These three frames are part of a computer animation model used with image 410 data to generate pseudo-random animation using a computer animation model and image 410. The computer animation model includes control points 427, which may have motion patterns and speeds adopted to create pseudo-random animation using image 410.

[0073] Then, Figure 4C The animation is further illustrated. (See example.) Figure 4C As shown, portions of image 410 are placed in frames 422, 424, and 426. As control point 427 moves, frame boundaries are configured to follow the movement of control point 427. For example, as the control point moves up and down, boundary 428 can move within display area 420 while maintaining its relative position to control point 427 and clipping edges at its intersection with display area 420. Portions of image 410 within each frame 422, 424, and 426 can be configured to match the movement of control point 427, or remain stationary and adjust the displayed portion of the image within each frame. The limitations on the movement of control point 427 used to generate pseudo-random animation are discussed in more detail below.

[0074] Although Figure 4B and Figure 4C A computer animation model of geometric overlay with a single control point is shown, but Figure 4D and Figure 4E This illustrates a computer animation model modified from an image. Figure 4D In the process, the computer animation model identifies eyebrow regions 450 and 451 within image 410, as well as the ranges of motion 452 and 453 of each eyebrow region 450, 451. When implementing pseudo-random animation, image 410 is displayed together with the modified eyebrow regions 450, 451 to generate animation on display 402 and within display area 420. During the video output of the animation, the eyebrow regions are treated as control points in the computer model and can be manipulated as described above. Figure 4E The movements are as shown in steps 460 and 461, both of which are associated with control points of the computer animation model.

[0075] therefore, Figures 4A to 4E An example of applying a computer animation model to image data according to the implementation described herein is shown. Figure 4B and Figure 4CIn this model, the visible output is the geometry of the boundary 428 around the frame adjusted using control point 427. Figure 4D and Figure 4E In this example, the visible output of the model is the movement of an eyebrow image as part of a motion image, where the eyebrow region serves as a control point. While these two examples illustrate a computer animation model according to an implementation described herein, it is evident that many other examples are possible. For instance, any part of the face or body can be selected as a control point and animated. In some implementations, the entire body can be animated. In one example, a full-body image can be used to generate a two-dimensional "puppet" from the image, where interconnected parts have control points, the movement of the interconnected parts is constrained by the connections between the parts, and the movement for each control point is set as described below. Furthermore, other objects besides the face and body can also be animated. An image of a tree can be analyzed to identify control points within the trunk and at the branches, and used to animate the movement of the tree. An image of a chair can be similarly animated using control points.

[0076] Furthermore, in addition to, as by Figure 4D and Figure 4E In addition to animate objects within an image as shown, any type of overlay can be animated. For example, although Figure 4B and Figure 4C Frame boundary animation is shown, but in some implementations, any type of overlay can be used as a computer animation model. For example, a computer model of fireworks can be constructed as a simple overlay, where an area for displaying fireworks is selected within the display area 410 of the display 402. Control points can be considered as fixed areas within the display area 410, or they can be independently assigned to the animation as they are presented within the display area 410. The display speed and movement of the animation can then be controlled for pseudo-random animation within a defined area of ​​the computer animation model of fireworks set by the designer of the specific model.

[0077] Therefore, it is evident that designers of such animations can select control points from a variety of models to apply the pseudo-random animation described herein to generate various types of video animations. As described above, such animations can be generated using a management interface (e.g., a computer animation model designer tool) or any such application as part of generating models for use with pseudo-random animations in messaging systems or any systems as described herein.

[0078] Then, Figure 5A and Figure 5B An aspect of the audio data that can be used for the pseudo-random animation described in this paper is shown. Figure 5A and Figure 5BAspects of audio data 500, which can be used with a system for generating and displaying animations according to some embodiments, are illustrated. Audio data 500 shows the amplitude of a specific set of audio inputs changing over time. Although audio data 500 is shown as time-based amplitude information, such information can be received at client device 102 and analyzed to obtain various audio characteristics. Figure 5B As shown, specific audio data 500 can be analyzed to obtain audio characteristics, including identifying rhythm 504 based on beat 502 and harmonics 514 that can be associated with other pattern data 512 within the audio data. The audio stream may originate from a microphone of client device 102, or it may originate from a file or other storage on the device. The rhythm of the audio data is a fundamental audio characteristic that can be identified as part of an audio data set by analyzing the audio data stream at a device such as client device 102. In addition to identifying beats within the audio data and rhythms associated with a portion of the audio data (e.g., the beats of each time interval of the audio data), other audio characteristics can be identified. Such audio characteristics may include, but are not limited to, melody analysis, harmonic analysis, frequency content of the music, beat consistency, variations in “speech” or expected contributors to the audio, or other direct details of sound, frequency, and variation within the audio data. Furthermore, indirect characteristics of the audio data can be assigned to audio characteristics, such as the “energy” of the audio data, the “danceability” of the audio data, variations and transitions of any identified characteristics, pauses or transitions between repeated audio characteristics, matching known or audio patterns, or any such characteristics. When a device analyzes an audio data stream, it can assign values ​​to any such characteristics, and these values ​​can be updated over time as the audio stream continues. For example, an audio stream can simply contain speech and be assigned a tempo value of 0. When a piece of music is played, analysis of the audio data stream can identify detected repeating beats and adjust the tempo value to match the detected beat repetitions in the audio stream. When the device processor identifies additional characteristics or changes in characteristics of the audio data, it can assign or update the values ​​of these characteristics.

[0079] Figure 6A Aspects of motion pattern 600, which can be used as part of an animation state space according to some embodiments, are shown. Figure 6ASix example motion patterns 610, 612, 614, 616, 618, and 620 are shown. Each motion pattern is shown in the movement space 602 of an example control point. For example, if motion pattern 618 is selected, the control point for which motion pattern 618 is selected will move in the circular pattern shown. If motion pattern 616 is selected, the control point will move linearly back and forth in a single direction, as shown. Motion patterns 610 to 618 are intended to illustrate simple motion patterns along a fixed path. Other motion patterns with more complex characteristics are also possible. For example, for motion pattern 620, although a specific path is not described, movement can occur anywhere within the circular shape, but movement is prohibited outside the circular shape in the area where the control point may still be within the constraints of acceptable movement of the control point defined by the movement space 602.

[0080] The use of a selected motion pattern 620 within the motion space 602 of all possible movements allows for the organized configuration of selected movements to match audio data, as well as the suppression of specific unwanted movements. A large number of simple motion pattern templates can be created and managed, while still allowing designers to design pseudo-random animations from a large state space without overwhelming them with choices or exceeding their processing resources. The state space size can be easily adjusted based on the actual processing resources available or anticipated by the designer, thereby limiting the number of motion patterns associated with a particular computer model or its implementation. For example, the template system can access any number of motion patterns, or the designer can create any number (e.g., thousands, tens of thousands, etc.) of motion patterns. When implementing a particular computer animation model, motion pattern data can be filtered so that only data associated with selected data (e.g., non-zero probability motion patterns) is included as part of specific communication. In simpler systems with a finite number of motion patterns, data for all motion patterns, including data for motion patterns set with zero probability, can be passed to simplify and standardize communication for certain types of animation.

[0081] According to the implementation described herein, the computer animation model will then have a motion state space based on multiple control points in the model, multiple motion modes available for each control point, and multiple animation velocities available for each model at each control point.

[0082] For example, if motion pattern 600 represents all possible motion patterns for control point 427, where the probability assigned to each motion pattern is equal, then during animation generation, an associated computer model will be used to generate an output video that incorporates the boundary 428 of the movement of control point 427 and has motion at a given time according to the motion pattern 600 assigned to control point 427. The motion assigned at a given time can be changed based on the probability assigned to a given motion state and the motion pattern for that particular motion state. Figure 6B This illustrates that, over time, the motion states 661, 663, 665, 667, and 669 of the computer animation model 650 change as the computer animation model is used to generate output video animation synchronized with the audio data 670.

[0083] As described herein, animations are assigned motion states at a given time or time interval. Motion states are selected from the state space of all possible motion states of the computer animation model. Different computer animation models can have different frames (e.g., combinations of variables or data structures) for the associated state space of the model. In one example implementation, the animation frame associated with the computer animation model includes A control points (e.g., control point 527 or control points associated with eyebrow regions 450, 451), B motion patterns (e.g., motion patterns 610 to 620), and C rhythm harmonics (e.g., harmonic multiples of rhythm 504 determined from beat 502), such that the number N of motion states of the computer animation model within the animation frame is:

[0084] (1)

[0085] Furthermore, as part of the framework, a probability is assigned to each of the N motion states M, such that the probability mass function P() assigned to the probability value of each motion state is:

[0086] (2) .

[0087] When a computer animation model has a value assigned to each element in the state space, a specific motion state is selected for use at any given time based on the probability for each motion state.

[0088] In a simple example using the above diagram, the frame of a computer animation model can have two control points (e.g., Figure 4D and Figure 4EThe model has six motion modes (e.g., motion modes 610, 612, 614, 616, 618, and 620). Furthermore, the template can have four harmonic velocities (e.g., 1X, 0.5X, 2X, and 4X). Each harmonic velocity is a multiple of the rhythm determined by audio data used in conjunction with the output animation generated using the computer animation model. In this example, the state space of the computer animation model comprises 576 motion states (e.g., (6 x 4)). 2 (Single motion states). Each motion state can have a different assigned probability value, such that some motion states occur more frequently than others. For example, the motion states of the above model using motion pattern 616, where two control points (e.g., the eyebrow region) occur at rhythmic harmonics equal to the rhythm of the music, can be set to occur 95% of the time, while the remaining motion states, assigned the same probability, occur during the other 5% of the time. Transitions between motion states (e.g., selection or reselection of motion states) can occur at fixed intervals or randomly. In other implementations, any possible triggers or intervals for transitions between motion states can be used, including thresholds or transitions in audio characteristics as described below.

[0089] As mentioned above, various audio characteristics can be determined based on audio data. In one example, an audio energy value, or "danceability" value, is determined for a piece of audio data, and this can be used as a threshold to determine whether to generate a pseudo-random animation that matches the audio data. For example, Figure 6C The audio data 680 in the middle can be identified as having the same characteristics as... Figure 6B The audio data in 670 has a similar rhythm. However, additional details (e.g., frequency content, beat intensity, beat consistency, etc.) can be used to determine additional characteristics that can be specific to the frame or a particular computer animation model and can be defined by the designer. Figure 6CIn the example, audio data 680 does not meet the animation threshold during the time periods associated with motionless states 671 and 675, but meets the motion threshold criterion during the time periods of motion states 673 and 679. As described above, motion states 673 and 679 can be randomly selected from the state space of the computer animation model being executed based on the assigned probabilities of all motion states, including the selected motion state 673 and selected motion state 679 during the time periods in which these motion states are used to generate the video animation. In some implementations, the computer animation model is frozen during motionless state 671 and does not move at all in the video frames. In other implementations, a default or "wait" animation can be used, transitioning to the selected motion state when the audio data threshold is met. Such implementations can use transition animations between such states, or they can move directly between states. Similarly, in the various implementations described herein, the randomly selected motion states can have transition animations used as transitions between computer model states, or the movement can be simply animated based on the current motion state when a new motion state is selected.

[0090] Figure 7A Aspects of a computer model, according to some embodiments, that can be used as part of a system for generating and displaying animations are shown. In the example details of the computer animation model in Figure 4 discussed above, a two-dimensional computer animation model is described. While complex models are possible in the two-dimensional case as described above (e.g., two-dimensional puppets of people or objects, multi-transformation models, such as fireworks models where moving objects appear and disappear), some embodiments use three-dimensional computer animation models.

[0091] Figure 7A A simplified computer animation model 700 is shown, depicting skin 710 surrounding three bones 730, 720, and 710, and bones, joints, or connection points 701, 702, 703, and 704. Each bone 710, 720, and 730 has a control point, which can be a connection point or any other such point directly on the bone. In one embodiment, connection point 701 is stationary, connection point 702 is a control point of bone 710, connection point 703 is a control point of bone 720, and connection point 704 is a control point of bone 730. The relative movement of the skin in regions 711, 721, and 722 is modified primarily but not exclusively by the motion patterns associated with individual control points, such that the first region 711 is primarily affected by joint 702, the second region 721 by joint 703, and the third region 722 by joint 704, wherein the joints are guided to follow motion patterns defined within the state space of the computer animation model 700. The aspects of this influence on the skin 701 are... Figure 7BThe skins 710A, 710B and 710C are shown in the middle.

[0092] Figure 7C Aspects of motion pattern 780, which can be used as part of an animation state space according to some embodiments, are shown, and Figure 7D Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations. Figure 7D A bone 760 with joints 750 and 770 is shown. Figure 7D In this context, joint 750 is a reference point for bone 760, while joint 770 is a control point, wherein the movement pattern of control point 770 is defined in a spherical surface 774 surrounding joint 770, and the movement pattern has a maximum range of motion 772 shown as ranges 772A and 772B.

[0093] Within the framework of the computer animation model, control point 770 has motion patterns, such as motion pattern 780, which is defined by the movement of joint 770 along surface 774 within a range of motion 772, wherein joint 750 serves as a fixed reference point for a specific motion pattern. Figure 7A and Figure 7B In the computer animation model 700, each bone can have the same or different associated motion patterns and ranges of motion. For example, joint 701 can be a reference point for bone 710, joint 702 can be a reference point for bone 720, and joint 703 can be a reference point for bone 730. Skins 710A, 710B, and 710C illustrate the effect of corresponding control point motion relative to the reference points of the control points on the skin. For example, skin 710C illustrates the effect of joint 704 moving when joint 703 is stationary, skin 710B illustrates the effect of joint 703 moving when joint 702 is stationary, and skin 701A illustrates the effect of joint 702 moving relative to joint 701.

[0094] Figure 7C A pattern depicted by control points relative to a plane perpendicular to a line created by extending a line through a reference point is shown. Each pattern 780 of joint 703 can then be considered a projection onto a plane perpendicular to a line extending from bone 720 through a point containing joint 703. In a user interface where the designer assigns probabilities to different motion states, the screen interface can display such projections and allow them to be selected and / or have associated inputs containing assigned probability values. Such an interface allows the designer to create patterns that will automatically be converted from a two-dimensional projection of the interface into motion patterns associated with control points, having an assigned set of other variables (e.g., harmonic velocity, phase relationship with the beat that determines the offset in the repetitive motion pattern, etc.).

[0095] In computer animation models, each control point (e.g., therefore each bone) can have an independently assigned motion pattern, and each motion pattern has a separate probability. Figure 7E Aspects of a system for generating and displaying animations, according to some embodiments, are illustrated. Figure 7E In the process, audio data 775 is received at a device that uses a computer animation model 700 to output video animation. The animation model 799 transitions through motion states 731 to 735, each motion state consisting of an independent combination of motion, velocity, and any other such assigned characteristics from the template. Each control point in the model can move independently, with the first region 711 moving through movements 775 to 778 based on joint 702 movement in different patterns described by the control point movement patterns of the selected state. Similarly, the second region 721 uses different movements 781 to 783, and the third region 722 uses movements 791 to 795. This can induce a variety of complex movements occurring in a pseudo-random manner synchronized with the beat.

[0096] For example, if the movement patterns of joints 702, 703, and 704 are set using all side-to-side motions synchronized with the harmonic speed of the rhythm equal to the audio data, then skin 710 will swing left and right in time with the beat. If joints 702 and 703 are set with the same side-to-side motions but spaced half a repetition apart, and joint 704 is circular, then the lower part of skin 710 will swing back and forth without swaying, while the top of skin 710 will move in a circle. Because these movements are synchronized with the harmonics of the audio data rhythm, some animations can give the impression of "dancing" or complex, varied movements synchronized with the rhythm of a complex pseudo-random pattern.

[0097] Computer animation models can extend this template in sophisticated ways using bones with control points and reference points, as well as associated motion patterns. Computer animation model 800 illustrates a model with bones 821, 831, 841, 851, and 861, and joints 710, 820, 830, 840, 850, and 860. Similar to computer animation model 700, each bone can have control points, which can be configured with motion patterns relative to reference points on the same bone. For example, for bone 861, joint 860 can be a control point, and joint 850 can be a reference point. To simplify the overall design of the computer animation model, it can have one or more global reference points or parent reference points. For example, joint 810 can be defined as a parent reference point, allowing it to operate as a reference point for at least one control point, but not as a control point itself. Computer animation models can have multiple parent reference points. For example, if bone 831 is designed to be fixed, joints 830 and 820 can be parent reference points, while joints 810, 840, 850, and 860 are control points.

[0098] In some systems, instead of assigning each control point an independently allocated motion, inverse kinematics can be used to confine the motion of multiple control points as part of a single motion state. For example, when joint 810 operates as a parent reference point, one motion state of the computer animation model 800 can use the motion patterns of each joint except joint 810. A second motion state can have a motion pattern of joint 840 determined relative to joint 810, where the motions of joints 830 and 820 are automatically determined to achieve the selected motion of joint 840. In such an implementation, when joint 840 has a motion determined relative to joint 810, the motion patterns discussed above can be used to determine joints 850 and 860.

[0099] Therefore, certain motion states can have one or more skeletal motion chains, where the motion of control points at the ends of the motion chains is selected as part of the motion state, and control points within the motion chains are automatically determined. During animation design, constraints on the motion chains can be presented to the designer as part of an interface with options for creating motion patterns within the constraints of the chain's range of motion. Alternatively, specific motions of the motion chain can be presented, where the designer selects among possible motions. As mentioned above, some frames can include a set of motion patterns, where the designer simply assigns probabilities to preferred motions. This can be used both for simple animations of a single bone as part of a chain from a parent reference point to multiple bones with control endpoints, and for inverse motion chains. Furthermore, the state space can include motion patterns for each individual control point, two states for motion patterns of the motion chain, or both states in the same state space of a computer animation model.

[0100] Then, Figures 9A to 9C Aspects of a computer model, according to some embodiments, are shown that can be used as part of a system for generating and displaying animations. Figure 9A The computer animation model 900 uses a penguin skin 902, which is configured to be animated using a pseudo-random computer animation model as described in this paper. Figure 9B The internal structure 910 of a computer animation model 900 with bones 940 to 952 is shown. Figure 9C The cover 911 of the structure 910 within the skin 902 is shown to illustrate how the computer animation model 900 can generate complex pseudo-random animations according to the embodiments described herein.

[0101] As described above, a computer animation model can include instructions for generating an output video animation configured to animate the model's control points to present pseudo-random motion synchronized with audio data. Figure 9C In the example, the skeletal model with structure 910 includes bones 940 to 952. The framework of the computer animation model includes data about bones 940 to 952, as well as motion constraints for each bone, the effect of each bone on skin 902, and any other such information that defines possible options for the computer animation model. Designers can adopt constraints provided by such a framework, including the default state space of the movement patterns provided for the bones, velocity harmonics, and other such information, and designers can modify this information to generate a network of computer animation models that can be distributed via a network and implemented on a device to create output video. Designer options may include: creating new motion patterns; selecting the probability of motion states, which include specific motion patterns and velocity harmonics, synchronization relationships between patterns of bones; selecting thresholds or probabilities for different animation options, or other such information for specific implementations of the model that can be distributed via a messaging system.

[0102] For example, in some implementations, bones 942 and 944 may have a circular or circular range of motion, or a range of motion above the head of the penguin model, allowing the arm flippers to be animated. Designers can choose the motion of bones 944 and 942 to restrict actual movement in a particular state space to simple flapping movements with only a few degrees of motion, and restrict forward and backward movement of the bones relative to planes of the body (e.g., the plane between the eyes and toes, or another such plane separating the front and back of the model's skin). Similarly, even if the frame can achieve more complex foot movements, the motion of bones 951 and 952 can be restricted to simple up-and-down "stomping" movements configured to match the beat or harmonics of a rhythm detected from audio data. Any such restrictions can be chosen as part of a user interface for selecting probabilities and / or state space elements for a particular model. Furthermore, as mentioned above, in addition to the state space of model 900 which includes the motion patterns of individual bones, some implementations of such a state space may include inverse kinematic motion patterns for certain control points. For example, an inverse kinematic movement pattern could enable bone 940 to move in a circular pattern while keeping the endpoints of bone 940 perpendicular to the ground plane, allowing the head of skin 902 to draw circles without tilting. Such state space elements could include automatic movements of bones 940, 930, and 920, while also having independent movements of bones 951, 952 (e.g., stomping), 942, and 944 (e.g., flipper waving).

[0103] Designers can access the system's design tools (e.g., the design tool management interface 208 of application 104 or other such tools) to create computer animation models from scratch (e.g., by creating models within the system) or by modifying frameworks available to the designer. In some implementations, this may involve designing a user interface. As described above, such a user interface may include one or more windows for illustrating animations in a particular state space, such that multiple animations of multiple different motion states of the computer animation model's state space are simultaneously displayed on the screen in different windows, wherein the probabilities and / or other design options for each state space have an input interface. Such a design interface may include options for selecting, for example, elements of different skins, modifying the model's skeleton or control points, generating motion patterns for adding to the state space, or other such options.

[0104] In some implementations, different computer-animated models can be displayed in such an interface as tiles describing a particular computer-animated model. In addition to including computer animation details, such a display may also include additional elements, such as incorporating the animation as part of an augmented reality image, or as part of an overlay or modification of stored data. Thus, such a display can show a single stored video clip in multiple windows of the display, where each window includes different animations from one or more computer-animated models.

[0105] In some implementations, tools may be available for filtering motion states or selecting groups of motion states. For example, the interface may allow setting all motion states below the harmonic level of the tempo to zero (e.g., all velocity harmonics below 1X). Some inputs may allow suppressing all motion at certain control points. Some inputs may be used to characterize certain motion patterns, such as motion that creates animation speeds above a certain velocity, or acceleration or jerk motion values ​​above a provided threshold. For motion patterns that are not strictly defined but only create random motion within a range (e.g., motion pattern 620), constraints may be provided for the motion within that range, such as the number of momentum changes per unit time, maximum velocity, or other such characteristics of the motion. In some implementations, energy-based motion activation may be used for parts of a skeleton or two-dimensional model. For example, in one implementation, the kinetic energy of the model may be defined by:

[0106] (3) ;

[0107] Where KE is the model motion energy for a specific set of motion data values ​​in the value set, and j is the sum of the number of joints used to iterate over multiple animation elements. In such a model:

[0108] (4) Moment of inertia

[0109] Where w() is the angular velocity at the joint summed with respect to a given value j, k is the iterative value for the number of child joints attached to the parent joint in the computer model, r is the radius of each joint from the energy reference point (e.g., the parent reference point or global reference point for a specific motion state), and the angular velocity is determined individually for each motion mode and each velocity harmonic in multiple motion modes and multiple velocity harmonics.

[0110] In such a system, each motion state can have an assigned energy value. If the state includes randomness in the motion (e.g., motion pattern 620), the average kinetic energy and maximum kinetic energy can be determined based on the allowed randomness. This information can then be used in various ways. In some implementations, the designer can select minimum and maximum energy values, and all motion states in the model's state space that exceed these thresholds can be suppressed (e.g., set to zero probability). In some implementations, different energy states can be matched with different characteristics of the audio data. For example, a set of energy thresholds can be used to set the state space for a first set of audio characteristics, and a second set of energy thresholds can be used to set the state space for different audio characteristics. Similarly, "energy" values ​​based on frequency content, beat, or other such audio analysis can be used to match high-energy audio data with a specific range of kinetic energy state spaces.

[0111] Using different state spaces in different situations allows, for example, different state spaces to be used for different ranges of musical tempos. For instance, a first state space of the model can be used when no music is detected, a second state space can be used when a tempo below a first threshold is detected, and a third state space can be used when a tempo above the first threshold is detected. This allows pseudo-random animation to be additionally synchronized with different situations and allows for customization of motion tailored to different audio data (e.g., “dance” movements or combinations of movements from a specific state space) within a single implementation of a computer animation model executed on the device.

[0112] Once the designer has finalized the configurable options for the computer animation model, the final model data can be obtained via messaging server system 108 or messaging client application 104. In some implementations, a user of messaging client application 104 can access the model data and include it in a message sent via messaging server system 108 to another client device. When receiving client device 102 plays the message, the model is implemented on receiving client device 102 via messaging client application 104 to generate a video display. If no audio is detected, or if no threshold audio characteristics are present, receiving device 102 can simply display the skin or other aspects of the computer animation model with default animation or no animation on the display. If audio data is present, receiving client device analyzes the audio data to obtain audio characteristics, such as tempo values, and then uses the motion state and tempo values ​​of the computer animation model to begin video animation. An initial motion state is randomly selected from possible motion states based on the probability of each motion state in the data received at receiving client device. As long as audio data is present and receiving client device 102 is configured to continue video animation, the computer animation model data will randomly transition between motion states in the model's state space. The animation ends when the audio data finishes or when the output animation is stopped by user input. In other implementations, other options may be available to stop the animation, such as animation duration, a short timer for the message containing data that includes pseudo-random animation, or other such options. If the animation is part of a short message, the model data is removed from the receiving client device 102 after the message is presented along with the pseudo-random animation and the deletion trigger is satisfied, making it impossible to add additional animation using a specific computer animation model unless that model is retrieved independently or otherwise made available outside the short message.

[0113] As described above, some implementations can use different thresholds to initiate dance animation as part of a pseudo-random animation model. Some models may require a certain beat intensity or consistency. Some models may have triggers, such as an audio password to start the animation. Some models can be configured to animate only when audio data with a rhythm between a certain threshold (e.g., between 30 beats per minute and 120 beats per minute) is present.

[0114] Figure 10 An example method 1000 according to some implementations described herein is shown. Figure 10This is a flowchart illustrating a method for managing the state space of a pseudo-random computer animation model according to some example embodiments. Method 1000 may involve operations at client device 102 in conjunction with operations of message passing server system 108. In some embodiments, method 1000 is embodied in computer-readable instructions stored in a non-transitory storage device of client device 102, which, when executed by the processing circuitry of client device 102, performs method 1000.

[0115] Method 1000 begins at operation 1002, which uses one or more processors to access a computer animation model that includes one or more control points. In various implementations, this operation can be performed by a mobile device, a design computer, or any other such device. The method then proceeds to operation 1004, where one or more processors associate multiple motion patterns with a first control point among the one or more control points. This may involve the selection or creation of motion patterns during the creation of the computer animation model from a basic design without a frame. If the computer animation implementation is frame-based, this may involve selecting motion patterns from a standard set of motion patterns or adjusting motion patterns previously assigned to the computer animation model. Then, in operation 1006, one or more velocity harmonics are associated with the first control point. As described in detail above, this does not set a specific animation speed for the motion pattern, but rather a speed that will be determined later with reference to the rhythm of audio data, which is used in conjunction with the generation of an actual display of pseudo-random animation on the screen using the set of motion states defined by method 1000.

[0116] Subsequently, velocity harmonics and motion modes are selected for the control points to define elements of the state space of the computer animation model. Then, operation 1008 involves generating a set of motion states for the computer animation model (e.g., a set of motion states defined by the elements of the state space defined above), the set of motion states including motion states for each combination of motion modes and velocity harmonics in multiple motion modes and one or more velocity harmonics. In other embodiments, additional elements, such as synchronization (e.g., phase) for the display of a specific motion mode relative to a beat or audio data reference, a threshold for enabling different state space motion states for different audio environments, or other such elements, may be part of the set of motion states. Once the set of motion states (e.g., selected states of the model's state space) is defined, operation 1010 involves assigning probability values ​​to each motion state in the set of motion states, wherein an associated probability value for the associated motion state of the first control point is associated with a first probability that the display animation of the computer animation model will realize the associated motion state of the first control point.

[0117] Some such implementations may involve generating an output video of a display animation including a computer-animated model, wherein the motion of the computer-animated model is randomly generated using probability values ​​for each of a plurality of motion states. Some implementations involve receiving a set of user inputs that selects a plurality of motion patterns from motion patterns for a first control point, and one or more velocity harmonics from rhythm harmonics, and automatically assigning a corresponding probability of zero to each motion state in a subset of motion states not associated with the plurality of motion patterns. Some such implementations operate by automatically assigning equal probability values ​​to the corresponding probabilities of each motion state in a second subset of motion states associated with the user input set. Other implementations operate where the user input set also selects probability values ​​for motion states associated with the plurality of motion patterns and one or more velocity harmonics.

[0118] As described above, in some embodiments, the computer animation model includes a skeleton and skin, wherein the skeleton consists of a plurality of bones connected via one or more joints, wherein each bone includes a control point, and wherein a first control point is located at a first position on a first bone of the plurality of bones. The first bone may be a sub-bone connected to a reference point via one or more connected bones within the skeleton. A first motion pattern among a plurality of motion patterns may be an inverse kinematic motion pattern, thereby determining the first motion pattern relative to a reference point, wherein the motion of one or more connected bones within the skeleton is determined to maintain connection with one or more connected bones while realizing the first motion pattern for a first position on the first bone, and wherein the motion of one or more connected bones is automatically determined. In some such embodiments, a user input set selecting a plurality of motion patterns selects a plurality of motion patterns as inverse kinematic motion patterns presented on a user interface display as two-dimensional patterns projected onto a plane selected by the user.

[0119] Alternatively, the first motion pattern among multiple motion patterns can be a forward kinematic motion pattern, thereby determining the first motion pattern relative to the connection point with a second bone among one or more connected bones without referring to a reference point.

[0120] Various such implementations can operate in a scenario where a second bone is associated with a second plurality of motion states and a second one or more velocity harmonics, and the set of motion states also includes motion states for each combination of motion patterns and velocity harmonics of the first and second bones. Similarly, some implementations can operate in a scenario where a user input set of multiple motion patterns is selected from motion patterns displayed on a user interface as a two-dimensional pattern illustrating the motion patterns of a bone rotating around a joint. Other implementations can operate in conjunction with a user interface for selecting or creating motion patterns and assigning motion patterns to a state space in any of the ways described herein.

[0121] Figure 11 Example method 1100 according to some implementations described herein is shown. Figure 11 This is a flowchart illustrating a method for generating and displaying animations using a pseudo-random computer animation model according to certain example embodiments. Method 1100 may involve operations at client device 102 in conjunction with operations of messaging server system 108. In some embodiments, method 1100 is embodied in computer-readable instructions stored in a non-transitory storage device of client device 102, which client device 102 executes when the instructions are executed by processing circuitry of client device 102.

[0122] Method 1100 begins with operation 1102, which uses one or more processors of the device to access a computer animation model, wherein the computer animation model includes multiple motion states. Operation 1104 then involves using one or more processors to generate video output and the computer animation model on the device's display, and operation 1106 involves using one or more processors to detect audio data from the device's microphone. Based on the timing of the audio data and a specific image associated with the computer animation model, the initial frame displayed on the video output using the computer animation model may include still images or default animated images generated and applied to the video frame. In operation 1108, the audio data is processed to determine a set of audio characteristics of the audio data received at the device's microphone. As described in detail above, when music with a beat is present, a rhythm value may be detected as part of this processing, wherein the rhythm value is used to determine the animation speed of the motion pattern of the computer animation model. Operation 1110 involves one or more processors randomly selecting a first motion state from multiple motion states. Operation 1112 involves using the set of audio characteristics to generate one or more motion values ​​for the first motion state (e.g., harmonic velocity multiplied by the rhythm value to determine the frequency of repetition of the motion pattern). Then operation 1114 involves generating a video using one or more motion values ​​from a computer animation model to create animated motion within the video. Animated motion within the video includes skin motion (e.g., a skin image of a 3D model viewed from the perspective of the video or a 2D surface image of a 2D model associated with the computer animation model).

[0123] Such an implementation can operate when the audio feature set includes rhythm values, and when generating one or more motion values ​​includes selecting a motion mode speed for a first control point of a computer animation model to match a first harmonic of the rhythm values. Such an implementation can also involve periodically and randomly selecting new motion states from multiple motion states, generating one or more new motion values ​​for the new motion state using the audio feature set, and updating the video with the new motion state and one or more new motion values ​​to change the animated actions in the video.

[0124] Some implementations also involve updating an audio feature set over time in response to changes in audio data, and updating one or more motion values ​​in response to changes in the audio feature set over time. In some such implementations, the computer animation model is associated with one or more audio animation criteria used to initiate animated actions in the video. Furthermore, in some implementations, the method may also involve comparing the audio feature set with one or more audio animation criteria and selecting a default state of the computer animation model when the audio feature set does not meet one or more audio animation criteria, wherein the default state is not included among the multiple motion states. As mentioned above, other criteria can be used to initiate animation or different animation state spaces can be selected based on audio data analysis and matched to the motion state space designed for the computer animation model.

[0125] In some implementations, the audio feature set includes audio energy values, and a first motion state is randomly selected from a subset of multiple motion states based on the audio energy values, wherein the subset of multiple motion states includes audio energy matching features associated with the audio energy values. In some such implementations, the audio feature set also includes beat values, rhythm values, melody values, and danceability values. Similarly, in some implementations, the audio energy values ​​and danceability values ​​are based on a fundamental frequency value associated with the melody of the audio data and a beat consistency value over time. Other implementations may use other such combinations or other data features.

[0126] Figure 12 Example method 1200 is shown according to some implementations described herein. Figure 12 This is a flowchart illustrating a method for managing the state space of a pseudo-random computer animation model according to some example embodiments. Method 1200 may involve operations at client device 102 in conjunction with operations of message passing server system 108. In some embodiments, method 1200 is embodied in computer-readable instructions stored in a non-transitory storage device of client device 102, which client device 102 executes when the instructions are executed by the processing circuitry of client device 102.

[0127] Method 1200 begins with operation 1202, which uses one or more processors of the device to identify multiple animation elements in the computer model. Then, in operation 1204, animation elements (e.g., control points or fixed motion elements, such as skeletons or two-dimensional patterns with associated motion) are selected. In operations 1205 and 1206, motion patterns(s) and velocity harmonics(s) are associated with the selected animation elements. In operation 1207, this operation is repeated until a check confirms that all animation elements have been set in the state-space description used for the implementation of the computer model. Then, in operation 1208, a set of motion data values ​​is generated, which includes state-space descriptions of multiple motion patterns and multiple velocity harmonics for the multiple animation elements in the computer model, and in operation 1210, a probability is assigned to each value in the set of motion data values ​​for the state-space description. This data, including the assigned probabilities, can then be distributed and used to create the pseudo-random animation described herein.

[0128] In various implementations, the probability of each value in the motion data value set is selected via user input through an overlay management interface configured to suppress unselected values ​​in the state space, thereby creating a selected state space based on the probability of motion combinations for multiple motion patterns, matched against the dance motion value set. In some such implementations, the computer model and multiple animation elements are generated by processing images using an overlay template. Then, some such implementations operate by: generating an output image on a user device's display using the image, computer model, and multiple animation elements; processing audio input to identify a set of audio characteristics of the audio data received at the user device's microphone; and animate the output image using a skinned model and the motion data value set (including the probability of each value in the motion data value set).

[0129] Similarly, some implementations also involve a configuration for identifying multiple motion modes, including receiving user selections of multiple motion modes from a system set of motion modes via user interface input over an overlay management interface, wherein a subset of the system set of motion modes is selected for each of a plurality of animation elements, and such implementations can also operate in the case where identifying multiple velocity harmonics includes selecting velocity harmonics for each of the multiple motion modes by the user selection, such that the state space description of the multiple motion modes includes the selected combination of motion mode and velocity harmonic for each of the multiple animation elements.

[0130] In some implementations, assigning a probability to each value in the set of motion data values ​​described for the state space may involve: displaying a first animation, including a first animation element animated for a selected rhythm via a first combination of a first motion mode and a first velocity harmonic; and receiving user input assigning corresponding probabilities to the first animation element, the first motion mode, and the first velocity harmonic.

[0131] Some implementations may operate by assigning probabilities by one or more processors to each value in a set of motion data values ​​described for a state space, involving the following operations: for each of a plurality of animation elements and for each of a selected combination of motion patterns and velocity harmonics for each animation element: displaying an output animation of a computer model using a corresponding combination of motion patterns and velocity harmonics for each corresponding animation element; and receiving corresponding user input assigning corresponding probabilities to generate probabilities for each value in the set of motion data values ​​described for a state space. In some such implementations, the set of audio characteristics includes rhythm values ​​and danceability values, and the danceability values ​​may involve values ​​derived from beat consistency scores and at least one musical energy criterion. In other implementations, other criteria for such audio characteristics may be used.

[0132] In some implementations, the output image is animated during a first time period when the audio data meets at least one music energy standard, and the output image is not animated during a second time period when the audio data does not meet at least one music energy standard.

[0133] In some implementations, the audio feature set may include multiple audio energy features, wherein each of the multiple audio energy features is associated with a different probability corresponding to each value in a set of motion data values ​​describing the state space to match different audio energy features and different motion values ​​in the state space.

[0134] Some implementations involve generating a standardized set of audio energy values ​​based on a standard set of audio input characteristics; selecting a minimum audio energy threshold and a maximum audio energy threshold; calculating model motion energy for each value in the set of motion data values ​​described for the state space using a first motion model; matching the first motion model to the standardized set of audio energy values; and assigning zero probability to the set of values ​​in the set of motion data values ​​described for the state space whose corresponding model motion energy is less than the minimum audio energy threshold or greater than the maximum audio energy threshold when matched from the first motion model to the standardized set of audio energy values. Other implementations may use other such threshold configurations to determine pseudo-random animation of computer animation models as described herein.

[0135] Figure 13Example method 1300 according to some implementations described herein is shown. Figure 13 This is a flowchart illustrating a method for designing and selecting a management state space for a pseudo-random computer animation model, according to some example embodiments. Method 1300 may involve operations at client device 102 in conjunction with operations of messaging server system 108. In some embodiments, method 1300 is embodied in computer-readable instructions stored in a non-transitory storage device of client device 102, which client device 102 executes method 1300 when the instructions are executed by the processing circuitry of client device 102.

[0136] Method 1300 begins with operation 1302, which uses one or more processors of a computing device to generate image modification data including a computer-animated model. This image modification data is configured to modify frames of a video image to insert and animate the computer-animated model within the frames of the video image. The computer-animated model in the image modification data includes one or more control points. Following operation 1302, operation 1304 involves automatically associating multiple motion patterns and one or more velocity harmonics with one or more control points using one or more processors of the computing device. In other embodiments, other elements may be associated with the control points, or existing associations may be modified.

[0137] Operation 1306 involves automatically generating multiple motion states of a computer animation model using multiple motion modes, one or more control points, and one or more velocity harmonics. In such an implementation, an initial set of motion states can be automatically assigned, with additional modifications made in response to designer criteria, designer selections, specific modifications to individual motion states or state space elements, or other such operations. Operation 1308 then involves automatically assigning probability values ​​to each of the multiple motion states, wherein each of the multiple motion states includes a velocity harmonic of each of one or more control points of the computer animation model and a motion mode of each of one or more control points of the computer animation model. Automatic assignment may involve assigning certain probability values, or a default set of probabilities, to certain motions or combinations of motions. Such probabilities can then be updated based on designer selections.

[0138] In some implementations, the velocity harmonics for each motion state are configured to set the animation speed of the motion pattern to repeat with respect to the harmonics of the rhythm values ​​of audio data collected by a user device performing image modification data. Example implementations may operate with the harmonics of the rhythm values ​​selected from 1, 2, 4, 0.5, 0.25, and 0.125 times the rhythm value. Other implementations may use any harmonics of designer choice to match complex motion with audio data.

[0139] In some implementations, the probabilistic selection user interface can be used by a designer to select probabilities for the state space using an interface on a computing device's display. In some such implementations, the probabilistic selection user interface includes an animation window comprising an animated video of a computer-animated model of a motion state, and selectable probability weights for that motion state. Similarly, in some implementations, the probabilistic selection user interface displays multiple animation windows, each associated with a corresponding motion state and a corresponding selectable probability weight for that motion state. In some such implementations, the probabilistic selection user interface also includes a filter input for sorting the motion states displayed within the probabilistic selection user interface, and in some such implementations, the filter input sorts the displayed motion states based on one or more of motion type, harmonic velocity, and control points.

[0140] In various implementations, each motion state may be associated with a selectable energy threshold, such that the corresponding selectable probability weight for the corresponding motion state is based on audio characteristics of audio data used by image modification data to animate an intra-frame computer animation model of a video image. In such implementations, the probability selection user interface may further include an energy threshold input for each motion state, and one or more selectable audio energy samples. In some such implementations, the computer animation model includes a two-dimensional overlay generated by analyzing the content of a frame of a video image and replacing portions of the frame of the video image with one or more animated elements. In other implementations, the computer animation model includes an overlay generated by analyzing the content of a frame of a video image and replacing portions of the frame of the video image with a representation of a three-dimensional model including skin on a skeletal model, the skeletal model including one or more control points.

[0141] Various example implementations and methods have been described above. It should be understood that although a specific set of structures and operations has been described, it is possible to introduce or repeat structures and operations within the scope of the implementations described herein, and the examples specifically described are not exhaustive.

[0142] Software Architecture

[0143] Figure 14 This is a block diagram illustrating example software architecture 1406, which can be used in conjunction with various hardware architectures described herein. Figure 14 This is a non-limiting example of a software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. Software architecture 1406 can be implemented in, for example... Figure 15 It runs on the hardware of the 1500 machine. Figure 15 The machine 1500 includes a processor 1504, a memory 1514, and I / O components 1518, etc. A representative hardware layer 1452 is shown and can represent, for example... Figure 15 The machine 1500 in the document. A representative hardware layer 1452 includes a processing unit 1454 having associated executable instructions 1404. The executable instructions 1404 represent executable instructions of the software architecture 1406, including implementations of the methods, components, etc., described herein. Hardware layer 1452 also includes a memory and / or storage module memory / storage device 1456 that also has executable instructions 1404. Hardware layer 1452 may also include other hardware 1458.

[0144] exist Figure 14 In the example architecture, software architecture 1406 can be conceptualized as a stack of layers, each providing a specific function. For example, software architecture 1406 may include layers such as operating system 1402, library 1420, application 1416, and presentation layer 1414. Operationally, application 1416 and / or other components within a layer can activate application programming interface (API) calls 1408 through the software stack and receive messages 1412 in response to API calls 1408. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide a framework / middleware 1418, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

[0145] Operating system 1402 can manage hardware resources and provide public services. Operating system 1402 may include, for example, a kernel 1422, services 1424, and drivers 1426. Kernel 1422 can act as an abstraction layer between hardware and other software layers. For example, kernel 1422 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 1424 can provide other public services to other software layers. Drivers 1426 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 1426 may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0146] Library 1420 provides common infrastructure used by application 1416 and / or other components and / or layers. Library 1420 provides functionality that allows other software components to perform tasks more easily than by directly interfacing with the functions of the underlying operating system 1402 (e.g., kernel 1422, service 1424, and / or driver 1426). Library 1420 may include system libraries 1444 (e.g., the C standard library), which provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1420 may include API libraries 1446, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., the OpenGL framework for rendering 2D and 3D graphics content on a display), database libraries (e.g., SQLite providing various relational database functions), network libraries (e.g., WebKit providing web browsing functionality), and so on. Library 1420 may also include various other libraries 1448 to provide many other APIs to application 1416 and other software components / modules.

[0147] The framework / middleware 1418 (sometimes also called middleware) provides a higher level of common infrastructure that can be used by application 1416 and / or other software components / modules. For example, the framework / middleware 1418 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 1418 can provide a wide range of other APIs that can be used by application 1416 and / or other software components / modules, some of which may be specific to a particular operating system 1402 or platform.

[0148] Application 1416 includes built-in application 1438 and / or third-party application 1440. Examples of representative built-in applications 1438 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1440 may include those using Android by entities other than the platform-specific vendor. TM or iOS TM Applications developed using a Software Development Kit (SDK) can be used on platforms such as iOS. TM ANDROID TM Mobile software running on the Windows® Phone mobile operating system or other mobile operating systems. Third-party application 1440 may activate API calls 1408 provided by the mobile operating system (e.g., operating system 1402) to facilitate the functions described herein.

[0149] Application 1416 can use built-in operating system functions (e.g., kernel 1422, service 1424, and / or driver 1426), libraries 1420, and frameworks / middleware 1418 to create a user interface for interacting with the system's user. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 1414. In these systems, the application / component "logic" may be separated from the user-interacting aspects of the application / component.

[0150] Figure 15 This is a block diagram illustrating components of a machine 1500 according to some example embodiments, the machine 1500 being capable of reading instructions 1404 from a machine-readable medium (e.g., a machine-readable storage medium) and executing any or more of the methods discussed herein. Specifically, Figure 15A graphical representation of machine 1500 is shown as an example of a computer system, within which instructions 1510 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1500 to perform any or more of the methods discussed herein. Therefore, the instructions 1510 can be used to implement the modules or components described herein. Instructions 1510 transform the general, non-programmable machine 1500 into a specific machine 1500 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 1500 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1500 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1500 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web home appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1510 specifying actions to be taken by machine 1500. Furthermore, although only a single machine 1500 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 1510 to perform any or more of the methods discussed herein.

[0151] Machine 1500 may include a processor 1504, a memory / storage device 1506, and I / O components 1518 that can be configured to communicate with each other, for example, via bus 1502. Memory / storage device 1506 may include memory 1514 and storage units 1516, such as main memory or other memory storage devices, which the processor 1504 can access, for example, via bus 1502. Storage units 1516 and memory 1514 store instructions 1510 embodying any or more of the methods or functions described herein. Instructions 1510 may also reside wholly or partially within memory 1514, storage units 1516, at least one processor 1504 (e.g., the processor's cache memory), or any suitable combination thereof during execution by machine 1500. Therefore, memory 1514, storage units 1516, and the memory of processor 1504 are examples of machine-readable media.

[0152] I / O component 1518 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O component 1518 included in a particular machine 1500 will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 1518 may include... Figure 15 Many other components are not shown. The I / O components 1518 are grouped by function only for the purpose of simplifying the following discussion, and this grouping is by no means limiting. In various example embodiments, the I / O components 1518 may include output components 1526 and input components 1528. Output components 1526 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 1528 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input; photo-optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0153] In other example implementations, I / O component 1518 may include biometric component 1530, motion component 1534, environmental component 1536, or positioning component 1538, as well as various other components. For example, biometric component 1530 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1534 may include accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1536 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), a sound sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases to ensure safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1538 may include a position sensor component (e.g., a Global Positioning System (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0154] Various technologies can be used to implement communication. I / O component 1518 may include communication component 1540, which is operable to couple machine 1500 to network 1532 or device 1520 via coupling 1524 and coupling 1522, respectively. For example, communication component 1540 may include network interface component or other suitable device to interface with network 1532. In other examples, communication component 1540 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, Bluetooth® component (e.g., Bluetooth® Low Energy), Wi-Fi® component, and other communication components that provide communication via other forms. Device 1520 may be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via Universal Serial Bus (USB)).

[0155] Furthermore, the communication component 1540 can detect identifiers or may include components operable to detect identifiers. For example, the communication component 1540 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be obtained via the communication component 1540, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location by detecting NFC beacon signals that can indicate a specific location, etc.

[0156] Glossary

[0157] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions 1510 executed by machine 1500, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions 1510. Instructions 1510 can be transmitted or received over network 1532 using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0158] In this context, "client device" refers to any machine 1500 that interfaces with communication network 1532 to obtain resources from one or more server systems or other client devices 102. Client device 102 may be, but is not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, multiple laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access network 1532.

[0159] In this context, "communication network" refers to one or more portions of network 1532, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 1532 or a portion of a network may include a wireless network or a cellular network, and coupling 1524 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling 1524 can implement any of the various types of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate evolution of GSM (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards setting organizations, other telematics protocols, or other data transmission technologies.

[0160] In this context, a "brief message" refers to a message that can be accessed for a limited time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0161] In this context, "machine-readable medium" means a component, device, or other tangible medium capable of temporarily or permanently storing instructions 1510 and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)) and / or any suitable combination thereof. The term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions 1510 (e.g., a centralized or distributed database 120, or associated caches and servers). The term "machine-readable medium" should also be considered to include any medium or combination of media capable of storing instructions 1510 (e.g., code) executable by machine 1500 such that, when executed by one or more processors 1504 of machine 1500, the instructions 1510 cause the machine to perform any or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" does not include the signal itself.

[0162] In this context, a “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, application programming interfaces (APIs), or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function within a related function. Components can constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry systems that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor 1508. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) uniquely tailored to perform the configured function, and is no longer the general-purpose processor 1508. It will be understood that cost and time considerations can drive the decision to implement hardware components mechanically in dedicated and permanently configured circuitry systems or in temporarily configured circuitry systems (e.g., configured by software). Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Considering the implementation where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor 1508 that is configured as a dedicated processor via software, the general-purpose processor 1508 can be configured as a different dedicated processor (e.g., including different hardware components) at different times. The software accordingly configures a particular processor 1508 or processor 1504 to constitute a particular hardware component at one time and a different hardware component at different times.Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission between two or more hardware components (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. Other hardware components can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed at least in part by one or more processors 1504 that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors 1504 can constitute components of a processor implementation that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors 1504. Similarly, the methods described herein can be implemented at least in part by processors, where one or more specific processors 1504 are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors 1504 or processor-implemented components. Furthermore, one or more processors 1504 can also be configured to support the performance of related operations in a "cloud computing" environment or operate as "Software as a Service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of machine 1500 including processor 1504), where these operations can be accessed via network 1532 (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)). The execution of certain operations can be distributed among processors 1504, residing not only within a single machine 1500 but also deployed across multiple machines. In some example implementations, processor 1504 or processor-implemented components can reside in a single geographic location (e.g., within a home environment, an office environment, or a server cluster). In other example implementations, the processor 1504 or the components implemented by the processor may be distributed across multiple geographical locations.

[0163] In this context, "processor" refers to any circuit or virtual circuit (physical circuitry simulated by logic executed on the actual processor 1508) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals used to operate machine 1500. For example, processor 1508 can be a central processing unit (CPU), a simplified instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. Processor 1508 can also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0164] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when an event occurred, such as giving a date and time of day, sometimes accurate to a fraction of a second.

Claims

1. A method comprising: Accessing a computer animation model using one or more processors, the computer animation model comprising one or more control points; Multiple motion patterns are associated with a first control point among the one or more control points through the one or more processors; One or more velocity harmonics are associated with the first control point through one or more processors; Generate a set of motion states for the computer animation model, the set of motion states including motion states for each combination of motion modes and velocity harmonics in the plurality of motion modes and one or more velocity harmonics; A probability value is assigned to each motion state in the set of motion states, wherein the associated probability value of the motion state associated with the first control point is associated with a first probability of the associated motion state of the first control point in the display animation of the computer animation model; and Generate an output video of a display animation including the computer animation model, the display animation having motions of the computer animation model randomly generated using probability values ​​for each of the plurality of motion states. The computer animation model includes A control points, B motion patterns, and C rhythmic harmonics, resulting in the following number of motion states for the computer animation model: N = (BxC) A , Where N is the number of motion states, and each motion state M is assigned a corresponding probability, such that the probability mass function P() assigned to each motion state is: P(M1, M2, M3…M… N-1 M N = 1.

2. The method according to claim 1, further comprising: Receive a set of user inputs, wherein the set of user inputs selects the plurality of motion modes of the first control point from the B motion modes, and selects one or more velocity harmonics from the C rhythm harmonics; and The probability of each motion state in a subset of motion states that are not associated with the plurality of motion patterns is automatically assigned to zero.

3. The method according to claim 2, further comprising: Automatically assign equal probability values ​​to the corresponding probabilities of each motion state in a second subset of motion states associated with the user input set.

4. The method according to claim 2, wherein, The user input set also selects probability values ​​for motion states associated with the plurality of motion patterns and the one or more velocity harmonics.

5. The method according to claim 2, wherein, The computer animation model includes a skeleton and skin, the skeleton including a plurality of bones connected via one or more joints, wherein each bone includes control points; The first control point includes a first position on a first bone among the plurality of bones.

6. The method according to claim 5, wherein, The first bone is a sub-bone connected to the reference point via one or more connected bones within the skeleton.

7. The method according to claim 6, wherein, The first motion pattern of the plurality of motion patterns is an inverse kinematic motion pattern, thereby determining the first motion pattern relative to the reference point, wherein the motion of the one or more connected bones within the skeleton is determined to maintain connection with the one or more connected bones while realizing the first motion pattern for the first position on the first bone, and wherein the motion of the one or more connected bones is automatically determined.

8. The method according to claim 7, wherein, The user input set of the multiple motion modes selected is selected as the inverse kinematic motion mode presented on the user interface display, and the multiple motion modes are projected onto the two-dimensional mode in the plane selected by the user.

9. The method according to claim 6, wherein, The first of the plurality of motion patterns is a forward kinematic motion pattern, thereby determining the first motion pattern relative to the connection point of the second bone in the one or more connected bones without referring to the reference point.

10. The method according to claim 9, wherein, The second bone is associated with a second plurality of motion states and a second or more velocity harmonics; and The set of motion states further includes motion states for each combination of motion patterns and velocity harmonics for the first and second bones.

11. The method according to claim 9, wherein, The user input set for selecting the plurality of motion modes is selected from the motion modes on the user interface display as a two-dimensional pattern showing the motion patterns of the skeleton rotating around the joint.

12. An apparatus comprising: A memory configured to store data of a computer animation model, the computer animation model including one or more control points; as well as One or more processors coupled to the memory; as well as A display, coupled to one or more processors, The one or more processors are configured to: Associate multiple motion patterns with a first control point among the one or more control points; Associate one or more velocity harmonics with the first control point; Generate a set of motion states for the computer animation model, the set of motion states including motion states for each combination of motion modes and velocity harmonics in the plurality of motion modes and one or more velocity harmonics; A probability value is assigned to each motion state in the set of motion states, wherein the associated probability value of the motion state associated with the first control point is associated with a first probability of the associated motion state of the first control point in the display animation of the computer animation model; and The generation of an output video is initiated for presentation on the display. The output video includes a display animation of the computer-animated model, wherein the display animation has motions of the computer-animated model randomly generated using probability values ​​for each of the plurality of motion states. The computer animation model includes A control points, B motion patterns, and C rhythmic harmonics, resulting in the following number of motion states for the computer animation model: N = (BxC) A , Where N is the number of motion states, and each motion state M is assigned a corresponding probability, such that the probability mass function P() assigned to the probability value of each motion state is: P(M1, M2, M3 … M N-1 M N = 1.

13. A non-transitory computer-readable medium comprising instructions that, when executed by a processing circuitry system of a device, cause the device to perform operations of a method, the operations comprising: Access a computer animation model, which includes one or more control points; Associate multiple motion patterns with a first control point among the one or more control points; Associate one or more velocity harmonics with the first control point; Generate a set of motion states for the computer animation model, the set of motion states including motion states for each combination of motion modes and velocity harmonics in the plurality of motion modes and one or more velocity harmonics; A probability value is assigned to each motion state in the set of motion states, wherein the associated probability value of the motion state associated with the first control point is associated with a first probability of the associated motion state of the first control point in the display animation of the computer animation model; and Generate an output video of a display animation including the computer animation model, the display animation having motions of the computer animation model randomly generated using probability values ​​for each of the plurality of motion states. The computer animation model includes A control points, B motion patterns, and C rhythmic harmonics, resulting in the following number of motion states for the computer animation model: N = (BxC) A , Where N is the number of motion states, and each motion state M is assigned a corresponding probability, such that the probability mass function P() assigned to each motion state is: P(M1, M2, M3…M… N-1 M N = 1.

14. The non-transitory computer-readable medium according to claim 13, wherein, The computer animation model also includes a skeleton and skin, the skeleton comprising a plurality of bones connected via one or more joints, wherein each bone includes control points; The first control point includes a first position on a first bone among the plurality of bones.

15. The non-transitory computer-readable medium according to claim 14, wherein, The first bone is a sub-bone connected to the reference point via one or more connected bones within the skeleton; as well as Wherein, the first motion mode of the plurality of motion modes is an inverse kinematic motion mode, thereby determining the first motion mode relative to the reference point, wherein the motion of the one or more connected bones within the skeleton is determined to maintain connection with the one or more connected bones while realizing the first motion mode for the first position on the first bone, and wherein the motion of the one or more connected bones is automatically determined.

16. The non-transitory computer-readable medium according to claim 14, wherein, The first bone is a sub-bone connected to the reference point via one or more connected bones within the skeleton; as well as The first motion pattern among the plurality of motion patterns is a forward kinematic motion pattern, which is determined relative to the connection point of the second bone among the one or more connected bones without referring to the reference point.

Images