Generating lip-sync augmented reality effects

Abstract

An audio track with human speech is played back using a device having a display screen displaying a video feed from a camera. A position of a mouth depicted in the video feed is detected. A playback timestamp of the audio track is compared to visual element-time stamp data of the audio track to identify a visual element corresponding to the timestamp of the audio playback, and the visual element is positioned at the position of the mouth detected in the video feed.

Description

Priority Statement

[0001] This patent application claims the benefit of priority to U.S. Application No. 18 / 609,960, filed March 19, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to augmented reality effects applied to video feeds including faces, and more particularly, to providing facial expressions to faces presented in the video feed in conjunction with an audio track being played back. Background Technology

[0003] Interactive system applications provide a medium for sharing user content such as photos or videos. In some instances, photos or videos can be supplemented with augmented reality (AR) effects or other effects that are generated in real time on the camera feed and displayed on the mobile device's screen for preview. Users can select and manipulate the effects to be applied to the live camera feed and, if satisfied, capture images including these effects or record videos including them. The captured videos or photos can then be shared with the user's contacts and friends on the interactive platform. In some examples, augmented reality effects can also be applied to live video streams or captured video streams to provide interaction between the user and their "friends" on the platform, or to provide an interactive one-on-one experience.

[0004] AR effects encompass various image processing operations corresponding to image modification, filters, media overlay, and transformations, which can be used to provide such interactive and video stream modification techniques. In some examples, these image processing operations provide interactive experiences of real-world environments, where real-world objects, surfaces, backgrounds, lighting, etc., are enhanced using computer-generated perceptual information. AR effects involve applying a selected AR experience to a set of data, parameters, and other assets required for an image or video feed. Attached Figure Description

[0005] In the accompanying drawings (not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits in the reference numerals indicate the drawing number in which that element was first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0006] Figure 1 It is a graphical representation of a networked environment in which the content of this disclosure can be deployed, based on some examples.

[0007] Figure 2 It is a graphical representation of a messaging system with both client-side and server-side functionalities, based on some examples.

[0008] Figure 3 It is a graphical representation based on examples such as data structures maintained in a database.

[0009] Figure 4 A system for implementing lip-synching is shown, based on some examples.

[0010] Figure 5A The image shows a user interface for use with lip-sync, based on some examples.

[0011] Figure 5B One aspect of the subject matter according to one implementation is shown.

[0012] Figure 6 This is a flowchart illustrating lip-syncing methods based on some examples.

[0013] Figure 7 The machine learning pipeline is shown based on some examples.

[0014] Figure 8 The training and use of machine learning programs are illustrated using some examples.

[0015] Figure 9 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0016] Figure 10 It is a block diagram illustrating a software architecture that can be implemented as an example. Detailed Implementation

[0017] Singing along to favorite songs is a common practice. Lip-syncing to a song while filming a performance video is fun, but difficult to achieve unless the user has a good understanding of the lyrics and rhythm. This is especially true for new songs, contrasting with older ones. This article discloses methods and systems for providing user-generated lip-syncing in video feeds, which can be used simply for singing along, or for capturing and sharing.

[0018] This functionality is achieved by animing the user's face (particularly the lips) in the video stream while the song is playing, based on a song-specific data file that includes pixel identifiers and corresponding timestamps. The pixel identifiers and timestamps specify which user expressions to use and when, ensuring the user's appearance is synchronized with the song's lyrics. In some examples, the facial animation is a graphical manipulation of the user's facial appearance itself, while in others, overlays or other image manipulations may be provided.

[0019] Although song lyrics are written as text, they are sung as a sequence of phonemes. A phoneme is the smallest unit of sound in a language that distinguishes words from each other. Phonemes are the basic building blocks of spoken language, used to convey meaning. Phonemes are different from letters or glyphs, which represent the written form of language.

[0020] Each phoneme is associated with a corresponding viseme. A viseme is the visual unit of speech, corresponding to the different shapes and movements of the mouth and face when a specific speech sound is produced. Vises focus on the visual aspects of speech, particularly the lip and facial movements associated with different sounds. To a large extent, vises are independent of language; therefore, sounds in any language can generally be represented by the same viseme.

[0021] The number of visual pixels can vary depending on the language and the specific analytical context. Generally, the concept of visual pixels is used to group similar lip and facial movements associated with certain phonemes together. In English, ten to fifteen visual pixels are sufficient to represent the visual aspects of speech. However, the exact number can vary, and some systems use fewer or more categories of visual pixels.

[0022] To achieve lip-sync for any song as described in more detail below, the song needs to be analyzed to generate a time series of pixel identifiers. These pixel identifiers are then provided to an application that identifies faces in the video stream, and visual effects are applied to the video stream based on the pixel identifiers at the appropriate time and location in the video stream (i.e., corresponding to the user's lower face, specifically the lips).

[0023] Networked computing environment

[0024] Figure 1 This is a block diagram illustrating an example interactive system 100 for facilitating interactions on a network, such as exchanging text messages, making text audio and video calls, or playing games. Interactive system 100 includes multiple user systems 102, each of which hosts multiple applications including an interactive client 104 and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on corresponding other user systems 102), an interactive server system 110, and a third-party server 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 may also communicate with the locally hosted applications 106 using an application programming interface (API).

[0025] Each user system 102 may include multiple user devices, such as mobile devices 114, head-mounted wearable devices 116, and computer client devices 118, which are communicatively connected to exchange data and messages.

[0026] Interactive client 104 interacts with other interactive clients 104 and with interactive server system 110 via network 108. The data exchanged between interactive clients 104 (e.g., interaction 120) and between interactive client 104 and interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] Interactive server system 110 provides server-side functionality to interactive client 104 via network 108. While some functions of interactive system 100 are described herein as being performed by interactive client 104 or interactive server system 110, the location of certain functions—whether within interactive client 104 or interactive server system 110—may be a design choice. For example, it might be technically preferred that specific technologies and functions are initially deployed within interactive server system 110, but later migrated to interactive client 104 of user system 102 with sufficient processing power.

[0028] The interactive server system 110 supports various services and operations provided to the interactive client 104. Such operations include sending data to and receiving data from the interactive client 104, and processing data generated by the interactive client 104. This data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, entity relationship information, and live event information. Data exchange within the interactive system 100 is activated and controlled via functions available through the user interface (UI) of the interactive client 104.

[0029] Specifically, the focus now shifts to interactive server system 110. Application Programming Interface (API) server 122 is coupled to interactive server 124 and provides it with a programming interface, making the functionality of interactive server 124 accessible to interactive client 104, other applications 106, and third-party server 112. Interactive server 124 is communicatively coupled to database server 126, thereby facilitating access to database 128, which stores data associated with the interactions processed by interactive server 124. Similarly, web server 130 is coupled to interactive server 124 and provides a web-based interface to interactive server 124. For this purpose, web server 130 handles incoming network requests via Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0030] Application Programming Interface (API) server 122 receives and sends interactive data (e.g., command and message payloads) between interactive server 124 and user system 102 (as well as interactive client 104 and other applications 106) and third-party server 112. Specifically, API server 122 provides a set of interfaces (e.g., routines and protocols) that interactive client 104 and other applications 106 can invoke or query to activate the functionality of interactive server 124. Application Programming Interface (API) server 122 exposes various functions supported by interaction server 124, including account registration; login functionality; sending interactive data from one interaction client 104 to another interaction client 104 via interaction server 124; transferring media files (e.g., images or videos) from interaction client 104 to interaction server 124; setting up a collection of media data (e.g., stories); retrieving a user's friend list for user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity relationship graph (e.g., entity graph 310); locating friends in the entity relationship graph; and opening (e.g., application events associated with interaction client 104).

[0031] Interactive server 124 hosts multiple systems and subsystems, as shown below. Figure 2 describe.

[0032] System Architecture

[0033] Figure 2 This is a block diagram illustrating further details of an interactive system 100 according to some examples. Specifically, the interactive system 100 is shown to include an interactive client 104 and an interactive server 124. The interactive system 100 includes multiple subsystems supported on the client side by the interactive client 104 and on the server side by the interactive server 124. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) may have components that enable it to operate independently and communicate with other services. Example components of a microservice subsystem may include: Functional logic: Functional logic implements the functions of the microservice subsystem and represents the specific capabilities or functions provided by the microservice. API Interface: Microservices can use lightweight protocols such as REST or messaging to communicate with other components through well-defined APIs or interfaces. The API interface defines the input and output of the microservice subsystem, as well as the way the microservice subsystem interacts with other microservice subsystems of the interactive system 100. Data storage device: The microservice subsystem can be responsible for its own data storage device, which can be in the form of a database, cache, or other storage mechanism (e.g., using database server 126 and database 128). This allows the microservice subsystem to operate independently of other microservices in the interactive system 100. Service discovery: Microservice subsystems can find other microservice subsystems in the interactive system 100 and communicate with them. The service discovery mechanism enables microservice subsystems to locate and communicate with other microservice subsystems in a scalable and efficient manner. Monitoring and logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable the tracking of the health and performance of microservice subsystems.

[0034] In some examples, the interactive system 100 may adopt a monolithic architecture, a service-oriented architecture (SOA), a function-as-a-service (FaaS) architecture, or a modular architecture.

[0035] The example subsystem is discussed below.

[0036] The image processing system 202 provides various functions that enable users to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0037] The camera device system 204 includes (e.g., in a camera device application) control software that (e.g., directly or via operating system controls) interacts with and controls the hardware camera device hardware of the user system 102 to modify and enhance real-time images captured and displayed via the interactive client 104.

[0038] Enhancement system 206 provides functionality related to the generation and distribution of enhancements (e.g., media overlays) for images captured in real-time by the camera device of user system 102 or retrieved from the memory of user system 102. For example, enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for interactive client 104 to enhance real-time images received via camera device system 204 or stored images retrieved from memory 906 of user system 102. These enhancements are selected and presented to the user of interactive client 104 by enhancement system 206 based on multiple inputs and data, such as: The geographical location of user system 102; and User entity relationship information of users in user system 102.

[0039] Enhancements may include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) at user system 102 for transmission in messages, or to video content such as video content streams or feeds sent from interactive client 104. Therefore, image processing system 202 can interact with and support various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0040] Media overlays may include text or image data that can be superimposed on photographs taken by user system 102 or video streams produced by user system 102. In some examples, media overlays may be location overlays (e.g., Venice Beach), names of on-site events, or names of businesses (e.g., beach cafes). In other examples, image processing system 202 uses the geographic location of user system 102 to identify media overlays that include the names of businesses located at the geographic location of user system 102. Media overlays may include additional tags associated with businesses. Media overlays may be stored in database 128 and accessed through database server 126.

[0041] Image processing system 202 provides a user-based publishing platform that allows users to select geographical locations on a map and upload content associated with those locations. Users can also specify which media overlays should be provided to other users. Image processing system 202 generates a media overlay that includes the uploaded content and associates it with the selected geographical location.

[0042] The augmented reality creation system 214 supports augmented reality developer platforms and includes applications for content creators (e.g., artists and developers) to create and publish augmented reality experiences (e.g., augmented reality experiences) on interactive clients 104. The augmented reality creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates.

[0043] In some examples, enhancement creation system 214 provides a merchant-based publishing platform that allows merchants to select specific enhancements associated with geographic locations via a bidding process. For instance, enhancement creation system 214 associates the media overlay of the highest-bidding merchant with a corresponding geographic location for a predefined amount of time.

[0044] Communication system 208 is responsible for implementing and processing various forms of communication and interaction within interactive system 100, and includes message sending and receiving system 210, audio communication system 216, and video communication system 212. Message sending and receiving system 210 is responsible for enabling temporary or time-limited access to content by interactive client 104. Message sending and receiving system 210 includes (e.g., within a short-duration timer system) multiple timers that selectively enable access (e.g., for presentation and display) of messages and associated content via interactive client 104 based on duration and display parameters associated with a message or set of messages (e.g., a story). Audio communication system 216 implements and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, video communication system 212 implements and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0045] User management system 218 is operationally responsible for managing user data and profiles, and maintaining entity information about the relationship between users and users of interaction system 100 (e.g., stored in entity table 308, entity diagram 310, and profile data 302).

[0046] The collection management system 220 is operationally responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of an event related to the content). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 220 is also responsible for publishing icons to the user interface of the interactive client 104, which provide notifications for specific collections. The collection management system 220 includes curatorial functions that enable collection managers to manage and curate specific content collections. For example, a curation interface allows event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 220 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users may be paid to include user-generated content in the collection. In such a case, the collection management system 220 operates to automatically pay such users for access to its content.

[0047] Map system 222 provides various geographic location (e.g., geolocation) functions and supports the presentation of map-based media content and messages via interactive client 104. For example, map system 222 enables the display (e.g., stored in profile data 302) of user icons or avatars on the map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content generated by such friends (e.g., a collection of messages including photos and videos). For example, on the map interface of interactive client 104, a message posted by a user from a specific geographic location to interactive system 100 can be displayed to the specific user's "friends" at that specific location within the context of the map. Users can also share their location and status information with other users of interactive system 100 via interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of interactive client 104.

[0048] Game system 224 provides various game functions within the context of interactive client 104. Interactive client 104 provides a game interface that offers a list of available games that a user can launch and play with other users of interactive system 100 within the context of interactive client 104. Interactive system 100 also enables specific users to invite other users to play specific games by sending invitations from interactive client 104. Interactive client 104 also supports sending and receiving audio, video, and text messages (e.g., chat) within the context of playing the game, provides leaderboards for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0049] External resource system 226 provides interactive client 104 with an interface to communicate with remote servers (e.g., third-party server 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts applications or smaller versions of applications (e.g., game applications, utility applications, payment applications, or ride-sharing applications) based on markup languages ​​(e.g., HTML5). Interactive client 104 can launch web-based resources (e.g., applications) by accessing HTML5 files from the third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by interactive server 124. The SDK includes an application programming interface (API) with functionality that can be invoked or activated by the web-based application. Interactive server 124 hosts a JavaScript library that provides access to a given external resource for specific user data of interactive client 104. HTML5 is an example of a technology used for programming games, but applications and resources programmed using other technologies can be used.

[0050] The advertising system 228 enables third parties to purchase advertisements to be presented to end users via the interactive client 104, and also handles the delivery and presentation of these advertisements.

[0051] Artificial intelligence and machine learning system 230 provides various services to different subsystems within interaction system 100. For example, AI and machine learning system 230 operates in conjunction with image processing system 202 and camera device system 204 to analyze images and extract information such as objects, text, or faces. This information can then be used by image processing system 202 to enhance, filter, or manipulate images. AI and machine learning system 230 can be used by enhancement system 206 to generate enhanced content and augmented reality experiences, such as adding virtual objects or animations to real-world images. Communication system 208 and messaging system 210 can use AI and machine learning system 230 to analyze communication patterns and provide insights into how users interact with each other, and provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic. AI and machine learning system 230 can also provide chatbot functionality for messaging interactions 120 between user systems 102 and between user systems 102 and interaction server system 110. Artificial intelligence and machine learning system 230 can also work with audio communication system 216 to provide speech recognition and natural language processing capabilities, thereby enabling users to interact with interactive system 100 using voice commands.

[0052] Data Architecture

[0053] Figure 3 This is a schematic diagram illustrating a data structure 300 that can be stored in a database 304 of an interactive server system 110, according to certain examples. Although the contents of the database 304 are shown as including multiple tables, it should be understood that data can be stored in other types of data structures, such as object-oriented databases.

[0054] Database 304 includes message data stored in message table 306. For any given message, this message data includes at least message sender data, message receiver (or recipient) data, and a payload. See below for reference. Figure 3 Further details are provided regarding information that can be included in the message and within the message data stored in message table 306.

[0055] Entity table 308 stores entity data and (for example, links to entity diagram 310 and profile data 302). Entities whose records are maintained in entity table 308 may include individuals, company entities, organizations, objects, locations, events, etc. Regardless of the entity type, any entity whose data is stored by interactive server system 110 can be an identifiable entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0056] Entity graph 310 stores information about relationships and associations between entities. As an example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as an individual user subscribing to digital content (e.g., a newspaper or other digital media export or brand) for a business or publishing user. Other relationships can be two-way, such as the "friendship" relationship between individual users of interactive system 100.

[0057] Certain licenses and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friendship between individual users) can include authorization for the posting of digital content items between the individual users, but certain restrictions or filters can be imposed on the posting of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a business user can impose varying degrees of restrictions on the posting of digital content from the business user to the individual user, and can significantly restrict or prevent the posting of digital content from the individual user to the business user. As an example of an entity, a specific user can record certain restrictions in a record for that entity within entity table 308 (e.g., via privacy settings). Such privacy settings can be applied to all types of relationships in the context of interaction system 100, or selectively applied to certain types of relationships.

[0058] Profile data 302 stores various types of profile data about a specific entity. Profile data 302 can be selectively used and presented to other users of the interactive system 100 based on privacy settings specified by the specific entity. In the case that the entity is an individual, profile data 302 includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a set of such avatar representations) selected by the user. A specific user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive system 100, and on a map interface displayed to other users by the interactive client 104. The set of avatar representations may include “status avatars,” which present a graphical representation of a status or activity that the user can choose to transmit at a specific time.

[0059] In the case that the entity is a group, in addition to the group name, members and various settings for the relevant group (e.g., notifications), the profile data 302 for the group may similarly include one or more avatars associated with the group.

[0060] Database 304 also stores enhancement data, such as overlays or filters, in enhancement table 312. Enhancement data is associated with and applied to videos (video data is stored in video table 314) and images (image data is stored in image table 316).

[0061] In some examples, filters are displayed as overlays on images or videos during presentation to the receiving user. Filters can be of various types, including user-selected filters from a set of filters presented to the sending user by the interactive client 104 while the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, geolocation filters specific to nearby or particular locations can be presented by the interactive client 104 within the user interface based on geographic location information determined by the user system 102's Global Positioning System (GPS) unit.

[0062] Another type of filter is a data filter, which can be selectively presented to the sending user by the interactive client 104 based on other inputs or information collected by the user system 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the sending user, the battery life of the user system 102, or the current time.

[0063] Other augmented data that can be stored in image table 316 includes augmented reality content items (e.g., corresponding to an application "lens" or augmented reality experience). Augmented reality content items can be real-time effects and sounds that can be added to images or videos.

[0064] Collection table 318 stores data related to collections of messages and associated image, video, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user whose record is maintained in entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. For this purpose, the user interface of interactive client 104 may include user-selectable icons that allow the sending user to add specific content to his or her personal story.

[0065] The collection can also constitute a "live story," which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic technologies. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a common location event at a specific time can be presented with the option to contribute content to a specific live story, for example, via the user interface of interactive client 104. Live stories can be identified to a user by interactive client 104 based on their location. The end result is a "live story" told from a community perspective.

[0066] Another type of content collection is called a "location story," which allows users of user system 102 located in a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributions to a location story may employ secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0067] As mentioned above, video table 314 stores video data, which in some examples is associated with messages whose records are maintained within message table 306. Similarly, image table 316 stores image data associated with messages whose message data is stored in entity table 308. Entity table 308 can associate various enhancements from enhancement table 312 with various images and videos stored in image table 316 and video table 314.

[0068] Database 304 also includes media database 402, voice track storage device 408, and pixel-timestamp database 412, as shown below. Figure 4 Described.

[0069] Figure 4A system 400 for implementing lip-sync is shown according to some examples. System 400 includes a media database 402, a scheduler 404, a voice separation unit 406, a voice track storage device 408, and a pixel analysis unit 410. System 400 is accessed by user equipment 414 and is provided with relevant information thereto.

[0070] Media database 402 includes a collection of media items, which includes audio with at least one vocal track. In some examples, media database 402 includes a collection of songs in audio format, but music videos or other media with audio tracks may also be included in this collection. Media database 402 is accessed by scheduler 404, which provides media data to vocal separation component 406 and optionally to user device 414.

[0071] Scheduler 404 assigns media items to vocal separation unit 406 for processing based on available processor capabilities and song popularity. Since vocal separation is processor-intensive, more popular songs are processed first. In some examples, vocal separation unit 406 includes a pre-trained neural network, such as a convolutional neural network (CNN) using the U-Net architecture. The neural network of vocal separation unit 406 is trained on tens of thousands of musical tracks, which have corresponding separate tracks (single instrument or vocal tracks) spanning multiple genres.

[0072] Based on this training, the neural network of the vocal separation component 406 can map the input mixed audio signal to the output signal of each specific source (e.g., vocals, drums). In some examples, the vocal track output signal can include the two vocal parts of a duet, or a vocal track corresponding to the lead vocal and backing vocals. The U-Net architecture can also be used to generate phonemes from vocal audio or raw music waveforms. This is similar to how it works for image segmentation, but instead of segmenting spatial pixels, it segments a temporal sequence.

[0073] In use, the vocal separation unit 406 processes the input audio track to provide an output consisting of multiple tracks, each corresponding to a different source in the audio mix. For example, there may be tracks for vocals, drums, bass, and other instruments. The vocal tracks are stored as vocal tracks in the vocal track storage device 408 along with information identifying the audio tracks from which these vocal tracks are derived. In some examples, the vocal separation unit 406 also provides the possible gender or age of the singer for a specific track.

[0074] Then, the vocal track of a specific audio track is provided to the visual analysis unit 410, which generates a list of visual identifiers for each vocal audio track, including a timestamp identifying the position of each individual visual in the audio track.

[0075] The visual analysis component 410 includes a trained acoustic machine learning model that represents the relationship between phonemes (basic speech sounds) and audio features. This acoustic model is trained using a large dataset of audio recordings and their corresponding transcriptions. The training process configures the acoustic model to recognize the acoustic characteristics of different phonemes.

[0076] In some examples, the acoustic model uses a Gaussian mixture model (GMM) to represent the probability distribution of acoustic features for each phonetic unit. A GMM is a statistical model that assumes the data is generated through a mixture of several Gaussian distributions. Each Gaussian component in the mixture represents a different state of a Hidden Markov Model (HMM), which is used to model temporal dependencies in speech.

[0077] The acoustic model is trained using a large dataset of audio recordings along with their corresponding transcriptions. The training process involves estimating the parameters of the GMM for each speech unit. The dataset used for training is diverse and representative of the target application domain to ensure that the model generalizes well to different speakers / singers and speaking / singing styles.

[0078] Feature extraction is performed on the vocal track to represent the fundamental characteristics of the sound. Commonly used features include Mel-frequency cepstral coefficients (MFCCs) and difference coefficients. These features describe the spectral content of the audio signal and capture information relevant to speech recognition, which the acoustic model uses to determine which phoneme corresponds to which feature. In some examples, the visual pixel analysis unit 410 uses a triphone model, which considers the context of a phoneme in light of preceding and following phonemes. This helps improve the accuracy of the model, especially when recognizing coarticulation effects, where the pronunciation of a phoneme is influenced by its neighboring phonemes.

[0079] Each phoneme is associated with a specific facial expression or lip shape (i.e., a visual element). The phoneme-to-visual element mapping is based on a predefined expected visual appearance of the mouth and face when each phoneme is pronounced. The visual element analysis component 410 converts the list of timestamps and phonemes into a list of timestamps and visual element identifiers based on this mapping. The list of timestamps and visual element identifiers, along with information identifying the original media item, is provided to the visual element-timestamp database 412. As used herein and in context, the term "visual element" refers both to the concept of a person having a specific mouth shape when uttering a particular phoneme and to the representation of the mouth shape corresponding to a particular phoneme, as discussed in the following paragraphs.

[0080] Also included in the database 128 of the interactive system 100 or locally stored in the user device 414 are different sets of visual elements, such as those representing different mouth styles. These styles can be cartoonish, realistic, two-dimensional, or three-dimensional, generated based on images of a specific user's or other people's mouths, or generated by manipulating a mesh corresponding to the user's mouth. Visual elements can include, for example, lipstick, facial hair, etc., and can include associated changes in a person's face, such as jaw movements. A specific set of visual elements or a specific mouth style can be selected by the user or automatically (e.g., randomly) based on audio track metadata or other contextual information.

[0081] Figure 5A A user interface 500 for use with lip-sync is shown according to some examples. In use, a person operating user device 414 opens an application such as interactive client 104 or application 106 that includes lip-sync functionality, or selects lip-sync functionality according to the options provided in the application. Audio tracks are received from the user via track selection and information interface 502. In some examples, tapping on selection and information interface 502 will open a search or browsing window for track selection, as is known. Once a track is selected, track catalog information is displayed in selection and information interface 502. If the audio track is not stored locally, it is obtained from a remote database server 126, such as media database 402 (downloaded or queued for streaming). The corresponding pixel-timestamp file is then obtained from pixel-timestamp database 412.

[0082] A video feed 508, captured by a front or rear camera on user equipment 414, is displayed on the screen of user equipment 414. In the illustrated example, the video feed 508 includes a face 510 with a mouth 512. An application running on user equipment 414 extracts facial features from the video feed 508 using various computer vision techniques and facial recognition algorithms. Extracting facial features includes identifying and capturing information about key elements of a person's face, such as the eyes, nose, mouth, and overall facial structure.

[0083] Face detection is the initial step in facial feature extraction. Algorithms such as Haar cascades, histogram of oriented gradients (HOG), or deep learning-based methods (e.g., using pre-trained convolutional neural networks like MTCNN, or OpenCV's DNN module in conjunction with a deep learning framework) can be used to locate faces within an image or video frame.

[0084] The application then performs facial keypoint detection, which involves identifying specific points on the face, such as the corners of the eyes, the tip of the nose, and the corners of the mouth. Algorithms such as shape predictors (e.g., dlib's shape predictor) or deep learning-based methods (e.g., using facial keypoint detection models like FaceNet or OpenFace) can be used for this task. Once facial keypoints are detected, various features can be extracted based on their location and relationships. Typically extracted features include distances between keypoints, angles formed by different facial components, and ratios between different measurements.

[0085] In some examples, 2D facial feature extraction is performed, while in others, 3D facial recognition techniques that determine depth information are used. This can be achieved using known techniques for estimating 3D facial geometry from 2D images, in which case a mesh representing the shape and location of the user's face and its features is generated. Facial feature extraction continues during the execution of the lip-sync method.

[0086] Upon receiving user input to play the audio track, the application begins rendering the audio, which can be done via speakers, earphones, or other associated speakers incorporated into the user device 414. The application uses the current timestamp of the audio track to retrieve the corresponding pixel identifier from the audio track's pixel-timestamp file, retrieves the corresponding pixel from the pixel set that the user has selected or that the application automatically selects, and applies it to the face 510 presented in the video feed 508 to provide, for example... Figure 5B The first instance of the animated mouth 514 shown.

[0087] Pixel localization is performed based on the positions of already detected relevant features—such as the positions of the corners of the lips in the mouth 512 of face 510. The pixel is scaled based on the distance between the positions of the corners of the lips of mouth 512 and applied to video feed 508, where the position of a feature of mouth 512 corresponds to the position of a feature of a specific pixel. In some examples, the specific pixel is rotated in the plane of the display to account for any left or right tilt of the user's head, thereby positioning the pixel at a specific rotation angle or along a line connecting the two corners of the lips of mouth 512 depicted in video feed 508, without scaling it based on the distance between the positions of mouth 512. This prevents rescaling based on inconsistencies between the user's mouth movements and the lyrics of the audio track. In such cases, the center of the pixel is located at the center of mouth 512.

[0088] In some examples, fixed distances in real life, such as the distance between a user's eyes or the width of a user's head, can be used to scale the pixels. Then, as the user moves closer to or further away from the user device 414, the (changed) size of that fixed distance in the video feed 508 can be used to scale the pixels to position them in the video feed independently of any mouth movements made by the user.

[0089] In some examples where a 3D mesh has already been defined for the user's face, the pixels can be rotated to take into account the depth of facial features in the video feed, such as the corners of the user's lips, and the appearance of the pixels can be scaled and projected accordingly.

[0090] As the audio track continues to play, the current audio timestamp is checked against the timestamps in the pixel-timestamp file to determine if the pixel identifier has changed relative to the current / previous pixel identifier. If so, the corresponding pixel is retrieved from the pixel set and positioned above the mouth of the face as described above. In some examples, known image processing techniques such as "morphing" between two pixels are performed to provide a smoother transition between pixels. Checking against the audio timestamp, updating pixels as needed, and repositioning pixels as needed will continue until the audio track finishes playing or user input to stop or pause playback is received.

[0091] In some examples, the vocal separation unit 406 will separate more than one vocal track, such as the two vocal parts of a duet, or a vocal track corresponding to the lead vocal and backing vocal. When only a single face is captured in the video feed 508, the pixel-timestamp file corresponding to the lead vocal is processed. If two or more pixel-timestamp files exist for an audio track, or if the pixel-timestamp file of an audio track includes two or more sets of pixel-timestamp data (each set of pixel-timestamp data corresponds to a different vocal track), then the pixel corresponding to the lead vocal is selected when only one face is detected in the video feed 508.

[0092] If two files or two sets of pixel-timestamp data correspond to different vocal tracks, and two faces are detected in video feed 508, different schemes can be used to assign vocal tracks to faces. For example, the lead vocal track can be initially assigned to the largest / largest face (or the largest / largest mouth) in the video at the beginning, or the lead vocal track can be updated periodically. In other examples, vocal tracks can be randomly assigned to detected mouths, or assigned based on the apparent gender or age of the person presented in the video. In some examples, assignment can follow identified characteristics (e.g., assigning a male voice in a duet to a face with facial hair), but in other cases, assignment can be intentionally incorrect for comedic effect, such as assigning a female voice to a face with facial hair.

[0093] If there is more than one face and more than one voice track, the method for animate processing of a single face described herein is executed in parallel for each face and each corresponding voice track.

[0094] Figure 6 This is a flowchart illustrating a lip-sync method according to some examples. For illustrative purposes, the operations of flowchart 600 are described herein as occurring in a serial or linear manner. However, multiple operations of flowchart 600 can occur in parallel. Furthermore, the operations of flowchart 600 do not need to be executed in the order shown, and / or one or more blocks of flowchart 600 do not need to be executed and / or can be replaced by other operations.

[0095] Figure 6 The operations shown are typically performed on user device 414 (user system 102). For clarity, flowchart 600 is discussed herein with reference to such an example. Various implementations are, of course, possible; some operations may occur in interactive server system 110, or one application may call another application or SDK to obtain the required functionality.

[0096] Flowchart 600 begins with operation 602, in which the application receives user input to select a song. In operation 604, the bibliographical or other identifying information of the selected song is used to query the pixel-timestamp database 412, and in operation 606, the application receives the pixel-timestamp file corresponding to the selected song from the pixel-timestamp database 412. In operation 608, the user or the application selects the appropriate set of pixels for the song, as discussed in more detail above.

[0097] Then, in operation 610, as described above, facial features and their positions are detected by an application, and a neutral visual element representing the absence of a phoneme (i.e., silence) is applied to the location of the mouth 512 in the face 510. After receiving user input to start playing a song, in operation 614, the application starts playing the audio.

[0098] In operation 616, the application detects facial features and their positions in the video feed. In operation 618, the application determines whether the position of the mouth 512 detected in the video stream has changed or is changing. If so, the application updates the display position of the pixel in operation 620 and proceeds to operation 622. If the application determines in operation 618 that the position of the facial feature has not changed, the method proceeds directly to operation 622.

[0099] The application then compares the song's current timestamp with the pixel-timestamp file to determine if the timestamp corresponds to a new pixel identifier. If not, the method returns to operation 614 and continues from there. If so, in operation 624, the new pixel is applied to the video feed 508 at the current position. The method then returns to operation 614 and continues from there. Machine Learning Pipeline

[0100] Figure 8 This is a flowchart illustrating a machine learning pipeline 800 based on some examples. The machine learning pipeline 800 can be used, for example, to generate a trained machine learning model 802 in a voice separation unit 406 and a pixel analysis unit 410 to perform operations associated with search and query responses. Overview

[0101] In a broad sense, machine learning can involve using computer algorithms to automatically learn patterns and relationships in data, potentially without explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning involves training a model using labeled data to predict outputs for new, unseen inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks. Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships within the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models such as autoencoders. Reinforcement learning involves training a model to make decisions in dynamic environments by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q-learning and policy gradient methods.

[0102] Examples of specific machine learning algorithms that can be deployed include logistic regression, a supervised learning algorithm for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another machine learning algorithm is Naive Bayes, another supervised learning algorithm for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random forest is another supervised learning algorithm used for classification, regression, and other tasks. Random forest builds a set of decision trees and combines their outputs to make predictions. Further examples include neural networks, which consist of layers of interconnected nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another machine learning algorithm used for recommender systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to reveal hidden patterns or relationships in the data. Support Vector Machine (SVM) is a supervised learning algorithm used for classification, regression, and other tasks. SVM searches for hyperplanes that separate different classes in the data. Other types of machine learning algorithms include decision tree algorithms, k-nearest neighbor algorithms, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0103] The performance of a machine learning model is typically evaluated on a separate test dataset that was not used during training to ensure that the model can generalize to new, unseen data.

[0104] While this article discusses several specific examples of machine learning algorithms, the principles discussed can be applied to other machine learning algorithms as well. Deep learning algorithms, such as convolutional neural networks, recurrent neural networks, and Transformers, as well as more traditional machine learning algorithms, such as decision trees, random forests, and gradient boosting, can be used in a variety of machine learning applications.

[0105] Three typical problems in machine learning are classification, regression, and generation. Classification problems (also known as categorization problems) aim to classify an item into one of several category values ​​(e.g., is the object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​as real numbers). Generation algorithms aim to produce new examples that resemble the examples provided for training. For example, a text generation algorithm is trained on a large number of text documents and configured to generate new, coherent text with statistical properties similar to the training data. Training phase

[0106] Generating a trained machine learning model 802 may include multiple stages that form part of a machine learning pipeline 800, said multiple stages including, for example... Figure 7 The following stages are shown: Data Collection and Preprocessing 702: This stage may include acquiring and cleaning data to ensure it is suitable for use in machine learning models. This stage may also include removing duplicate values, handling missing values, and converting the data to a suitable format. Feature engineering 704: This stage may include selecting training data 806 and transforming it to create features useful for predicting the target variable. Feature engineering may include (1) receiving features 808 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 808 (e.g., unstructured or unlabeled data for unsupervised learning) in the training data 806. Model selection and training 706: This stage may include selecting a suitable machine learning algorithm and training it on preprocessed data. This stage may also involve splitting the data into training and test sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance. Model Evaluation 708: This stage may include evaluating the performance of the trained model (e.g., a trained machine learning model 802) on a separate test dataset. This stage can help determine if the model is overfitting or underfitting, and whether the model is suitable for deployment. Prediction 710: This stage involves using a trained model (e.g., a trained machine learning model 802) to generate predictions for new, unseen data. Validation, refinement, or retraining 712: This stage may include updating the model based on feedback generated in the prediction stage (such as new data or user feedback). Deployment 714: This phase may include integrating the trained model (e.g., a trained machine learning model 802) into a wider system or application (e.g., a web service, mobile application, or IoT device). This phase may involve installing the API, building the user interface, and ensuring the model is scalable and capable of handling large amounts of data.

[0107] Figure 8Further details of two example phases are shown: training phase 804 (e.g., part of model selection and training 706) and prediction phase 810 (part of prediction 710). Prior to training phase 804, features 808 are identified through feature engineering 704. This may include identifying information-rich, discriminative, and independent features to effectively operate the trained machine learning model 802 in pattern recognition, classification, and regression. In some examples, training data 806 includes labeled data known for the pre-identified features 808 and one or more outcomes. Each of the features 808 can be a variable or attribute, such as an individual measurable characteristic of a process, item, system, or phenomenon represented by the dataset (e.g., training data 806). By way of example only, features 808 can also be of different types such as numeric features, strings, and graphs, and can include one or more of content 812, concepts 814, attributes 816, historical data 818, and / or user data 820.

[0108] In the training phase 804, the machine learning pipeline 800 uses the training data 806 to find the correlations between features 808 that affect the prediction results or the prediction / inference data 822.

[0109] The trained machine learning model 802 is trained during the training phase 804 of the machine learning program training 824 using training data 806 and identified features 808. The machine learning program training 824 evaluates the features 808 as being correlated with the training data 806. The result of the training is the trained machine learning model 802 (e.g., a trained or learned model).

[0110] Furthermore, the training phase 804 may involve machine learning, in which the training data 806 is structured (e.g., labeled during preprocessing operations). The trained machine learning model 802 implements a neural network 826 capable of performing operations such as classification and clustering. In other examples, the training phase 804 may involve deep learning, in which the training data 806 is unstructured, and the trained machine learning model 802 implements a deep neural network 826 capable of performing feature extraction and classification / clustering operations.

[0111] In some examples, neural network 826 can be generated during training phase 804 and implemented within a trained machine learning model 802. Neural network 826 includes a hierarchical (e.g., layered) organization of neurons, with each layer consisting of multiple neurons or nodes. Neurons in the input layer receive input data, while neurons in the output layer produce the network's final output. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0112] In a neural network 826, each neuron can operationally compute a function such as an activation function, which takes as input a weighted sum of the outputs of neurons in the previous layer and a bias term. The output of this function is then passed as input to neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is passed from that neuron (e.g., a sending neuron) to neurons connected in subsequent layers (e.g., receiving neurons). The connections between neurons have associated weights that define the effect of input from the sending neuron on the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks can use different activation functions and learning algorithms, thus affecting their performance on different tasks. The hierarchical organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0113] In some examples, and by way of example only, neural network 826 can also be one of several different types of neural networks, such as a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetric connection neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a Transformer network.

[0114] In addition to the training phase 804, a validation phase can be performed on a separate dataset called the validation dataset. The validation dataset is used to tune the model's hyperparameters, such as the learning rate and regularization parameters. Tuning the hyperparameters improves the model's performance on the validation dataset.

[0115] Once the model is fully trained and validated, the testing phase allows it to be tested on a new dataset. The test dataset is used to evaluate the model's performance and ensure that it has not overfitted the training data.

[0116] In the prediction phase 810, the trained machine learning model 802 uses features 808 to analyze the query data 828 to generate inferences, results, or predictions, as examples of the prediction / inference data 822. For example, during the prediction phase 810, the trained machine learning model 802 generates outputs. The query data 828 is provided as input to the trained machine learning model 802, and in response to receiving the query data 828, the trained machine learning model 802 generates the prediction / inference data 822 as output.

[0117] In some examples, the trained machine learning model 802 may be a generative AI model. Generative AI is a term that can refer to any type of artificial intelligence capable of creating new content from training data 806. For example, generative AI can produce text, images, videos, audio, code, or synthetic data that are similar to but not identical to the original data. In the case where the trained machine learning model 802 is generative AI, query data 828 may include text, audio, images, videos, digital or media content cues, and output prediction / inference data 822 may include text, images, videos, audio, code, or synthetic data.

[0118] Some of the techniques that can be used in generative AI are: Convolutional Neural Networks (CNNs): CNNs can be used for image recognition and computer vision tasks. For example, a CNN can be designed to extract features from an image by using filters or kernels that scan the input image and highlight important patterns. Recurrent Neural Networks (RNNs): RNNs can be used to process sequential data such as speech, text, and time-series data. RNNs employ feedback loops that enable them to capture temporal dependencies and remember past inputs. Generative Adversarial Networks (GANs): A GAN can consist of two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network tries to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time. Variational Autoencoders (VAEs): VAEs encode input data into a latent space (e.g., a compressed representation) and then decode it back to output data. The latent space can be manipulated to generate new variations of the output data. VAEs can use self-attention mechanisms to process input data, enabling them to handle long sequences of text and capture complex dependencies. Transformer Models: Transformer models use attention mechanisms to learn relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data such as text or speech, as well as non-sequential data such as images or code.

[0119] Machine architecture

[0120] Figure 9 This is a schematic representation of machine 900, within which instructions 902 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 900 to perform any or more of the methods discussed herein. For example, instructions 902 can cause machine 900 to perform any or more of the methods described herein. Instructions 902 transform a general, unprogrammed machine 900 into a specific machine 900 programmed to perform the described and illustrated functions in the described manner. Machine 900 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 900 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 900 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 902 specifying actions to be taken by machine 900. Furthermore, although only a single machine 900 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 902 to perform any or more of the methods discussed herein. For example, machine 900 may include user system 102 or any of a plurality of server devices forming part of interactive server system 110. In some examples, machine 900 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0121] Machine 900 may include a processor 904, a memory 906, and input / output (I / O) components 908 that can be configured to communicate with each other via a bus 910. In the example, processor 904 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 912 and 914 that execute instruction 902. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 9 Multiple processors 904 are shown, but machine 900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0122] Memory 906 includes main memory 916, static memory 918, and memory cells 920, all of which are accessible by processor 904 via bus 910. Main memory 906, static memory 918, and memory cells 920 store instructions 902 that implement any one or more of the methods or functions described herein. Instructions 902 may also reside wholly or partially within main memory 916, static memory 918, machine-readable medium 922 within memory cells 920, at least one processor of processor 904 (e.g., within the processor's cache memory), or any suitable combination thereof during execution by machine 900.

[0123] I / O component 908 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 908 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It will be understood that I / O component 908 may include... Figure 9Many other components are not shown. In various examples, I / O component 908 may include user output component 924 and user input component 926. User output component 924 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 926 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0124] In another example, I / O component 908 may include biometric component 928, motion component 930, environmental component 932, or positioning component 934, as well as various other components. For example, biometric component 928 includes components for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Biometric component may include a brain-computer interface (BMI) system that allows communication between the brain and external devices or machines. This can be achieved by recording brain activity data, converting this data into a computer-understandable format, and then using the resulting signals to control the device or machine.

[0125] Examples of BMI technology types include: BMI, based on electroencephalography (EEG), uses electrodes placed on the scalp to record electrical activity in the brain. Invasive BMI uses electrodes surgically implanted in the brain. Optogenetics BMI uses light to control the activity of specific nerve cells in the brain.

[0126] Any biometric data collected by biometric components is captured and stored only with user approval and deleted upon user request. Furthermore, such biometric data can be used for very limited purposes, such as identification verification. To ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if applicable). Any use of biometric data can be strictly limited to identification verification purposes, and the data cannot be shared or sold to any third party without the user's explicit consent. Additionally, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0127] The moving part 930 includes an acceleration sensor part (e.g., an accelerometer), a gravity sensor part, and a rotation sensor part (e.g., a gyroscope).

[0128] The environmental component 932 includes, for example, one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., an infrared sensor that detects nearby objects), gas sensors (e.g., a gas detection sensor that detects the concentration of hazardous gases for safety purposes or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0129] Regarding the camera device, user system 102 may have a camera device system including, for example, a front-facing camera on the front surface of user system 102 and a rear-facing camera on the rear surface of user system 102. The front-facing camera may be used, for example, to capture still images and videos (e.g., “selfies”) of the user of user system 102, which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera device mode, which are similarly enhanced with the enhancement data. In addition to the front-facing and rear-facing cameras, user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0130] Furthermore, the camera system of user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even include triple, quadruple, or quintuple rear camera configurations on the front and rear sides of user system 102. For example, these multiple camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0131] The positioning component 934 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0132] Various technologies can be used to achieve communication. I / O component 908 also includes communication component 936, which is operable to couple machine 900 to network 938 or device 940 via appropriate coupling or connection. For example, communication component 936 may include a network interface component or another suitable device that interfaces with network 938. In further examples, communication component 936 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, Bluetooth component, etc. ® Components (e.g., Bluetooth) ® Low energy consumption), Wi-Fi ® Components and other communication components that provide communication via other modes. Device 940 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0133] Furthermore, the communication component 936 can detect identifiers, or include components operable to detect identifiers. For example, the communication component 936 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph™, MaxiCode, PDF417, UltraCode, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be obtained via the communication component 936, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location by detecting NFC beacon signals that can indicate a specific location, etc.

[0134] Various memories (e.g., main memory 916, static memory 918, and the memory of processor 904) and storage unit 920 may store one or more sets of instructions and data structures (e.g., software) embodied or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 902) cause various operations to implement the disclosed examples when executed by processor 904.

[0135] Instruction 902 can be sent or received over network 938 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 936) and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 902 can be sent or received via a transmission medium coupled to device 940 (e.g., peer-to-peer coupling).

[0136] Software Architecture

[0137] Figure 10 This is a block diagram 1000 illustrating a software architecture 1002 that can be installed on any one or more of the devices described herein. The software architecture 1002 is supported by hardware, such as a machine 1004 including a processor 1006, memory 1008, and I / O components 1010. In this example, the software architecture 1002 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1002 includes layers such as an operating system 1012, libraries 1014, frameworks 1016, and applications 1018. Operationally, application 1018 activates API call 1020 via the software stack and receives message 1022 in response to API call 1020.

[0138] Operating system 1012 manages hardware resources and provides public services. Operating system 1012 includes, for example, a kernel 1024, services 1026, and drivers 1028. Kernel 1024 acts as an abstraction layer between hardware and other software layers. For example, kernel 1024 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 1026 can provide other public services to other software layers. Drivers 1028 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1028 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® low-power drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0139] Library 1014 provides common low-level infrastructure used by application 1018. Library 1014 may include system library 1030 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1014 may include API library 1032, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., an OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing functionality), etc. Library 1014 may also include a wide variety of other libraries 1034 to provide many other APIs to application 1018.

[0140] Framework 1016 provides a common high-level infrastructure for use by application 1018. For example, framework 1016 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 1016 can provide a wide range of other APIs that can be used by application 1018, some of which may be specific to a particular operating system or platform.

[0141] In the example, application 1018 may include home application 1036, contact application 1038, browser application 1040, book reader application 1042, location application 1044, media application 1046, messaging application 1048, game application 1050, and a variety of other applications such as third-party application 1052. Application 1018 is a program that performs the functions defined in the program. One or more applications 1018 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1052 (e.g., an application developed by an entity other than a platform vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software that runs on mobile operating systems such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, a third-party application 1052 can activate API call 1020 provided by the operating system 1012 to facilitate the functionality described herein.

[0142] Example

[0143] Various examples are envisioned. Example 1 is a system comprising: at least one processor; at least one memory unit storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: playing back an audio track; detecting the position of a mouth depicted in a video feed; comparing a playback timestamp of the audio track with pixel-timestamp data of the audio track to identify a pixel corresponding to the audio playback timestamp; and positioning the pixel at the position of the mouth detected in the video feed.

[0144] In Example 2, the subject of Example 1 includes, wherein the operation further includes: detecting an updated position of a mouth depicted in the video feed; and positioning the pixel at the updated position of the mouth detected in the video feed.

[0145] In Example 3, the subject matter of Examples 1 to 2 includes, wherein the operation further includes: comparing the updated playback timestamp of the audio track with the pixel-timestamp data of the audio track to identify the updated pixel corresponding to the timestamp of the audio playback; and locating the updated pixel at the position of the mouth detected in the video feed.

[0146] In Example 4, the subject matter of Examples 1 to 3 includes, wherein detecting the position of a mouth in the video feed includes detecting the corner of the lips of the mouth in the video feed, and wherein positioning the viewpoint at the position of the mouth detected in the video feed includes positioning the corner of the lips of the mouth depicted in the viewpoint at the corner of the lips of the mouth in the video feed.

[0147] In Example 5, the subject matter of Examples 1 to 4 includes, wherein detecting the position of a mouth in the video feed includes detecting the angle of the mouth in the video feed, and wherein positioning the viewpoint at the detected position of the mouth in the video feed includes rotating the viewpoint by the detected angle.

[0148] In Example 6, the subject matter of Examples 1 to 5 includes, wherein the operation further includes: scaling the visual pixels based on the feature size of the head detected in the video feed.

[0149] In Example 7, the subject matter of Examples 1 to 6 includes, wherein the pixel-timestamp data includes a first set of pixel-timestamps and a second set of pixel-timestamps for two voice tracks, and wherein the operation further includes: detecting a first mouth and a second mouth in the video feed; positioning the first pixel at the position of the first mouth in the video feed based on the first set of pixel-timestamps; and positioning the second pixel at the position of the second mouth in the video feed based on the second set of pixel-timestamps.

[0150] In Example 8, the subject of Example 7 includes, wherein the first set of pixel-timestamps corresponds to the lead vocals, and in the video feed, the first mouth is larger than the second mouth.

[0151] In Example 9, the subject of Examples 7 and 8 includes the fact that the assignment of the first set of pixel-timestamps to the mouth is done randomly.

[0152] Example 10 is a method executed by one or more processors, the method comprising: playing back an audio track; detecting the position of a mouth depicted in a video feed; comparing a playback timestamp of the audio track with pixel-timestamp data of the audio track to identify a pixel corresponding to the audio playback timestamp; and positioning the pixel at the position of the mouth detected in the video feed.

[0153] In Example 11, the subject of Example 10 includes detecting an updated position of a mouth depicted in the video feed; and positioning the pixel at the updated position of the mouth detected in the video feed.

[0154] In Example 12, the subject matter of Examples 10 to 11 includes comparing an updated playback timestamp of the audio track with pixel-timestamp data of the audio track to identify an updated pixel corresponding to the timestamp of the audio playback; and locating the updated pixel at the position of a mouth detected in the video feed.

[0155] In Example 13, the subject matter of Examples 10 to 12 includes, wherein the pixel-timestamp data includes a first set of pixel-timestamps and a second set of pixel-timestamps for two voice tracks, and the method further includes: detecting a first mouth and a second mouth in the video feed; locating a first pixel at the position of the first mouth in the video feed based on the first set of pixel-timestamps; and locating a second pixel at the position of the second mouth in the video feed based on the second set of pixel-timestamps.

[0156] In Example 14, the subject of Example 13 includes the following: the first set of pixel-timestamps corresponds to the lead vocals, and in the video feed, the first mouth is larger than the second mouth.

[0157] In Example 15, the subject of Examples 13 to 14 includes the fact that the assignment of the first set of pixel-timestamps to the mouth is done randomly.

[0158] Example 16 is a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: playing back an audio track; detecting the position of a mouth depicted in a video feed; comparing a playback timestamp of the audio track with pixel-timestamp data of the audio track to identify a pixel corresponding to the audio playback timestamp; and positioning the pixel at the position of the mouth detected in the video feed.

[0159] In Example 17, the subject of Example 16 includes the following: detecting the position of a mouth in the video feed includes detecting the angle of the mouth in the video feed; and positioning the viewpoint at the detected position of the mouth in the video feed includes rotating the viewpoint by the detected angle.

[0160] In Example 18, the subject of Examples 16 to 17 includes, wherein the operation further includes: scaling the visual pixels based on the feature size of the head detected in the video feed.

[0161] In Example 19, the subject matter of Examples 16 to 18 includes, wherein the pixel-timestamp data includes a first set of pixel-timestamps and a second set of pixel-timestamps for two voice tracks, and wherein the operation further includes: detecting a first mouth and a second mouth in the video feed; positioning the first pixel at the location of the first mouth in the video feed based on the first set of pixel-timestamps; and positioning the second pixel at the location of the second mouth in the video feed based on the second set of pixel-timestamps.

[0162] In Example 20, the subject of Example 19 includes the fact that the assignment of the first set of pixel-timestamps to the mouth is done randomly.

[0163] Example 21 is a method for processing audio tracks, which is executed by one or more processors, including: retrieving audio tracks from a database of audio tracks; extracting vocal tracks from the audio tracks; analyzing the vocal tracks to determine visuals corresponding to lyrics at corresponding time positions in the audio tracks; and saving a visual-timestamp data file, the visual-timestamp data file including visual identifiers corresponding to the determined visuals and the corresponding time positions in the audio tracks.

[0164] In Example 22, the subject of Example 21 includes the use of a pre-trained neural network trained on a music track database to extract vocal tracks, the music tracks having one or more corresponding separate vocal tracks spanning multiple genres.

[0165] In Example 23, the subject matter of Examples 21 to 22 includes that the analysis of a vocal track is performed at least in part by a trained audio machine learning model that determines phonemes corresponding to audio features in the vocal track.

[0166] In Example 24, the subject of Example 23 includes the training of an audio machine learning model using a large dataset of audio recordings and their corresponding phoneme transcriptions.

[0167] In Example 25, the topics of Examples 23 and 24 include, where the analysis of the vocal track also includes mapping the identified phonemes to corresponding visual pixels.

[0168] Example 26 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any one of Examples 1 to 25.

[0169] Example 27 is an apparatus that includes a unit for implementing any one of Examples 1 to 25.

[0170] Example 28 is a system for implementing any one of Examples 1 through 25.

[0171] Example 29 is a method for implementing any one of Examples 1 through 25.

[0172] Glossary

[0173] "Carrier signal" refers to any intangible medium, such as a medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible medium that facilitates the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0174] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0175] "Communications network" means, for example, one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can implement any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0176] A “component” refers to, for example, a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide specific processing or control functionality, such as partitioning or modularization. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components, and part of a program that typically performs a specific function within a related function. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine), which is uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be understood that a decision may be made, for cost and time considerations, whether to implement the hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of a hardware component being temporarily configured (e.g., programmed), without needing to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, this general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components.Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute components of a processor implementation that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least partially by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some of the operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "Software as a Service" (SaaS) operation. For example, at least some of the operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some examples, the processor or processor-implemented component can reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component can be distributed across multiple geographic locations.

[0177] "Computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0178] A "brief message" is a message that is accessible for a limited time, such as a short period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0179] "Machine storage medium" refers to, for example, a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term "signal medium."

[0180] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0181] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0182] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user, and that the user interacts with to perform actions or interactions on the user equipment, including interactions with other users or computer systems.

Claims

1. A system comprising: At least one processor; At least one memory unit stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: Play back the audio track; Detect the position of the mouth depicted in the video feed; The playback timestamp of the audio track is compared with the pixel-timestamp data of the audio track to identify the pixel corresponding to the audio playback timestamp; and The pixel is positioned at the location of the mouth detected in the video feed.

2. The system according to claim 1, wherein, The operation also includes: Detect the updated position of the mouth depicted in the video feed; and The pixel is positioned at the updated location of the mouth detected in the video feed.

3. The system according to claim 1, wherein, The operation also includes: The updated playback timestamp of the audio track is compared with the pixel-timestamp data of the audio track to identify the updated pixel corresponding to the audio playback timestamp; and The updated pixel is positioned at the location of the mouth detected in the video feed.

4. The system according to claim 1, wherein, Detecting the position of a mouth in the video feed includes detecting the corner of the lips of the mouth in the video feed, and wherein positioning the viewpoint at the detected position of the mouth in the video feed includes positioning the corner of the lips of the mouth depicted in the viewpoint at the corner of the lips of the mouth in the video feed.

5. The system according to claim 1, wherein, Detecting the position of the mouth in the video feed includes detecting the angle of the mouth in the video feed, and wherein positioning the viewpoint at the detected position of the mouth in the video feed includes rotating the viewpoint by the detected angle.

6. The system according to claim 1, wherein, The operation also includes: The pixels are scaled based on the feature size of the head detected in the video feed.

7. The system according to claim 1, wherein, The pixel-timestamp data includes a first set of pixel-timestamps and a second set of pixel-timestamps for two vocal tracks, and the operation further includes: Detect the first and second mouth portions in the video feed; Based on the first set of pixel-timestamps, the first pixel is positioned at the location of the first mouth in the video feed; and The second pixel is positioned at the location of the second mouth in the video feed based on the second set of pixel-timestamps.

8. The system according to claim 7, wherein, The first set of pixel-timestamps corresponds to the lead vocalist's voice, and in the video feed, the first mouth is larger than the second mouth.

9. The system according to claim 7, wherein, The assignment of the first set of visual-timestamps to the mouth is done randomly.

10. A method, the method being executed by one or more processors, the method comprising: Play back the audio track; Detect the position of the mouth depicted in the video feed; The playback timestamp of the audio track is compared with the pixel-timestamp data of the audio track to identify the pixel corresponding to the audio playback timestamp; as well as The pixel is positioned at the location of the mouth detected in the video feed.

11. The method of claim 10, further comprising: Detect the updated position of the mouth depicted in the video feed; as well as The pixel is positioned at the updated location of the mouth detected in the video feed.

12. The method of claim 10, further comprising: The updated playback timestamp of the audio track is compared with the pixel-timestamp data of the audio track to identify the updated pixel corresponding to the timestamp of the audio playback; as well as The updated pixel is positioned at the location of the mouth detected in the video feed.

13. The method according to claim 10, wherein, The pixel-timestamp data includes a first set of pixel-timestamps and a second set of pixel-timestamps for two vocal tracks, and the method further includes: Detect the first and second mouth portions in the video feed; Based on the first set of pixel-timestamps, the first pixel is positioned at the location of the first mouth in the video feed; and The second pixel is positioned at the location of the second mouth in the video feed based on the second set of pixel-timestamps.

14. The method according to claim 13, wherein, The first set of pixel-timestamps corresponds to the lead vocalist's voice, and in the video feed, the first mouth is larger than the second mouth.

15. The method according to claim 13, wherein, The assignment of the first set of visual-timestamps to the mouth is done randomly.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform an operation, the operation comprising: Play back the audio track; Detect the position of the mouth depicted in the video feed; The playback timestamp of the audio track is compared with the pixel-timestamp data of the audio track to identify the pixel corresponding to the audio playback timestamp; as well as The pixel is positioned at the location of the mouth detected in the video feed.

17. The non-transitory computer-readable storage medium according to claim 16, wherein, Detecting the position of the mouth in the video feed includes detecting the angle of the mouth in the video feed, and wherein positioning the viewpoint at the detected position of the mouth in the video feed includes rotating the viewpoint by the detected angle.

18. The non-transitory computer-readable storage medium according to claim 16, wherein, The operation also includes: The pixels are scaled based on the feature size of the head detected in the video feed.

19. The non-transitory computer-readable storage medium according to claim 16, wherein, The pixel-timestamp data includes a first set of pixel-timestamps and a second set of pixel-timestamps for two vocal tracks, and the operation further includes: Detect the first and second mouth portions in the video feed; Based on the first set of pixel-timestamps, the first pixel is positioned at the location of the first mouth in the video feed; and The second pixel is positioned at the location of the second mouth in the video feed based on the second set of pixel-timestamps.

20. The non-transitory computer-readable storage medium according to claim 19, wherein, The assignment of the first set of visual-timestamps to the mouth is done randomly.

Images