Generate spliced ​​data stream

Abstract

Systems and methods provide for a server computer to receive a plurality of messages from a plurality of user computing devices, each message in the plurality of messages comprising a data stream, determine a subset of messages in the plurality of messages associated with a similar geographic location and time period, determine a group of messages in the subset of messages based on a match score for each pair of messages, and stitch the group of messages together based on a time period of each message in the group of messages to generate a stitched data stream from the data streams of each message, wherein the stitched data stream includes messages of data streams having overlapping time periods such that more than one data stream may exist within a given time period.

Description

[0001] This application is a divisional application of a patent application with an application date of March 23, 2018, application number 201880021595.5, and invention name “Generating spliced ​​data streams”.

[0002] Priority claim

[0003] This patent application claims priority to U.S. application serial number 15 / 470,004, filed on March 27, 2017, and also claims priority to U.S. application serial number 15,470,025, filed on March 27, 2017, the entire contents of which are incorporated herein by reference. Background Art

[0004] Messaging systems may receive millions of messages from users who wish to share media content such as audio, images, and video between user devices (e.g., mobile devices, personal computers, etc.). The media content of these messages may be associated with a common geographic location, a common time period, a common event, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The various figures in the drawings depict only example embodiments of the present disclosure and should not be considered limiting of its scope.

[0006] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, according to some example embodiments.

[0007] Figure 2 is a block diagram showing further details regarding a messaging system according to some example embodiments.

[0008] Figure 3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system according to some example embodiments.

[0009] Figure 4 is a schematic diagram illustrating the structure of a message generated by a messaging client application for communication according to some embodiments.

[0010] Figure 5 is a schematic diagram illustrating an example access restriction process in which access to content (e.g., an ephemeral message and associated multimedia data payload) or a collection of content (e.g., an ephemeral message story) may be time-restricted (e.g., to make it ephemeral).

[0011] Figure 6 is a flow chart illustrating aspects of a method according to some example embodiments.

[0012] Figure 7 An example of a spectrum diagram is shown according to some example embodiments.

[0013] Figure 8 An example of a detected maximum value of a spectrogram is shown according to some example embodiments.

[0014] Fig. 9 Shown are examples of maxima of spectrograms concatenated together into a final audio fingerprint in accordance with some example embodiments.

[0015] Figure 10-11 Each shows a visual representation of a spliced ​​data stream including a plurality of messages received from a plurality of user devices according to some example embodiments.

[0016] Fig.12 is a flow chart illustrating aspects of a method according to some example embodiments.

[0017] Figure 13-15 Each shows an example user interface of a user computing device according to some example embodiments.

[0018] Fig.16 is a block diagram illustrating an example of a software architecture that may be installed on a machine according to some example embodiments.

[0019] Fig.17 A diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein is shown according to an example embodiment. DETAILED DESCRIPTION

[0020] The system and method described herein relate to processing media content items to be shared between devices via a messaging system. For example, a user may wish to share one or more videos, images, etc. with one or more other users. These messages may be associated with common audio content (such as songs, concerts, speeches, etc.). The embodiments described herein provide a mechanism for splicing together data streams of multiple messages received from multiple user computing devices in the future, to create a dense audio splicing associated with the common audio timeline (common audio timeline) of the data stream. For example, audio splicing is used to automatically splice together a group of messages or data streams associated with the same audio content. Given a group of messages or data streams, the system described herein extracts the audio fingerprint of each message, creates an audio match to identify the message associated with the same audio content by matching the audio fingerprint across all message pairs, and then creates an audio splicing by finding a path in a group of audio matches. The spliced ​​audio or spliced ​​data stream can then be provided to one or more user computing devices for users to view the spliced ​​data stream.

[0021] The stitched data stream may include messages with data streams that overlap in time periods, such that more than one data stream may exist in a given time period. When a user is viewing the stitched data stream, an example embodiment allows the user to switch to various other data streams in any given time period. For example, a user may be watching the lead singer of a concert and switch the view to see the guitarist or drummer or the audience at that time in the concert, etc. In this way, the user can switch between alternate views in any given time period in the common audio timeline.

[0022] Figure 1 1 is a block diagram illustrating a networked system 100 (e.g., a messaging system) for exchanging data (e.g., messages and associated content) over a network. The networked system 100 includes a plurality of client devices 102, each of which hosts a plurality of client applications 104. Each client application 104 is communicatively coupled to instances of other client applications 104 and to a server system 108 via a network 106.

[0023] The client device 102 may also be referred to herein as a user device or user computing device. The client device 102 may include, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, a computer in a vehicle, or any other communication device that a user may use to access the networked system 100. In some embodiments, the client device 102 may include a display module (not shown) to display information (e.g., in the form of a user interface). In further embodiments, the client device 102 may include one or more of a touch screen, an accelerometer, a gyroscope, a camera, a microphone, a global positioning system (GPS) device, etc. The client device 102 may be a user's device for creating media content items such as videos, images (e.g., photos), audio, and sending and receiving messages containing such media content items to other users and from other users. The elements of such media content from multiple messages may then be spliced ​​together, as described in further detail in the embodiments described below.

[0024] One or more users may interact with the client device 102 (e.g., a person, machine, or other tool that interacts with the client device 102). In an example embodiment, a user may not be part of the system 100, but may interact with the system 100 via the client device 102 or other tool. For example, a user may provide input to the client device 102 (e.g., touch screen input or alphanumeric input), and the input may be transmitted to other entities in the system 100 (e.g., server system 108, etc.) via the network 106. In this case, in response to receiving the input from the user, the other entities in the system 100 may transmit information to the client device 102 via the network 106 for presentation to the user. In this way, the user may interact with various entities in the system 100 using the client device 102.

[0025] The system 100 may further include a network 106. One or more portions of the network 106 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, another type of network, or a combination of two or more such networks.

[0026] The client device 102 may access the web client via a web client (e.g., a browser such as that provided by Internet developed by the company The client device 102 may include a web browser) or one or more client applications 104 to access various data and applications provided by other entities in the system 100. As described above, the client device 102 may include one or more client applications 104 (also referred to as "application software"), such as, but not limited to, a web browser, a messaging application, an electronic mail (email) application, an e-commerce site application, a map or location application, a media content editing application, a media content viewing application, etc.

[0027] In one example, the client application 104 can be a messaging application that allows a user to take a photo or video, add a caption, or otherwise edit the photo or video, and then send the photo or video to another user. The message can be short-lived and removed from the receiving user device after viewing or after a predetermined amount of time (e.g., 10 seconds, 24 hours, etc.). A short message refers to a message that can be accessed within the duration of a limited time. A short message can be text, an image, a video, and other such content that can be spliced ​​together according to the embodiments described herein. The access time of a short message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technology, the message is temporary.

[0028] The messaging application may further allow the user to create a gallery. A gallery may be a collection of photos and videos that may be viewed by other users who "follow" the user's gallery (e.g., subscribe to view and receive updates in the user's gallery). Galleries may also be short-lived (e.g., lasting 24 hours, for the duration of an event (e.g., during a concert, sporting event, etc.), or other predetermined time).

[0029] The short message may be associated with a message duration parameter, the value of which determines the amount of time that the short message will be displayed by the client application 104 to the recipient user of the short message. The short message may be further associated with a message recipient identifier and a message timer. The message timer may be responsible for determining the amount of time that the short message is shown to a specific recipient user identified by the message recipient identifier. For example, the short message may only be shown to the relevant recipient user for a period of time determined by the value of the message duration parameter.

[0030] In another example, a messaging application may allow a user to store photos and videos and create a gallery that is not ephemeral and can be sent to other users. For example, assembling photos and videos from a recent vacation to share with friends and family.

[0031] In some embodiments, one or more client applications 104 may be included in a given one of the client devices 102 and configured to provide a user interface and at least some functionality locally, wherein the application 104 is configured to communicate with other entities in the system 100 (e.g., the server system 108) as needed for data and / or processing capabilities that are not available locally (e.g., accessing location information, authenticating a user, authenticating a payment method, accessing media content stored on a server, synchronizing media content between the client device 102 and the server computer, etc.). Conversely, one or more applications 104 may also not be included in the client device 102, and the client device 102 may then use its web browser to access one or more applications hosted on other entities in the system 100 (e.g., the server system 108).

[0032] The server system 108 may provide server-side functionality to one or more client devices 102 via a network 106 (e.g., the Internet or a wide area network (WAN)). The server system 108 may include an application programming interface (API) server 110, an application server 112, a message server application 114, as well as a media content processing server 116, a social networking system 122, and a data stream stitching system 124, each of which may be communicatively coupled to each other and to one or more data storage devices (such as a database 120).

[0033] According to some example embodiments, the server system 108 may be a cloud computing environment. In an example embodiment, the server system 108 and any servers associated with the server system 108 may be associated with a cloud-based application. The one or more databases 120 may be storage devices that store such as unprocessed media content, original media content from users (e.g., high-quality media content), processed media content (e.g., media content formatted for sharing with the client device 102 and viewing on the client device 102), spliced ​​audio data streams, user information, user device information, etc. The one or more databases 120 may include cloud-based storage devices external to the server system 108 (e.g., hosted by one or more third-party entities external to the server system 108). Although the storage device is shown as a database 120, it is understood that the system 100 can access the storage device and store data in the storage device, and the storage device is such as a database 120, a blob storage device, and other types of storage methods.

[0034] Thus, each client application 104 is able to communicate and exchange data with another client application 104 and the server system 108 via the network 106. The data exchanged between the client applications 104 and the data exchanged between the client applications 104 and the server system 108 include functions (e.g., instructions to call the functions) and payload data (e.g., text, audio, video or other multimedia data).

[0035] The server system 108 provides server-side functionality to certain client applications 104 via the network 106. Although certain functions of the system 100 are described herein as being performed by the client application 104 or by the server system 108, it is understood that the location of certain functions within the client application 104 or the server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain techniques and functions within the server system 108, but later migrate the techniques and functions to the client application 104 where the client device 102 has sufficient processing power.

[0036] The server system 108 supports various services and operations provided to the client applications 104. Such operations include sending data to the client applications 104, receiving data from the client applications 104, and processing data generated by the client applications 104. The data may include, for example, message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information and live event information, date and time stamps. The data exchange within the networked system 100 is invoked and controlled through the available functions via the user interface (UI) of the client application 104.

[0037] In server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120 in which data associated with messages processed by application server 112 is stored.

[0038] The API server 110 server receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the API server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the client application 104 in order to call functions of the application server 112. The API server 110 exposes various functions supported by the application server 112, including account registration, login functions, sending messages from a particular client application 104 to another client application 104 via the application server 112, sending media files (e.g., images or videos) from a client application 104 to a messaging application server 114, and for possible access by another client application 104, setting a collection of media data (e.g., a story), obtaining a list of friends of the user of the client device 102, obtaining such a collection, obtaining messages and content, adding and removing friends from a social graph, the position of friends within the social graph, open and application events (e.g., related to the client application 104), etc.

[0039] The application server 112 hosts multiple applications and subsystems, including a messaging server application 114, a media content processing system 116, a social networking system 122, and a data stream splicing system 124. The messaging server application 114 implements a variety of message processing technologies and functions, particularly technologies and functions related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., called stories or galleries). These collections are then made available to the client application 104 by the messaging server application 114. In view of the hardware requirements for such processing, other processor- and memory-intensive data processing can also be performed on the server side by the messaging server application 114.

[0040] The application server 112 also includes a media content processing system 116 that is dedicated to performing various media content processing operations, typically related to images or videos received within the payload of messages at the messaging server application 114. The media content processing system 116 can access one or more data storage devices (e.g., database 120) to retrieve stored data for processing media content and store the results of the processed media content.

[0041] The social networking system 122 supports various social networking functional services and makes these functionalities and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph 304 within the database 120. Examples of functionalities and services supported by the social networking system 122 include identification of other users of the networked system 100 who have relationships with a particular user or who the particular user "follows," as well as identification of the particular user's following and other entities.

[0042] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to one or more databases 120 in which data associated with messages processed by the messaging server application 114 is stored.

[0043] The messaging server application 114 may be responsible for generating and delivering messages between users of the client devices 102. The messaging application server 114 may utilize any of a number of messaging networks and platforms to deliver messages to users. For example, the messaging application server 114 may deliver messages using electronic mail (e-mail), instant messaging (IM), short message service (SMS), text, fax, or voice (e.g., voice over IP (VoIP)) messages via a wired (e.g., Internet), plain old telephone service (POTS), or wireless network (e.g., mobile, cellular, WiFi, long term evolution (LTE), Bluetooth).

[0044] The data stream splicing system 124 may be responsible for generating a spliced ​​data stream from data streams included in a plurality of messages received from a plurality of user computing devices (eg, client devices 102 ), as described in further detail below.

[0045] Figure 2 is a block diagram showing further details about system 100 according to an example embodiment. Specifically, system 100 is shown to include messaging client application 104 and application server 112, which in turn contains multiple subsystems, namely short timer system 202, collection management system 204 and annotation system 206.

[0046] The short-term timer system 202 is responsible for implementing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the short-term timer system 202 includes a plurality of timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on duration and display parameters associated with the message or a collection of messages (e.g., a SNAPCHAT story). Further details regarding the operation of the short-term timer system 202 are provided below.

[0047] The collection management system 204 is responsible for managing media collections (e.g., collections of text, image video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be available within a specified time period (such as the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing icons that provide notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0048] The collection management system 204 further includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate a collection of content. In some embodiments, a user may be paid a reward (e.g., money, non-monetary points or credits associated with a communication system or third-party reward system, travel miles, access to artwork or professional footage, etc.) to include user-generated content in a collection. In such cases, the curation interface 208 operates so as to automatically pay such a user for the use of their content.

[0049] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to the generation and publication of media overlays for messages processed by the networking system 100. The annotation system 206 is operable to provide a media overlay (e.g., a SNAPCHAT filter) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 is operable to provide a media overlay to the messaging client application 104 based on other information such as social network information of the user of the client device 102. The media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., images or videos) at the client device 102. For example, the media overlay includes text that can be overlaid on a photo generated by the client device 102. In another example, the media overlay includes an identification of a location overlay (e.g., Venice Beach), the name of a live event, or the name of a business overlay (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed through the database server 118.

[0050] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify the environment in which a specific media overlay will be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0051] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with a geographic location via a bidding process. For example, the annotation system 206 associates the media coverage of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

[0052] Figure 3 is a schematic diagram illustrating data that may be stored in database 120 of server system 108 according to certain example embodiments. Although the contents of database 120 are illustrated as including a plurality of tables, it is understood that data may be stored in other types of data structures (eg, as an object-oriented database).

[0053] The database 120 includes message data stored in a message table 314. The entity table 302 stores entity data, including an entity graph 304. The entities for which records are maintained in the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the type, any entity associated with which the server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).

[0054] The entity graph 304 additionally stores information about relationships and associations between entities. For example only, such relationships may be social relationships, relationships based on professional interests (e.g., working in a common company or organization), or relationships based on activities.

[0055] The database 120 also stores annotation data (e.g., in the form of filters) in the annotation table 312. The filters whose data are stored in the annotation table 312 are associated with videos (whose data are stored in the video table 310) and / or images (whose data are stored in the image table 308) and applied to the videos and / or images. In one example, a filter is an overlay displayed as an overlay on an image or video during presentation to a recipient user. The filter can be of various types, including a user-selected filter from a gallery of filters presented to a sending user by the messaging client application 104 when the sending user writes a message. Other types of filters include geographic location filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by a GPS unit of the client device 102, the messaging client application 104 can present a neighborhood or specific location-specific geographic location filter in a user interface. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other inputs or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0056] Other annotation data that may be stored in the image table 308 is so-called "shot" data. A "shot" may be real-time special effects and sounds that may be added to an image or video.

[0057] As described above, the video table 310 stores video data associated with messages in which records are maintained within the message table 314. Similarly, the image table 308 stores image data associated with messages in which message data is stored in the entity table 302. The entity table 302 may associate various annotations from the annotation table 312 with various images stored in the image table 308 and various videos in the video table 310.

[0058] The story table 306 stores data about a collection of messages and associated images, video, or audio data, which are compiled into a collection (e.g., a SNAPCHAT story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content created and sent / broadcasted by that user. To this end, the user interface of the messaging client application 104 can include a user-selectable icon to enable the sending user to add specific content to his or her personal story.

[0059] A collection may also constitute a "live story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from different locations and events. For example, a user whose client device 102 has location services enabled and who is at a public location event at a particular time may be presented with an option via the user interface of the messaging client application 104 to contribute content to a particular live story. Live stories may be identified to a user by the messaging client application 104 based on his or her location. The end result is a "live story" told from a community perspective.

[0060] Another type of content collection is called a "location story," which enables users whose client devices 102 are located in a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some embodiments, contributions to location stories may require a second degree of authentication verifying that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0061] Figure 4 is a schematic diagram illustrating the structure of a message 400 generated by a client application 104 for communicating with another client application 104 or a messaging server application 114 according to some embodiments. The content of a particular message 400 is used to populate a message table 314 stored in a database 120 that is accessible by a messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in-flight" or "in-flight" data of a client device 102 or application server 112. The message 400 is shown to include the following components:

[0062] Message identifier 402 : a unique identifier that identifies the message 400 .

[0063] Message text payload 404 : text generated by a user via the user interface of the client device 102 and included in the message 400 .

[0064] Message image payload 406 : image data captured by a camera component of the client device 102 or retrieved from a memory of the client device 102 and included in the message 400 .

[0065] Message video payload 408 : video data captured by a camera component or retrieved from a memory component of the client device 102 and included in the message 400 .

[0066] Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0067] Message annotation 412: Annotation data (eg, filter, sticker, or other enhancement) representing an annotation to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400.

[0068] Message duration parameter 414: A parameter value indicating, in seconds, the amount of time that the contents of message 400 (eg, message image payload 406, message video payload 408, message audio payload 410) will be presented to or accessible to a user via messaging client application 104.

[0069] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitudinal coordinates) associated with the content payload of the message 400. Multiple message geolocation parameter 416 values ​​may be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within the message image payload 406, or a specific video within the message video payload 408).

[0070] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "stories") associated with a particular content item in message image payload 406 of message 400. For example, multiple images within message image payload 406 may each be associated with multiple content collections using an identifier value.

[0071] Message tags 420: Each message 400 may be tagged with a plurality of tags, each tag indicating the subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tags 420 indicating the relevant animal. The tag values ​​may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0072] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 from which the message 400 was generated and sent.

[0073] • Message recipient identifier 424: Indicates an identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​of the user of the client device 102 to which the message 400 is addressed.

[0074] The content (e.g., values) of the various components of message 400 may be pointers to locations in a table where the content data values ​​are stored. For example, the image value in message image payload 406 may be a pointer to a location (or its address) within image table 308. Similarly, the value within message video payload 408 may point to data stored within video table 310, the value stored within message annotation 412 may point to data stored in annotation table 312, the value stored within message story identifier 418 may point to data stored in story table 306, and the values ​​stored within message sender identifier 422 and message recipient identifier 424 may point to user records stored within entity table 302.

[0075] Figure 5 is a schematic diagram illustrating an access restriction process 500 in which access to content (e.g., a short message 502, and an associated multimedia data payload) or a collection of content (e.g., a short message story 504) can be time-limited (e.g., short-lived).

[0076] The short message 502 is shown to be associated with a message duration parameter 506, the value of which determines the amount of time that the client application 104 displays the short message 502 to the receiving user of the short message 502. In one embodiment, where the client application 104 is a SNAPCHAT application client, the short message 502 can be viewed by the receiving user for up to 10 seconds, depending on the amount of time specified by the sending user using the message duration parameter 506.

[0077] The message duration parameter 506 and the message receiver identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the short message 502 is shown to the specific receiving user identified by the message recipient identifier 424. In particular, the short message 502 will only be shown to the relevant receiving user for a period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output for a more general short time timer, which is responsible for the overall timing of displaying content (e.g., short message 502) to the receiving user.

[0078] exist Figure 5 502 is to be included in a message story 504 (e.g., a personal SNAPCHAT story or an event story). The message story 504 has an associated story duration parameter 508, the value of which determines the duration for which the message story 504 is presented and accessible to users of the networked system 100. The story duration parameter 508 may be, for example, the duration of a concert, wherein the message story 504 is a collection of content related to the concert. Alternatively, a user (owner user or curator user) may specify a value for the story duration parameter 508 when performing setup and creation of the message story 504.

[0079] In addition, each short message 502 within the short message story 504 has an associated story engagement parameter 510, the value of which determines the duration for which the short message 502 will be accessible within the context of the short message story 504. Thus, a particular short message story 504 may "expire" and become inaccessible within the context of the short message story 504 before the short message story 504 itself expires according to the story duration parameter 508. The story duration parameter 508, the story engagement parameter 510, and the message recipient identifier 424 each provide input to the story timer 514, which first operatively determines whether a particular short message 502 of the short message story 504 will be displayed to a particular receiving user, and if so, how long to display it. Note that as a result of the message recipient identifier 424, the short message story 504 also knows the identity of the particular receiving user.

[0080] Thus, the story timer 514 is operable to control the overall lifespan of the associated short message story 504, as well as the individual short messages 502 included in the short message story 504. In one embodiment, each and all short messages 502 within the short message story 504 remain visible and accessible for the period of time specified by the story duration parameter 508. In another embodiment, a certain short message 502 may expire based on the story engagement parameter 510 within the context of the short message story 504. Note that the message duration parameter 506 may also determine the duration for which a particular short message 502 is displayed to a receiving user, even within the context of the short message story 504. Thus, the message duration parameter 506 determines the duration for which a particular short message 502 is displayed to a receiving user, regardless of whether the receiving user views the short message 502 within or outside the context of the short message story 504.

[0081] The short timer system 202 may also be operable to remove a particular short message 502 from a short message story 504 based on a determination that it has exceeded an associated story engagement parameter 510. For example, when a sending user has established a story engagement parameter 510 of 24 hours from posting, the short timer system 202 will remove the associated short message 502 from the short message story 504 after the designated 24 hours. The short timer system 202 also operates to remove a short message story 504 when the story engagement parameter 510 of each and all short messages 502 within the short message story 504 has expired, or when the short message story 504 itself has expired according to the story duration parameter 508.

[0082] In some use cases, the creator of a particular short message story 504 may specify an indefinite story duration parameter 508. In this case, the expiration of the story engagement parameter 510 for the last remaining short message 502 within the short message story 504 will determine when the short message story 504 itself expires. In this case, a new short message 502 with a new story engagement parameter 510 added to the short message story 504 effectively extends the life of the short message story 504 to a value equal to the story engagement parameter 510.

[0083] In response to the short timer system 202 determining that the short message story 504 has expired (e.g., is no longer accessible), the short timer system 202 communicates with the system 100 (and, for example, in particular the messaging client application 104) so ​​that the indicia (e.g., icon) associated with the relevant short message story 504 is no longer displayed within the user interface of the client application 104. Similarly, when the short timer system 202 determines that the message duration parameter 506 of a particular short message 502 has expired, the short timer system 202 causes the client application 104 to no longer display the indicia (e.g., icon or text logo) associated with the short message 502.

[0084] Figure 6 1 is a flow chart illustrating aspects of a method 600 for generating a spliced ​​data stream from a plurality of messages received from a plurality of user computing devices (e.g., client devices 102) according to some example embodiments. For example, the server system 108 may determine that a subset of the plurality of messages is associated with common audio content, and generate a spliced ​​data stream from the subset of the plurality of messages for an audio timeline associated with the common audio content. For illustrative purposes, the spliced ​​data stream for Figure 1 The method 600 is described with reference to the networked system 100 of FIG. 1 . It is understood that in other embodiments, the method 600 may be implemented using other system configurations.

[0085] In operation 602, the server system 108 (e.g., via a server computer associated with the data stream stitching system 124) receives a plurality of messages from a plurality of user computing devices (e.g., client devices 102). Each of the plurality of messages may include media content such as an image, a data stream, and the like. In one example, the data stream may be a video captured by a user during a concert, a lecture, or other event including audio. Each message may also include other data, such as geographic location data, a date and time stamp of the message (e.g., when the message was created or sent), data and time stamp of the data stream or other media content (e.g., when the data stream or other media content of the message was captured or recorded), and the like.

[0086] In operation 604, the server computer determines a subset of messages in the plurality of messages that are associated with similar geographic locations and time periods. For example, the server computer may analyze each of the plurality of messages to determine the geographic location of each of the plurality of messages and the time period of each of the messages. In one example, the server computer may determine the geographic location of each of the plurality of messages using data included in the message, such as the global positioning system (GPS) coordinates of the user computing device from which the message was sent. In another example, the server computer may determine the time period based on a timestamp included in the message (the timestamp indicates the time when the message was sent or the time when the media content (e.g., data stream) included in the message was collected) or based on the time when the server computer received the message, etc. The server computer may detect that a subset of messages in the plurality of messages includes a geographic location associated with a predetermined area of ​​the same GPS coordinates and a time period within a predetermined time window (e.g., 1 minute, 5 minutes, 10 minutes, 20 minutes, etc.).

[0087] In one example, the server system 108 may pre-segment the world (e.g., a world map) into predefined grids. For each grid, the server computer may aggregate all messages with geographic locations that fall into each grid within a specific time period (e.g., every ten minutes). Each grid may be the same size, or each grid size may vary based on population density in an area, the size of a town or city, etc. Thus, a subset of messages may be an aggregation of messages with geographic locations within one specific grid that occurred within the same (e.g., ten minute) time period.

[0088] In operation 606, the server computer extracts an audio fingerprint for each message in the subset of messages for use in determining whether common audio exists between the messages in the subset of messages. For example, the server computer may analyze the audio included in each message (e.g., the audio of the data stream of the message) and calculate the audio fingerprint. There are many different methods that can be used to calculate audio fingerprints. One example is to extract features from a spectrogram and use the extracted features to compare with other spectrograms to determine whether key spectrograms match and the exact locations where they match. Using a spectrogram is just one example of a method for calculating an audio fingerprint. It is understood that other methods may be used instead or in combination with other methods.

[0089] In one example, the following response is returned based on analyzing the audio included in the message to compute an audio fingerprint:

[0090]

[0091]

[0092] The spectrogram representation may be computed using a Fast Fourier Transform (FFT) across a time window.

[0093] For example, the parameters used in the FFT window may include:

[0094] SAMPLING_RATE=44100

[0095] WINDOW_SIZE=4096

[0096] OVERLAP_RATIO=0.5

[0097] Figure 7 An example of a spectrogram 700 is shown. Using the spectrogram, a diamond maxima filter with a size of NEIGHBORHOOD_SIZE=20 may be used to extract the maximum value (or peak value). Figure 8 An example of a detected maximum value of the spectrogram 800 is shown.

[0098] like Fig. 9 As shown in the spectrogram 900 of FIG. 1 , after extracting the maxima, the pairs of maxima are concatenated together to form the final audio fingerprint. First, the maxima can be sorted by the time they occur. Then, each maximum is concatenated with the first FAN_VALUE maximum after it, provided that they are within the MIN_TIME_DELTA and MAX_TIME_DELTA time frames. The fingerprint can be generated using the following three values:

[0099] 1. Frequency of the first maximum value (11 bits)

[0100] 2. Frequency of the second largest value (11 bits)

[0101] 3. Time difference between two maximum values ​​(8 bits)

[0102] In this example, these three values ​​are packed into a single unsigned integer to form an audio fingerprint. This audio fingerprint is then associated with the time frame of the first maximum value. This results in the fingerprint hashing as follows:

[0103] TIME_DELTA|FREQ1|FREQ2

[0104] 8-digit | 11-digit | 11-digit

[0105] The parameters that can be used for hashing are as follows:

[0106] FAN_VALUE=15

[0107] MIN_TIME_DELTA=1

[0108] MAX_TIME_DELTA=255

[0109] In one example, the audio fingerprint is designed in a way that is resistant to noise. Therefore, as long as there is some shared common audio between the two data streams, the two data streams can still be effectively matched.

[0110] return Figure 6 In operation 608, the server computer groups the subset of messages into a plurality of pairs of messages so that each message in the subset of the plurality of messages can be compared with each other message. Each pair of messages may include a first message and a second message.

[0111] In operation 610, the server computer compares the audio fingerprint of the first message in each pair of messages with the audio fingerprint of the second message to determine a match score for each pair of messages. In this way, the server computer can match audio fingerprints between all pairs of messages in a subset of messages (e.g., audio content in each data stream from each message). If the match score for a pair of messages is above a predetermined threshold, the server computer can consider the pair of messages to be a match (e.g., containing a portion of the same audio content) and determine the exact position where each message in the pair aligns. In one example, a match can be found if enough fingerprints match in exact temporal order.

[0112] In another example, the server computer can determine at what point in time two messages should be connected so that their intersecting tracks match exactly. The matching can be done in spectrogram space and can be accurate to 1 / 20 second. For example, the first message can include the first ten seconds of audio content (e.g., from time 00:00 to time 00:10), and the second message can include five seconds of audio content from time 00:08 to time 00:13, etc.

[0113] In one example, the server computer can determine whether there are enough pairs of messages that are all connected together into a common audio timeline. A common audio timeline is associated with a particular audio content (e.g., a song, a speech, a concert, etc.). A common audio timeline can include the full length of the audio content (e.g., the entire song, speech, concert, etc.), or a common audio timeline can include a portion of the full length of the audio content (e.g., the last 45 seconds of a song, the middle of a speech, the first ten minutes of a concert, etc.).

[0114] A predetermined threshold may be used to determine if there are enough pairs of messages that are all connected together into a common audio timeline. For example, the predetermined threshold may be a specific amount of time in which the connected messages must exist (e.g., a minimum of 30 seconds, 1 minute, 5 minutes, etc.), a specific number of messages that must be connected (e.g., two, ten, twenty, fifty, one hundred, etc.), or other thresholds. If there are enough pairs of messages connected together into a common audio timeline, the server computer may splice together the messages associated with the common audio timeline.

[0115] In operation 612, the server computer determines a group of messages in the subset of messages associated with the common audio timeline. For example, the server computer may determine a group of messages based on the matching scores of each pair of messages. In one example, a group of messages may include all messages in the subset of messages. In another example, a group of messages may be a further subset of the subset of messages (e.g., the messages in the subset of messages may not all be associated with the common audio content, but may be associated with different audio content instead).

[0116] In operation 614, the server computer stitches together a group of messages to generate a stitched data stream for the data stream of each message in the group of messages. For example, the server computer may stitch together a group of messages based on the time period of each message. For example, message A may be used for the time period 00:10-00:20 in the common audio timeline, message B may be used for the time period 00:20-00:23, message C may be used for the time period 00:22-00:31, and so on. The data streams may overlap in the time period, so that there may be more than one data stream in a given time period.

[0117] In one example, the server computer may determine a start message for splicing. The start message may be the message that occurs first in a common audio timeline of a group of messages. For example, there may be one message with a time period starting at 00:10 in the audio timeline, and another message with a time period starting at 00:15 in the audio timeline, and so on. The server computer determines which message is the earliest in the audio timeline (e.g., the message starting at time 00:10).

[0118] In one example, there may be more than one message with a similar start time, any of which may be the start message. For example, a first message may start at time 00:10:01, a second message may start at 00:10:02, and a third message may start at 00:10:01. The server computer may select the start message from the messages by random selection, based on a quality score of the message indicating a quality level of the message (e.g., image clarity, audio clarity, stability of the stream, etc.), based on an attention score (e.g., based on user attention), or other means, or a combination of these methods.

[0119] In one example, the server computer determines all possible starting points for splicing by considering messages or data streams from only the messages that appear in front of all matches. The server computer can then run a recursive breadth-first search starting at each starting point to generate a dense audio splice from the audio matches.

[0120] In another example, the server computer can optimize the splice. For example, the optimized splice can include the minimum number of messages or data streams from messages necessary to span the entire dense splice. These can be efficiently calculated for each dense splice using a greedy algorithm. For example, starting with a first message or data stream, the server computer can determine the longest message or data stream that intersects the first message to select a second message. Then, the server computer can determine the longest message or data stream that intersects the second message to select a third message, and so on, until the entire common audio timeline is completed.

[0121] Fig.10A visual representation of a spliced ​​data stream 1000 is shown, which includes multiple messages or data streams 1002-1030 received from various users via multiple user devices. Each box 1002-1030 represents a message or data stream in the spliced ​​data stream 1000 received from a user. For example, each message or data stream in each box 1002-1030 can include a time period during a common audio timeline (e.g., 1002 can be the first 10 seconds of the common audio timeline, 1004 can be the next 10 seconds of the common audio timeline, 1006 can be the next 5 seconds of the common audio timeline, etc.). In this example, the spliced ​​data stream 1000 includes messages from a live audio event.

[0122] Fig.11 Another visual representation of a spliced ​​data stream 1100 is shown, which includes multiple messages or data streams received from various users via multiple user devices. Fig.10 in Fig.11 , each box represents a message or data stream received from a user in a spliced ​​data stream 1100. In this example, the spliced ​​data stream 1100 includes messages from a live sporting event.

[0123] In one example, the spliced ​​data stream may include messages that are prioritized based on a random selection of an order of display in a timeline. In another example, the spliced ​​data stream may include messages that are prioritized according to one or more rules. For example, the spliced ​​data stream may include messages that are prioritized based on a quality score for an order of display in a common audio timeline. In one example, each message or data stream may be analyzed to determine a score for the quality of the data stream based on image quality, jitter of the data stream, audio quality, lighting, etc.

[0124] In another example, the stitched data stream may include messages that are prioritized based on a user or a user profile of the user associated with the user computing device. For example, the stitched data stream may include messages from the user or from other users associated with the user (e.g., friends, family, other users that the user "follows," etc.). The server computer may prioritize messages generated or sent by the user or other users associated with the user.

[0125] Reference again Figure 6In operation 616, the server computer provides the spliced ​​data stream to the one or more user computing devices. In one example, the server computer may send the spliced ​​data stream to the one or more computing devices. In another example, the server computer may provide access to the spliced ​​data stream from the one or more computing devices (such as via a connection or otherwise), which will allow the user computing device to access the spliced ​​data stream on the server computer (e.g., stream the spliced ​​data stream from the server computer).

[0126] like Figure 13-15 As shown in the example of , one or more user devices can display the spliced ​​data stream on a display of one or more user devices. In one example, the spliced ​​data stream is displayed on the user device as a continuous data stream that transitions from one message data stream to the next message data stream to a continuous common audio stream. In one example, the audio from each data stream of the spliced ​​data stream is averaged to provide a better quality audio stream. Thus, the continuous common audio stream can include audio that is an average of the audio associated with each data stream in the spliced ​​data stream to provide a better quality audio stream, or the continuous audio stream can include the original audio received in multiple messages.

[0127] Example embodiments allow a user to switch the display between alternate data streams within a given time period in a spliced ​​data stream. Figure 13-15 , a user may be viewing display 1306 of a spliced ​​data stream of a recent concert on computing device 1302. Display 1306 may show the lead singer in the concert, and the user may want to see other views of that time in the audio timeline. In one example, display 1306 may indicate in some way that there are alternative views of the currently viewed data stream (e.g., via button 1304, a highlighted frame, a pop-up message, or other indicator). The user may move the device (e.g., pan, tilt, etc.), touch display 1306 of the device (e.g., touch a button or connection on display 1306), or otherwise provide an indication that he wishes to view an alternative display. Upon receiving the indication from the user, the user computing device 1302 may display an alternative view (e.g., a data stream) that may show, for example, Fig.14 The audience shown in Fig.15 The stitched data stream would then continue to play from that view until the end of that view, and then transition to the next view aligned with that view, etc. Thus, the display of the stitched data stream transitions to the next message data stream when the previous data stream ends or when the user indicates that he wishes to view an alternate display.

[0128] Fig.121 is a flow chart illustrating aspects of a method 1200 for providing an alternative data stream for display on a user computing device (eg, client device 102) according to some example embodiments. Figure 1 The method 1200 is described with reference to the networked system 100 of FIG. It is understood that in other embodiments, the method 1200 may be implemented using other system configurations.

[0129] In operation 1202, a computing device receives a request for a replacement data stream. The computing device in these examples can be a user computing device (e.g., client device 102) or a server computer (e.g., a server computer associated with server system 118). For example, during the display of a spliced ​​data stream including multiple individual data streams associated with a common audio timeline on a user computing device, the user computing device or server computer can receive a request for a replacement data stream for multiple data streams instead of the active data stream currently displayed on the computing device (e.g., depending on whether the spliced ​​data stream is located on the user computing device or streamed from the server computer, and / or whether the function for providing the replacement data stream is located on the user authoring device or on the server computer). As explained in further detail above, multiple data streams associated with a common audio timeline are received from multiple computing devices and spliced ​​together to form a spliced ​​data stream. As described above, in one example, a common audio timeline can be used for audio content that is an average of audio associated with multiple individual data streams of the spliced ​​data stream.

[0130] In one example, as described above, a user may indicate a desire to switch to an alternate data stream in a display on a user computing device. For example, as described above (e.g., via a motion input, a touch input, etc.), the user computing device may receive an indication from a user of the computing device to switch to an alternate data stream. In one example, the user computing device may send a request to the server system 108, or in another example, the user computing device may process the request itself. If the user computing device sends a request to the server system 108, a computing device such as a server computer receives the request for the alternate data stream. If the user computing device processes the request itself, the user computing device receives the request for the alternate data stream.

[0131] In operation 1204, the computing device determines a subset of multiple individual data streams of the individual data streams of the spliced ​​data stream associated with the time period of the active data stream in the common audio timeline. For example, the computing device can determine the time period of the active data stream based on the data received in the request for the replacement data stream. In one example, the computing device can determine that the active data stream is associated with the time period 00:30 to 00:33 of the common audio timeline. The request for the replacement data stream may include the time period, or may include the current timestamp (e.g., 00:30:49, 00:32:10, etc.) at which the active data stream is being displayed when the request is sent. The computing device can analyze multiple data streams to determine which of the multiple data streams includes a time period that overlaps with the time period of the active data stream.

[0132] In operation 1206, the computing device selects a replacement data stream from a subset of multiple individual data streams of the spliced ​​data stream associated with the time period of the active data stream in the common audio timeline. In one example, in order to select a replacement data stream from a subset of multiple data streams, the computing device may determine the quality score of each data stream in the subset of multiple data streams of the spliced ​​data stream associated with the time period of the active data stream. In this example, the computing device selects the replacement data stream based on the quality score of the replacement data stream. For example, the computing device may select the data stream with the highest quality score of the replacement data stream. In one example, the quality score may be based on the quality score of the video or image in the data stream and / or the attention score of the data stream for multiple users. In another example, the computing device may randomly select the replacement data stream from a subset of multiple data streams.

[0133] In operation 1208, the computing device may provide the alternative data stream for displaying the computing device. In one example, the server computer may provide the alternative data stream to the user computing device for display to the user. In another example, the user computing device may display the alternative data stream to the user.

[0134] The display of the active data stream on the computing device can be switched to the alternate data stream on the computing device in the common audio timeline. In this way, the user can view the alternate data stream as described above. The display of the spliced ​​data stream continues from the alternate data stream and switches to the next data stream in the plurality of data streams that occurs after the alternate data stream ends in the common audio timeline.

[0135] Fig.16 16 is a block diagram 1600 showing a software architecture 1602 that may be installed on any one or more of the above devices. For example, in various embodiments, some or all elements of the software architecture 1602 may be used to implement the client device 102 and the server system 108 (including the server systems 110, 112, 114, 116, 118, 122, and 124). Fig.16 This is merely a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 1602 is comprised of, for example, Fig.17 1700, which includes a processor 1710, a memory 1730, and an I / O component 1750. In this example, the software architecture 1602 can be conceptualized as a stack of layers, each of which can provide specific functionality. For example, the software architecture 1602 includes layers such as an operating system 1604, a library 1606, a framework 1608, and an application 1610. In operation, consistent with some embodiments, the application 1610 calls an application programming interface (API) call 1612 through the software stack and receives a message 1614 in response to the API call 1612.

[0136] In various embodiments, the operating system 1604 manages hardware resources and provides public services. The operating system 1604 includes, for example, a kernel 1620, services 1622, and drivers 1624. Consistent with some embodiments, the kernel 1620 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 1620 provides functions such as memory management, processor management (e.g., scheduling), component management, network connections, and security settings. Services 1622 can provide other public services to other software layers. According to some embodiments, the driver 1624 is responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, the driver 1624 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, serial communication drivers (such as Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0137] In some embodiments, the library 1606 provides a low-level common infrastructure utilized by the application 1610. The library 1606 may include a system library 1630 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1606 may include an API library 1632, such as a media library (e.g., a library that supports the rendering and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering two-dimensional (2D) and three-dimensional (3D) graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1606 may also include a variety of other libraries 1634 to provide many other APIs to the application 1610.

[0138] According to some embodiments, the framework 1608 provides a high-level common architecture that can be utilized by the applications 1610. For example, the framework 1608 provides various graphical user interface (GUI) functions, high-level resource management, advanced location services, etc. The framework 1608 can provide a wide range of other APIs that can be utilized by the applications 1610, some of which can be specific to a particular operating system 1604 or platform.

[0139] In an example embodiment, applications 1610 include a home application 1650, a contacts application 1652, a browser application 1654, a book reader application 1656, a location application 1658, a media application 1660, a messaging application 1662, a game application 1664, and a broad category of other applications such as third-party applications 1666 and media content applications 1667. According to some embodiments, application 1610 is a program that executes the functions defined in the program. Various programming languages ​​can be used to create one or more applications 1610 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party application 1666 (e.g., a third-party application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on mobile operating systems such as IOS TM ANDROID TM , PHONE or other mobile operating system). In this example, the third-party application 1666 can call the API call 1612 provided by the operating system 1604 to perform the functions described herein.

[0140] As described above, some embodiments may specifically include a messaging application 1662. In some embodiments, this may be a stand-alone application that is used to manage communications with a server system such as server system 108. In other embodiments, this functionality may be integrated with another application such as a media content viewing application 1667. The messaging application 1662 may request and display various media content items, and may provide the user with the ability to enter data related to media content items via a touch interface, keyboard, or using a camera device of the machine 1700, communicate with the server system 108 via I / O components 1750, and receive and store media content items in memory 1730. The presentation of media content items and user input associated with the media content items may be managed by the messaging application 1662 using different frameworks 1608, library 1606 elements, or operating system 1604 elements operating on the machine 1700.

[0141] Fig.17 1 is a block diagram illustrating components of a machine 1700 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein in accordance with some embodiments. Specifically, Fig.17A schematic diagram of a machine 1700 in the example form of a computer system within which instructions 1716 (e.g., software, programs, applications 1610, applet, application program, or other executable code) may be executed for causing the machine 1700 to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine 1700 operates as a standalone device or may be coupled (e.g., via a network connection) to other machines. In a networked deployment, the machine 1700 may operate in the capacity of a server system 108, 110, 112, 114, 116, 118, 122, 124, etc. or a client device 102 in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1700 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1716 that specify actions to be taken by the machine 1700, either continuously or otherwise. In addition, while only a single machine 1700 is shown, the term "machine" may also be taken to include a collection of machines 1700 that individually or jointly execute instructions 1716 to perform any one or more of the methodologies discussed herein.

[0142] In various embodiments, the machine 1700 includes a processor 1710, a memory 1730, and an I / O component 1750, which may be configured to communicate with each other via a bus 1702. In an example embodiment, the processor 1710 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) includes, for example, a processor 1712 and a processor 1714 that may execute instructions 1716. The term "processor" is intended to include a multi-core processor 1710 that may include two or more independent processors 1712, 1714 (also referred to as "cores") that may execute instructions 1716 simultaneously. Although Fig.17 Multiple processors 1710 are shown, but the machine 1700 may include a single processor 1710 having a single core, a single processor 1710 having multiple cores (e.g., a multi-core processor 1710), multiple processors 1712, 1714 having single cores, multiple processors 1710, 1712 having multiple cores, or any combination thereof.

[0143] According to some embodiments, the memory 1730 includes a main memory 1732, a static memory 1734, and a storage unit 1736 accessible to the processor 1710 via the bus 1702. The storage unit 1736 may include a machine-readable medium 1738 on which are stored instructions 1716 embodying any one or more of the methods or functions described herein. The instructions 1716 may also reside, completely or at least partially, within the main memory 1732, within the static memory 1734, within at least one of the processors 1710 (e.g., within a cache memory of the processor), or within any suitable combination thereof during execution thereof by the machine 1700. Thus, in various embodiments, the main memory 1732, the static memory 1734, and the processor 1710 are referred to as the machine-readable medium 1738.

[0144] As used herein, the term "memory" refers to a machine-readable medium 1738 capable of storing data temporarily or permanently, and may be considered to include, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, flash memory, and cache. Although the machine-readable medium 1738 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions 1716. The term "machine-readable medium" may also be considered to include any medium or combination of multiple media capable of storing instructions (e.g., instructions 1716) for execution by a machine (e.g., machine 1700), such that the instructions 1716, when executed by one or more processors (e.g., processor 1710) of the machine 1700, causes the machine 1700 to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network including multiple storage devices or apparatuses. Thus, the term "machine-readable medium" may be taken to include, but is not limited to, one or more data storage repositories in the form of solid-state memory (e.g., flash memory), optical media, magnetic media, other non-volatile memory (e.g., erasable programmable read-only memory (EPROM)), or any suitable combination thereof. The term "machine-readable medium" specifically excludes non-transitory signals per se.

[0145] I / O components 1750 include a variety of components for receiving input, providing output, generating output, sending information, exchanging information, collecting measurements, etc. In general, it is understood that I / O components 1750 may include Fig.17Many other components not shown in the figure. The I / O components 1750 are grouped according to function, which is only used to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1750 include output components 1752 and input components 1754. The output components 1752 include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components 1754 include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide the location and force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0146] In some further example embodiments, the I / O component 1750 includes, among various other components, a biometric component 1756, a motion component 1758, an environmental component 1760, or a position component 1762. For example, the biometric component 1756 includes a component that detects expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measures biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifies a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1758 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 1760 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor component (e.g., a machine olfactory detection sensor, a gas detection sensor for detecting dangerous gas concentrations for safety or measuring pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1762 includes a positioning sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that can detect the air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0147] Communication can be achieved using a variety of technologies. I / O components 1750 may include a communication component 1764 that is operable to couple machine 1700 to network 1780 or device 1770 via coupler 1782 and coupler 1772, respectively. For example, communication component 1764 includes a network interface component or another suitable device that interfaces with network 1780. In other examples, communication component 1764 includes a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g., low power ), Components and other communication components that provide communication via other modes. Device 1770 may be another machine 1700 or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0148] In addition, in some embodiments, the communication component 1764 detects an identifier or includes a component operable to detect an identifier. For example, the communication component 1764 includes a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, digital graphics, Max codes, PDF417, Super codes, Uniform Commercial Code Reduced Space Symbols (UCC RSS)-2D bar codes, and other optical codes), an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag), or any suitable combination thereof. In addition, various information can be derived via the communication component 1764 that can indicate a particular location, such as a location via an Internet Protocol (IP) geographic location, a location via a wireless network ... Signal triangulation location, via detection Or the location of an NFC beacon signal, etc.

[0149] In various example embodiments, one or more portions of network 1780 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 1780, or a portion of network 1780, may include a wireless or cellular network, and coupling 1782 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 1782 may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0150] In an example embodiment, instructions 1716 are sent or received over a network 1780 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1764), and utilize any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, in other example embodiments, instructions 1716 are sent or received to device 1770 using a transmission medium via coupling 1772 (e.g., a peer-to-peer coupling). The term "transmission medium" may be taken to include any intangible medium capable of storing, encoding, or carrying instructions 1716 to be executed by machine 1700, and includes digital or analog communication signals or other intangible media to facilitate communication of such software implementations.

[0151] Furthermore, because machine-readable medium 1738 does not contain propagating signals, machine-readable medium 1738 is non-transitory (in other words, does not have any short-lived signals). However, marking machine-readable medium 1738 as "non-transitory" should not be interpreted to mean that the medium cannot be moved. Medium 1738 should be considered to be transportable from one physical location to another. In addition, because machine-readable storage medium 1738 is tangible, medium 1738 can be considered a machine-readable device.

[0152] Throughout the specification, multiple instances can be implemented as components, operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as individual operations, one or more individual operations can be performed in parallel, and the operations do not need to be performed in the order shown. The structure and function presented as the individual components in the example configuration can be implemented as a combined structure or component. Similarly, the structure and function presented as a single component can be implemented as a plurality of components separated. These and other variations, modifications, additions and improvements fall within the scope of this paper theme.

[0153] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure.

[0154] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered restrictive, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents granted by these claims.

[0155] As used herein, the term "or" may be interpreted in an inclusive or exclusive manner. In addition, multiple instances may be provided as a single instance for the resources, operations, or structures described herein. In addition, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and specific operations are shown in the context of a specific illustrative configuration. Other allocations of functions may be conceivable, and these other allocations may fall within the scope of various embodiments of the present disclosure. Typically, structures and functions presented as separate resources in an example configuration may be implemented as a combined structure or resource. Similarly, structures and functions presented as a single resource may be implemented as separate resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and drawings are considered to be illustrative rather than restrictive.

Claims

1. A method performed by at least one processor, the method comprising: causing display of a spliced ​​data stream as a continuous data stream transitioning from one message data stream to the next message data stream to a continuous audio stream, the spliced ​​data stream comprising a plurality of individual message data streams from messages received from a plurality of computing devices and associated with a common audio timeline; During display of the spliced ​​data stream, receiving a request for a replacement message data stream in the spliced ​​data stream to replace a currently displayed active message data stream; as well as Display of the active message data stream is caused to transition in the common audio timeline to display the replacement message data stream, and after the replacement message data stream ends, display of the spliced ​​data stream is transitioned to the next message data stream in the spliced ​​data stream.

2. The method according to claim 1, wherein: The spliced ​​data stream also includes message data streams having overlapping time periods on the common audio timeline, such that more than one message data stream exists in at least one time period of the spliced ​​data stream.

3. The method according to claim 1, wherein: Prior to causing display of the active message data stream to transition to display of the replacement message data stream, the method includes selecting the replacement message data stream from the message data streams of the spliced ​​data stream having a time period in the common audio timeline corresponding to a time period of the active message data stream by performing operations including: determining a quality score for each message data stream having a time period in the common audio timeline corresponding to the time period of the active message data stream; and Based on the quality scores of the replacement message data streams, the replacement message data stream is selected from the message data streams having a time period in the common audio timeline that corresponds to the time period of the active data stream.

4. The method according to claim 3, wherein: The quality score is based on at least one of a quality of a video or image in the message data stream and an attention score of the message data stream for a plurality of users.

5. The method according to claim 1, wherein: Prior to causing the display of the active message data stream to switch to displaying the replacement message data stream, the method includes: The replacement message data stream having a time period in the common audio timeline corresponding to the time period of the active message data stream is selected from the message data streams of the spliced ​​data stream by randomly selecting the replacement message data stream from the message data streams having a time period in the common audio timeline corresponding to the time period of the active message data stream.

6. The method according to claim 5, further comprising: The time period of the active message data stream is determined based on data received in the request for the replacement message data stream.

7. The method according to claim 1, wherein: Prior to causing the display of the active message data stream to switch to displaying the replacement message data stream, the method includes: The message data streams in the spliced ​​data stream are analyzed to determine a message data stream having a time period overlapping with a time period of the active message data stream.

8. The method according to claim 1, further comprising: receiving an instruction from a user of the computing device to switch to an alternate message data stream; as well as The request for the replacement message data stream is sent.

9. The method according to claim 1, wherein: The common audio timeline is for audio content of an average of audio associated with the plurality of individual message data streams of the spliced ​​data stream.

10. A server computer, comprising: processor; as well as A computer-readable medium coupled to the processor, the computer-readable medium including instructions stored thereon, the instructions executable by the processor to cause a computing device to perform operations including: causing display of a spliced ​​data stream as a continuous data stream transitioning from one message data stream to the next message data stream to a continuous audio stream, the spliced ​​data stream comprising a plurality of individual message data streams from messages received from a plurality of computing devices and associated with a common audio timeline; During display of the spliced ​​data stream, receiving a request for a replacement message data stream in the spliced ​​data stream to replace a currently displayed active message data stream; as well as Display of the active message data stream is caused to transition in the common audio timeline to display the replacement message data stream, and after the replacement message data stream ends, display of the spliced ​​data stream is transitioned to the next message data stream in the spliced ​​data stream.

11. The server computer according to claim 10, wherein: The spliced ​​data stream also includes message data streams having overlapping time periods on the common audio timeline, such that more than one message data stream exists in at least one time period of the spliced ​​data stream.

12. The server computer according to claim 10, wherein: Prior to causing the display of the active message data stream to switch to displaying the replacement message data stream, the operations include: selecting the replacement message data stream from the message data streams of the spliced ​​data stream having a time period in the common audio timeline corresponding to a time period of the active message data stream by performing operations including: determining a quality score for each message data stream having a time period in the common audio timeline corresponding to the time period of the active message data stream; and Based on the quality scores of the replacement message data streams, the replacement message data stream is selected from the message data streams having a time period in the common audio timeline that corresponds to the time period of the active data stream.

13. The server computer according to claim 12, wherein: The quality score is based on at least one of a quality of a video or image in the message data stream and an attention score of the message data stream for a plurality of users.

14. The server computer according to claim 10, wherein: Before causing the display of the active message data stream to switch to displaying the replacement message data stream, the operation includes: selecting the replacement message data stream having a time period corresponding to the time period of the active message data stream in the common audio timeline from the message data streams of the spliced ​​data stream by randomly selecting the replacement message data stream from the message data streams having a time period corresponding to the time period of the active message data stream in the common audio timeline.

15. The server computer of claim 14, the operations further comprising: The time period of the active message data stream is determined based on data received in the request for the replacement message data stream.

16. The server computer according to claim 10, wherein: Prior to causing the display of the active message data stream to switch to displaying the replacement message data stream, the operations include: The message data streams in the spliced ​​data stream are analyzed to determine a message data stream having a time period overlapping with a time period of the active message data stream.

17. The server computer according to claim 10, wherein: The common audio timeline is for audio content of an average of audio associated with the plurality of individual message data streams of the spliced ​​data stream.

18. A non-transitory computer-readable medium comprising instructions stored thereon, the instructions executable by at least one processor to cause a computing device to perform operations comprising: causing display of a spliced ​​data stream as a continuous data stream transitioning from one message data stream to the next message data stream to a continuous audio stream, the spliced ​​data stream comprising a plurality of individual message data streams from messages received from a plurality of computing devices and associated with a common audio timeline; During display of the spliced ​​data stream, receiving a request for a replacement message data stream in the spliced ​​data stream to replace a currently displayed active message data stream; as well as Display of the active message data stream is caused to transition in the common audio timeline to display the replacement message data stream, and after the replacement message data stream ends, display of the spliced ​​data stream is transitioned to the next message data stream in the spliced ​​data stream.

Images