Visual tag emerging pattern detection

Abstract

Systems, devices, media, and methods for identifying emerging viewing patterns for visual media such as still images and videos are provided. Emerging viewing patterns are identified by identifying visual tags for visual media viewed by users, selecting a subset of tags by applying a classification-based filter, generating pattern candidates from the subset, evaluating consumption metrics for each of the generated patterns, and ranking the generated pattern candidates in response to the consumption metrics to identify emerging viewing patterns for users.

Description

Technology Field

[0001] This application claims priority to U.S. application serial number 17 / 120,933 filed on December 14, 2020, the contents of which are incorporated herein by reference.

[0002] The examples described in this disclosure generally relate to applications running on client devices and server systems supporting such devices. More specifically, this disclosure covers, but is not limited to, systems and methods for identifying emerging visual media viewing patterns, for example, to customize content and advertisements for viewers.

[0003] Visual media, such as still images and videos, are routinely viewed by users through electronic devices. Visual media is typically stored by server systems and transmitted to users' electronic devices for viewing. Brief explanation of the drawing

[0004] The disclosed subject matter of the claim is best understood from the following detailed description when read in relation to the attached drawings, in which the same elements have the same reference numerals. Where multiple similar elements exist, a single reference numeral may be assigned to multiple similar elements, with specific elements designated by lowercase designations. Lowercase designations may be omitted when designating elements collectively or when designating one or more non-specific elements. To facilitate the identification of any specific element or operation, the most significant number or number of reference numerals designates the drawing number where the element is first introduced. This emphasizes that, in general practice, various features of the drawings are not drawn to actual scale unless otherwise indicated. Conversely, the dimensions of various features may be enlarged or reduced for clarity. The drawings include the following drawings.

[0005] Figure 1 is a block diagram of an exemplary emerging pattern detection system for an online client-server system.

[0006] FIG. 2a is a block diagram illustrating an exemplary emerging pattern detection system.

[0007] FIG. 2b is a block diagram illustrating exemplary components for implementing emerging pattern detection for a client device.

[0008] FIG. 3 is a block diagram illustrating server and client components for implementing emerging pattern detection.

[0009] Figure 4 is a flowchart illustrating an exemplary method for implementing emerging pattern detection on a server system.

[0010] FIG. 5 is a flowchart illustrating an exemplary method for collecting data from a client device to implement emerging pattern detection on a server system.

[0011] FIG. 6 is a flowchart illustrating an exemplary method for generating pattern candidates to be used in the exemplary method of FIG. 4.

[0012] FIG. 7 is a flowchart illustrating an exemplary method for evaluating pattern candidates to be used in the exemplary method of FIG. 4.

[0013] FIG. 8 is a schematic diagram of an exemplary hardware configuration for a client device implemented as a mobile device.

[0014] FIG. 9 is a schematic representation of a machine in the form of a computer system according to some examples, wherein a set of instructions to cause the machine to perform any one or more of the methodologies described herein may be executed.

[0015] FIG. 10 is a block diagram illustrating a software architecture in which aspects of the present disclosure may be implemented according to some examples. Specific details for implementing the invention

[0016] The aspects of the claimed subject matter disclosed herein relate to identifying emerging viewing patterns for visual media (e.g., still images or video) using visual tags. On social media platforms, users generate various types of content that become available for consumption by other users. Consumption may change over time depending on user cohorts (i.e., age group, gender), location, or external events (i.e., politics, economy, culture). Understanding the topics viewers are engaged with is useful for (i) recommending new content, (ii) retaining users, and (iii) finding new partnerships. Instead of viewing all available visual tags individually through computer vision detection, filtering techniques are applied to select more specific visual tags (e.g., only leaf nodes in a classification system), and combinations of these selected tags are evaluated to effectively identify emerging viewing patterns and trends.

[0017] The following description includes systems, methods, techniques, instruction sequences, and computing machine program products that exemplify examples of the present disclosure. In the following description, for the purposes of explanation, many specific details are provided to provide an understanding of various examples of the disclosed claims. However, it will be apparent to those skilled in the art that examples of the disclosed claims can be practiced without these specific details. Generally, well-known instruction instances, protocols, structures, and techniques are not necessarily illustrated in detail.

[0018] According to one example, a method for identifying emerging viewing patterns for visual media (e.g., still images or video) viewed by groups of users is provided. The method identifies visual tags for each of a plurality of visual media viewed by a first group of groups of users, selects a subset of the identified visual tags by applying a classification-based filter, generates pattern candidates from the selected subset, each pattern candidate includes two or more of the identified visual tags from the subset of the identified visual tags, evaluates consumption metrics of the visual media by the first group of users for each of the generated pattern candidates, and identifies emerging viewing patterns by ranking the generated pattern candidates in response to the evaluated consumption metrics to identify emerging viewing patterns for the first group of users.

[0019] According to another example, a system for identifying emerging viewing patterns is provided. The system includes a processor. The processor identifies visual tags for each of a plurality of visual media viewed by a first group of groups of users, selects a subset of the identified visual tags by applying a classification-based filter, generates pattern candidates from the selected subset — each pattern candidate includes two or more of the identified visual tags from the subset of the identified visual tags — evaluates consumption metrics of the visual media by the first group of users for each of the generated pattern candidates, and is configured to rank the generated pattern candidates in response to the evaluated consumption metrics to identify emerging viewing patterns for the first group of users.

[0020] According to another example, a non-transient processor-readable storage medium storing processor-executable instructions is provided, wherein the processor-executable instructions, when executed by the processor of the machine, cause the machine to perform operations to identify emerging viewing patterns of visual media. The operations performed by the machine include: identifying visual tags for each of a plurality of visual media viewed by a first group of groups of users; selecting a subset of identified visual tags by applying a classification-based filter; generating pattern candidates from the selected subset — each pattern candidate includes two or more of the identified visual tags from the subset of identified visual tags —; evaluating consumption metrics of the visual media by the first group of users for each of the generated pattern candidates; and ranking the generated pattern candidates in response to the evaluated consumption metrics to identify emerging viewing patterns for the first group of users.

[0021] FIG. 1 is a block diagram illustrating a system (100) according to some examples, and the system (100) is configured to identify emerging viewing patterns of users. The system (100) includes one or more client devices, such as a client device (110). The client device (110) includes, but is not limited to, a mobile phone, a desktop computer, a laptop, a PDA (portable digital assistant), a smartphone, a tablet, an ultrabook, a netbook, a laptop, a multi-processor system, a microprocessor-based or programmable consumer electronics, an electronic device, a game console, a set-top box, a vehicle computer, or any other communication device that a user may use to access the system (100). In some examples, the client device (110) includes a display module (not shown) for displaying information (e.g., in the form of user interfaces). In additional examples, the client device (110) includes one or more of touch screens, accelerometers, gyroscopes, cameras, microphones, and GPS (Global Positioning System) devices. The client device (110) may be a user's device used to access and utilize an online social platform. For example, the client device (110) may be used to create an account by entering information, exchange information through the network (102), etc.

[0022] In one example, the client device (110) is a user's device that is using a social media application on the device. The client device (110) may call a server for a social platform (e.g., hosted by the server system (108)) directly through the social media application or through one or more third-party servers (128) (e.g., utilizing one or more third-party applications (130)). The application server (104) tracks the visual media provided to the client device (110) and the corresponding viewing statistics collected from the client device (110) (e.g., duration of video, video viewing time, number of times an image is viewed, etc.) as a dataset in the database (126). By analyzing the dataset using the techniques disclosed herein, the application server (104) can automatically detect viewing patterns / preferences and deliver visual media to the client device according to the viewing patterns / preferences.

[0023] One or more users may be people, machines, or other means of interacting with the client device (110). In examples, the user may not be part of the system (100) but may interact with the system (100) through the client device (110) or other means. For instance, the user may provide input (e.g., touch screen input or alphanumeric input) to the client device (110), and the input may be communicated to other entities of the system (100) (e.g., third-party servers (128), server system (108), etc.) via the network (102). In such cases, other entities of the system (100) may communicate information to be presented to the user to the client device (110) via the network (102) in response to receiving input from the user. In this way, the user interacts with various entities of the system (100) using the client device (110).

[0024] The system (100) further includes a network (102). One or more parts of the network (102) may be an ad hoc network, an intranet, an extranet, a VPN (virtual private network), a LAN (local area network), a WLAN (wireless LAN), a WAN (wide area network), a WWAN (wireless WAN), a MAN (metropolitan area network), part of the Internet, part of a PSTN (public switched telephone network), a cellular telephone network, a wireless network, a WiFi network, other types of networks, or a combination of two or more of these networks.

[0025] A client device (110) can access various data and applications provided by other entities of the system (100) through a web client (112) (e.g., a browser) and / or one or more client applications (114). The client device (110) may include one or more client application(s) (114) (also referred to as “apps”) such as (but not limited to) a web browser, a messaging application, an email application, an e-commerce site application, a mapping or location application, etc. Client applications may also display visual media on the display of the client device (110), collect viewing data for visual media viewed by a user of the electronic device, and generate viewing consumption metrics.

[0026] In some examples, one or more client application(s) (114) are included in a given part of the client device (110) and are configured to provide at least some of the user interfaces and functions locally, and the client application(s) (114) are configured to communicate as needed with other entities of the system (100) (e.g., third-party server(s) (128), server system (108), etc.) for data processing capabilities that are not locally available (e.g., to perform tasks such as accessing location information, authenticating a user, etc.). Conversely, one or more client application(s) (114) may not be included in the client device (110), and the client device (110) may use its own web browser to access one or more applications hosted on other entities of the system (100) (e.g., third-party server(s) (128), server system (108), etc.).

[0027] The server system (108) provides server-side functions to one or more third-party server(s) (128) and one or more client devices (110) via a network (102) (e.g., the Internet or a WAN (wide area network)). The server system (108) includes an application program interface (API) server (120), a web server (122), and a pattern detection system (124) that can be coupled to communicate with one or more database(s) (126). One or more database(s) (126) may be storage devices that store data (e.g., datasets) related to users of the server system (108), applications associated with the server system (108), viewing consumption metrics (e.g., based on demographic information), cloud services, housing market data, etc. One or more database(s) (126) may further store information related to third-party server(s) (128), third-party application(s) (130), client device (110), client application(s) (114), users, etc. In one example, one or more database(s) (126) may be cloud-based storage.

[0028] According to some examples, the server system (108) may be a cloud computing environment. The server system (108) and any servers associated with the server system (108) may be associated with a cloud-based application. In one example, the server system (108) includes a pattern detection system (124) for detecting emerging patterns in visual media viewing. The pattern detection system (124) may include one or more servers and may be associated with a cloud-based application. The pattern detection system (124) may distribute visual metric applications to collect and store viewing metrics on client devices (e.g., as part of a social media application or as an update thereto), obtain viewing metrics from the distributed apps on these client devices, and store the results in a database(s) (126). The pattern detection system (124) analyzes a dataset containing viewing metrics from client devices to identify emerging viewing patterns.

[0029] The system (100) further includes one or more third-party server(s) (128). One or more third-party server(s) (128) may include one or more third-party application(s) (130). One or more third-party application(s) (130) running on the third-party server(s) (128) may interact with the server system (108) through the API server (120) via a program interface provided by the API server (120). For example, one or more of the third-party applications (130) may request and utilize information from the server system (108) through the API server (120) to support one or more features or functions on an application hosted by a third party or a website hosted by a third party.

[0030] FIG. 2a is a block diagram illustrating an example of an emerging pattern detection system (124). The pattern detection system (124) includes a visual tag engine (202), a classification system engine (204), a candidate generator (206), a metric evaluation engine (208), and a viewing metric database (210). The viewing metric database (210) is stored in a database (126) of a server system (108). The device capability collection engine (202) and the device inference engine are executed on an application server (104) of the server system (108). The visual tag engine (202), the classification system engine (204), the candidate generator (206), and the metric evaluation engine (208) are executed on the server system (108).

[0031] The visual tag engine (202) of the device capability collection engine (202) is configured to analyze visual media and generate tags corresponding to objects identified in the visual media (e.g., sun, water, boy, girl, etc.). In one example, the process of generating tags is implemented using a machine-learning algorithm that compares the visual media to a database trained using the collection of tagged images.

[0032] Machine learning refers to algorithms that improve gradually through experience. By processing a large number of different input datasets, machine learning algorithms develop improved generalizations for specific datasets and can subsequently use these generalizations to generate accurate outputs or solutions when processing new datasets. Generally, a machine learning algorithm includes one or more parameters that gradually improve the algorithm by adjusting or changing in response to new experiences; and a learning-like process.

[0033] In the context of computer vision, mathematical models attempt to emulate tasks achieved by the human visual system, where the goal of using computers is to extract information from images and accurately understand their content. Computer vision algorithms have been developed in various fields, including artificial intelligence and autonomous driving, to extract and analyze data from digital images and videos.

[0034] Deep learning refers to a type of machine learning method based on or modeled after artificial neural networks. An artificial neural network is a computing system composed of a number of simple, highly interconnected processing elements (nodes) that process information based on their dynamic states in response to external inputs. Large-scale artificial neural networks can have hundreds or thousands of nodes.

[0035] A convolutional neural network (CNN) is a type of neural network frequently applied to analyze visual images, including digital photographs and videos. In a CNN, the connection patterns between nodes are typically modeled after the organization of the human visual cortex, which includes individual neurons arranged to respond to overlapping regions within the field of view. Neural networks suitable for use in the decision process described herein are based on VGG16, VGG19, ResNet50, Inception V3, Xception, or one of other CNN-compatible architectures.

[0036] The classification system engine (204) selects a subset of identified visual tags by applying a classification system-based filter. The candidate generator (206) generates pattern candidates from the selected subset. In one example, each of the pattern candidates includes two or more of the identified visual tags from the subset of identified visual tags. The metric evaluation engine (208) evaluates the consumption metrics of visual media by a first group of users for each of the generated pattern candidates and ranks the generated pattern candidates based on the evaluated consumption metrics to identify emerging viewing patterns.

[0037] FIG. 2b is a block diagram illustrating an example of client applications (114) to be used to identify emerging viewing patterns. These client applications (114) include a viewing metric recorder (212) and a viewing metric database (214). The viewing metric recorder (212) monitors the viewing of visual media on an electronic device and stores one or more parameters associated with each visual media (e.g., how long or how much the visual media has been watched). The viewing metric database (214) stores the recorded viewing metrics (e.g., to be transmitted to a server system (108) for analysis).

[0038] FIG. 3 illustrates examples of components of a server system (108) and a client device (110) for implementing emerging pattern detection. A video server (306) communicates with a metric evaluation engine (208) and a gateway (318) (e.g., a gateway to a full-service network such as the Internet) when scheduling visual media to be served to the client device (110). The video server (306) notifies the metric evaluation engine (208) of the visual media being scheduled. Additionally, the video server (306) can determine the visual media to be served later based on input from the metric evaluation engine (208) once sufficient data is available to detect emerging viewing patterns of the user of the electronic device (110).

[0039] Visual media (e.g., videos (314)) received by the electronic device (110) is routed to a video controller (310) to be displayed on the display of the electronic device (110). The video controller (310) receives commands from the user of the electronic device (110) (via a user interface such as a touchscreen) to control the playback of the visual media. The video controller (310) tracks data associated with playback (e.g., viewing time, total viewing time, number of views, etc.) and stores the data in a viewing metric database (214).

[0040] FIG. 4 is a flowchart illustrating an exemplary method (400) to be executed by a server system (e.g., server system (108)) to detect emerging viewing patterns, FIG. 5 is a flowchart illustrating an exemplary method (500) to be executed by an electronic device (e.g., client device (110)) to collect viewing metrics, FIG. 6 is a flowchart illustrating an exemplary method (600) to be used in the exemplary method of FIG. 4 to generate pattern candidates, and FIG. 7 is a flowchart illustrating an exemplary method (700) to be used in the exemplary method of FIG. 4 to evaluate pattern candidates.

[0041] The following description of the methods refers to a pattern detection system (124) running on a server system (108) and applications (114) on client devices (110), but other systems and devices for identifying emerging patterns will be understood from the description herein. Although a flowchart may describe operations as a sequential process, most of the operations may be performed in parallel or simultaneously. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed. A process may correspond to a method, procedure, etc. Steps of the methods may be performed wholly or partially, and may be performed together with some or all of the steps of other methods, and / or may be performed by any number of different systems, such as the systems described in FIG. 1, FIG. 2a, FIG. 2b, FIG. 3 and FIG. 8-10.

[0042] In block (402), the server system (108) distributes a viewing metric application (114) to user devices. The viewing metric application is configured to run on electronic devices (110), collect viewing metrics for visual media presented on the electronic devices, and return the viewing metrics to the server system (108) for pattern analysis to identify emerging viewing patterns. The visual media may include still images, videos, or a combination thereof.

[0043] In block (404), the server system (108) receives viewing metrics from the electronic device (110). The server system (108) can receive viewing metrics from the client application (114) of the electronic device (110) via the network (102).

[0044] In block (406), the server system (108) identifies visual tags for visual media viewed by users via electronic devices (110). The visual tags identify objects in each piece of visual media served to the client devices (110). For example, a first video or still image may be tagged to include a person, a boy, and clothing objects, and a second video or still image may be tagged to include a person, a girl, an animal, and a dog objects. In one example, the tagged elements for each piece of visual media are determined using an object detection machine learning model, and the determined objects are stored in a database (e.g., database (126)) along with identifiers for the visual media containing these objects.

[0045] In block (408), the server system (108) selects a subset of identified visual tags by applying a classification-based filter (block (408)). The classification-based filter may include a hierarchical tree structure. In one example, level 1 of the hierarchical tree structure is a broad category, e.g., real objects, level 2 is a first subset, e.g., people, clothing, etc., and level 3 is a second subset, e.g., boys among people and shorts among clothing. The server system selects a subset of identified visual tags from identified visual tags located at one or more specific levels in the classification of the classification-based filter. In one example, the server system (108) selects only visual tags from one or more specific levels of abstraction (e.g., levels 2 and 3, because other levels may be too general or too specific for detecting meaningful viewing patterns). According to this example, one or more other levels may be selected according to the classification. In another example, the server system (108) selects only tags from levels having specific characteristics (e.g., leaf nodes, i.e., nodes without subset nodes). Other selection techniques, including combinations of one or more of the provided examples, may be utilized to select visual tags considered within the scope of the invention.

[0046] In block (410), the server system (110) generates pattern candidates. In one example, each of the pattern candidates includes two or more of the identified time tags from a subset of identified time tags. In one example, the server system generates pattern candidates by generating all combinations of identified time tags from a subset of identified time tags having N time tags (block (602)), where N is 2 or more. In one implementation of this example, all generated combinations are used as pattern candidates. In another implementation, one or more combinations are removed from the generated combinations to derive pattern candidates. For example, the system can identify prohibited combinations (e.g., combinations with parent-child relationships, such as shoes and sandals) within the generated combinations within the classification of a classification-based filter (block (604)), and can omit these prohibited combinations from the pattern candidates (block (606)).

[0047] In block (412), the server system (110) evaluates viewing metrics for each of the generated pattern candidates. The server system may evaluate the consumption metrics of visual media by a first group of users (e.g., a group of users with similar characteristics such as an age group, a gender group, etc.) for each of the generated pattern candidates. In one example, the server system (110) receives one or more consumption metrics from each user of the first group of users (block (702)), scores each of the generated pattern candidates in response to the received consumption metrics (block (706); for example, by calculating the ratio of the average viewing time percentage to the associated time length during which the group watched each visual media), and evaluates the consumption metrics by aggregating the scores (block (706); for example, by aggregating the calculated ratio for each pattern candidate).

[0048] For videos (having associated time lengths), consumption metrics may include total viewing time, and scoring may include calculating the ratio of the average time percentage to the associated time length for which the first group watched each of the videos, and for each pattern candidate, aggregating the ratio of the average time percentage to the associated time length for which the first group watched each of the videos. For still images, consumption metrics may include the number of times viewed or shared, and scoring may include calculating the ratio of the total number of unique images viewed by the first group to the total number of times viewed or shared, and for each pattern candidate, aggregating the ratio of the average number of times viewed or shared to the total number of unique images viewed by the first group.

[0049] In one example, a z-score is generated to explain the difference between the consumption metric of a target pattern and the consumption metric of other patterns. According to this example, the consumption metric may be based on a combination of two content consumption metrics for a video (e.g., the VTR (view time ratio) for all videos having a specific pattern and the sum of the total number of records having a specific pattern). The VTR (view time ratio) can be calculated as presented in Equation (1). (1) Here, V is the set of viewers of the video, and size(V) is the number of viewers of the video.

[0050] In block (414), the server system (110) ranks the generated pattern candidates. In one example, the generated pattern candidates are ranked according to their z-scores. According to this example, a higher z-score indicates a pattern that is more different from the mean (normal behavior) and thus indicates an emerging pattern, while a lower z-score indicates a non-emerging pattern.

[0051] In block (416), the server system (110) visually presents the rankings. In one example, the pattern detection system (124) generates a graph for visual representation on the display. According to this example, the pattern detection system (124) selects 10 pattern candidates with the highest z-scores and presents these candidates in order based on their z-scores, along with the objects constituting these candidates.

[0052] In one example, rankings (e.g., based on z-scores) are verified using multi-fold (e.g., 5-fold) cross-validation. According to this example, assuming K-fold cross-validation, the pattern detection system (124) separates the dataset into K folds, each of which contains different groupings of training and validation sets. For each fold, the pattern detection system (124) uses the training set to find the best pattern, for example, based on the highest z-score. Then, the pattern detection system (124) labels visual media in the validation set that have high scores as + and do not have high scores as -. After labeling the entire dataset as + and -, the pattern detection system compares user engagement (viewing time, etc.) of groups + and - to obtain z-scores. This process is repeated for each fold to obtain an intermediate z-score (i.e., an unbiased z-score that becomes the z-score).

[0053] In block (502), an electronic device (e.g., a client device (110)) downloads and installs a viewing metric application (114). In one example, the client device (110) receives the viewing metric application (114) from a server system (108) via the network (102) through the API server (120) of the application server (104).

[0054] In block (504), the client device (110) executes a viewing metric application (114). The processor of the client device (110) (e.g., CPU (830); FIG. 8) executes the viewing metric application (114). In one example, the processor creates a viewing metric database (214) in the memory of the client device (110) to record viewing metrics associated with visual media presented by the client device (110).

[0055] In block (506), the client device (110) records viewing metrics. In one example, the processor of the client device (110) collects viewing metrics (e.g., from the video controller (310)) and records the viewing in memory.

[0056] In block (508), the client device (110) stores viewing metrics. In one example, the client device stores viewing metrics recorded in the viewing metric database (214) of the client device (110)'s memory.

[0057] In block (510), the client device (110) uploads viewing metrics to the server system (108). In one example, the client device (110) retrieves viewing metrics from the viewing metrics database (214) and transmits the viewing metrics to the subsystem (108) via the network (102) through the API server (120) of the application server (104).

[0058] FIG. 8 is a high-level functional block diagram of an exemplary client device (110) implemented as an exemplary mobile device (890) comprising a viewing metric recorder (212) and a viewing metric database (214). The mobile device (890) comprises a flash memory (840A) comprising programming to perform all or a subset of the functions described herein to view the metric recorder (212) and to view the metric database (214). The mobile device (890) may include a camera (870) comprising at least one visible light camera (e.g., first and second visible light cameras having overlapping fields of view or a visible light camera and a depth sensor having substantially overlapping fields of view). The memory (840A) may further include a plurality of images or videos generated through the camera (870).

[0059] As illustrated, the mobile device (890) includes an image display (880), an image display driver (882) that controls the image display (880), and a controller (884). In the example of FIG. 8, the image display (880) and the user input device are integrated together as a touch screen display.

[0060] Examples of touchscreen type mobile devices that may be used include smartphones, PDAs (Personal Digital Assistants), tablet computers, laptop computers, or other portable devices (but are not limited thereto). However, the structure and operation of touchscreen type devices are provided as examples; the subject matter described herein is not intended to be limited thereto. Accordingly, for the purposes of this discussion, FIG. 8 provides block diagram examples of an exemplary mobile device (890) having a touchscreen display for displaying content and receiving user input as a user interface (or as part thereof).

[0061] As illustrated in FIG. 8, the mobile device (890) includes at least one digital transceiver (XCVR) (810), illustrated as WWAN XCVRs, for digital wireless communications over a wide area wireless mobile communication network. The mobile device (890) also includes additional digital or analog transceivers, such as short-range XCVRs (820) for short-range network communications over NFC, VLC, DECT, ZigBee, Bluetooth™, or WiFi. For example, the short-range XCVRs (820) may take the form of any available bidirectional WLAN (wireless local area network) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0062] To generate location coordinates for positioning the mobile device (890), the mobile device (890) may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device (890) may utilize one or both of short-range XCVRs (820) and WWAN XCVRs (810) to generate location coordinates for positioning. For example, cellular network, WiFi, or Bluetooth™-based positioning systems can generate very accurate location coordinates, especially when used in combination. These location coordinates may be transmitted to the eyewear device via one or more network connections through the XCVRs (810, 820).

[0063] Transceivers (810, 820) (network communication interfaces) comply with one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers (810) include, for instance, transceivers configured to operate according to 3GPP (3rd Generation Partnership Project) Type 2 (or 3GPP2) and 3GPP network technologies including LTE, sometimes referred to as “4G”, and CDMA (Code Division Multiple Access) without limitation (but not limited thereto). For instance, transceivers (810, 820) provide bidirectional wireless communication of information including digitized audio signals, still images and video signals, web page information for display as well as web-related inputs, and various types of mobile message communications to and from a mobile device (890).

[0064] The mobile device (890) further comprises a microprocessor described herein as a CPU (830), sometimes referred to as a host controller. A processor is a circuit having elements structured and arranged to perform one or more processing functions, namely, typically various data processing functions. Individual logic components may be used, but examples utilize components that form a programmable CPU. For example, the microprocessor includes one or more integrated circuit (IC) chips that integrate electronic elements to perform the functions of the CPU. For example, the processor (830) may be based on any known or available microprocessor architecture, such as RISC (Reduced Instruction Set Computing), using an ARM architecture, as is commonly used in mobile devices and other portable electronic devices today. Of course, other processor circuits may be used to form the CPU (830) or processor hardware in smartphones, laptop computers, and tablets.

[0065] The microprocessor (830) functions as a programmable host controller for the mobile device (890) by configuring the mobile device (890) to perform various operations according to programming or instructions executable by the processor (830), for example. For example, these operations may include various general operations of the mobile device as well as operations related to programming for the viewing metric recorder (212) and the viewing metric database (214). The processor may be configured using hardwired logic, but the typical processors of mobile devices are general processing circuits configured by the execution of programming.

[0066] A mobile device (890) includes a memory or storage device system for storing data and programming. In this example, the memory system may include flash memory (840A) and random access memory (RAM) (840B). The RAM (840B) functions as a short-term storage unit for instructions and data processed by the processor (830), such as an operation data processing memory. The flash memory (840A) typically provides a long-term storage unit.

[0067] Accordingly, in the example of a mobile device (890), flash memory (840A) is used to store programs or instructions for execution by the processor (830). Depending on the type of device, the mobile device (890) stores and executes a mobile operating system that enables specific applications, such as programming for a viewing metric recorder (212) and a viewing metric database (214), to be executed. Applications, such as the viewing metric recorder (212) and the viewing metric database (214), may be native applications, hybrid applications, or web applications (e.g., dynamic web pages executed by a web browser) executed on the mobile device (890). Examples of mobile operating systems include Google Android, Apple iOS (iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry operating systems, etc.

[0068] FIG. 9 is a schematic representation of a machine (900) on which instructions (908) (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine (900) to perform any one or more of the methodologies discussed herein. For example, instructions (908) can cause the machine (900) to perform any one or more of the methods described herein. Instructions (908) transform an unprogrammed general machine (900) into a specific machine (900) programmed to perform the described and illustrated functions in the described manner. The machine (900) may operate as a standalone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine (900) may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0069] The machine (900) may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart device), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing commands (908) specifying actions to be taken by the machine (900) sequentially or otherwise. Furthermore, although only a single machine (900) is exemplified, the term “machine” should also be interpreted to include a set of machines that execute commands (908) individually or jointly to perform any one or more of the methodologies discussed herein.

[0070] The machine (900) may include processors (902), memory (904), and I / O components (942) that can be configured to communicate with each other via a bus (944). In one example, the processor (902) (e.g., a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a Digital Signal Processor (DSP), an ASIC, a Radio-Frequency Integrated Circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor (906) and a processor (910) that execute instructions (908). The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although FIG. 9 illustrates multiple processors (902), the machine (900) may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

[0071] Memory (904) includes main memory (912), static memory (914), and a storage unit (916), both of which are accessible by the processor (902) via a bus (944). The main memory (904), static memory (914), and storage unit (916) store instructions (908) that implement any one or more of the methodologies or functions described herein. Instructions (908) may also reside in the main memory (912), in the static memory (914), in a machine-readable medium (918) (e.g., a non-transient machine-readable storage medium) in the storage unit (916), in at least one of the processors (902) (e.g., in the processor's cache memory), or in any suitable combination thereof during the execution of instructions by the machine (900).

[0072] Furthermore, the machine-readable medium (918) is non-transient in that it does not implement radio signals (i.e., it does not have any transient signals). However, labeling the machine-readable medium (918) as "non-transient" should not be interpreted to mean that the medium is immobile; the medium should be considered as being movable from one physical location to another. Additionally, because the machine-readable medium (918) is tangible, the medium may be a machine-readable device.

[0073] The I / O components (942) may include a wide variety of components that perform functions such as receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (942) included in a specific machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, whereas headless server machines may not include such touch input devices. It will be recognized that the I / O components (942) may not include many other components not shown in FIG. 9. In various examples, the I / O components (942) may include output components (928) and input components (930). Output components (928) may include visual components (e.g., displays, such as PDP (plasma display panel), LED (light emitting diode) displays, LCD (liquid crystal display), projectors, or CRT (cathode ray tube)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components (930) may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, photo-optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing mechanisms), tactile input components (e.g., physical buttons, touch screens providing the force and location of touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0074] In additional examples, the I / O components (942) may include, among a number of other components, bio-components (932), motion components (934), environment components (936), or position components (938). For example, the bio-components (932) include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and identifying individuals (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification). The motion components (934) include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental components (936) include, for example, light sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or for measuring contaminants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment. Position components (938) include location sensor components (e.g., Global Positioning System (GPS) receiver components), altitude sensor components (e.g., altimeters or barometers for detecting air pressure from which altitude can be induced), orientation sensor components (e.g., magnetometers), etc.

[0075] Communication can be implemented using a wide variety of technologies. The I / O components (942) each further include communication components (940) operable to couple the machine (900) to the network (920) or devices (922) via coupling (924) and coupling (926). For example, the communication components (940) may include a network interface component or other suitable device for interfacing with the network (920). In additional examples, the communication components (940) may include wired communication components, wireless communication components, cellular communication components, NFC (Near Field Communication) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for providing communication via other modes. The devices (922) may be other machines or any of a wide variety of peripheral devices (e.g., peripheral devices coupled via USB).

[0076] Furthermore, the communication components (940) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (940) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional bar codes, such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes, such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). Furthermore, various information, such as a location via Internet Protocol (IP) geolocation, a location via Wi-Fi® signal triangulation, or a location via detecting an NFC beacon signal that can indicate a specific location, can be derived through the communication components (940).

[0077] Various memories (e.g., memory (904), main memory (912), static memory (914), memory of processors (902), storage unit (916)) may store one or more sets of instructions and data structures (e.g., software) that are implemented or used by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions (908)) enable various operations to be implemented when executed by processors (902).

[0078] Commands (908) may be transmitted or received via a network (920), using a transmission medium, via a network interface device (e.g., a network interface component included in the communication components (940)), and using any one of a number of well-known transmission protocols (e.g., HTTP (hypertext transfer protocol)). Similarly, commands (908) may be transmitted to or received from a device (922) via coupling (926) (e.g., peer-to-peer coupling) using a transmission medium.

[0079] FIG. 10 is a block diagram (1000) illustrating a software architecture (1004) that may be installed in any one or more of the devices described herein. The software architecture (1004) is supported by hardware such as a machine (1002) that includes processors (1020), memory (1026), and I / O components (1038). In this example, the software architecture (1004) may be conceptualized as a stack of layers, each layer providing a specific function. The software architecture (1004) includes layers such as an operating system (1012), libraries (1010), frameworks (1008), and applications (1006). Operablely, applications (1006) invoke API calls (1050) through the software stack and receive messages (1052) in response to the API calls (1050).

[0080] The operating system (1012) manages hardware resources and provides common services. The operating system (1012) includes, for example, a kernel (1014), services (1016), and drivers (1022). The kernel (1014) acts as an abstraction layer between the hardware and other software layers. For example, the kernel (1014) provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functions. The services (1016) may provide other common services for other software layers. The drivers (1022) are responsible for controlling the underlying hardware or interfacing with it. For example, drivers (1022) may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB (Universal Serial Bus) drivers), WI-FI® drivers, audio drivers, power management drivers, etc.

[0081] Libraries (1010) provide low-level common infrastructure used by applications (1006). Libraries (1010) may include system libraries (1018) (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Furthermore, the libraries (1010) may include API libraries (1024), such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4 (Moving Picture Experts Group-4), Advanced Video Coding (H.264 or AVC), MP3 (Moving Picture Experts Group Layer-3), AAC (Advanced Audio Coding), AMR (Adaptive Multi-Rate) audio codecs, JPEG or JPG (Joint Photography Experts Group) or PNG (Portable Network Graphics), graphics libraries (e.g., OpenGL frameworks used to render graphics content on a display in two dimensions (2D) and three dimensions (3D)), database libraries (e.g., SQLite for providing various relational database functions), web libraries (e.g., WebKit for providing web browsing functions), etc. The libraries (1010) may also include a wide variety of other libraries (1028) to provide many different APIs to the applications (1006).

[0082] Frameworks (1008) provide high-level common infrastructure used by applications (1006). For example, frameworks (1008) provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Frameworks (1008) may provide a wide range of other APIs that can be used by applications (1006), some of which may be specific to a particular operating system or platform.

[0083] In one example, applications (1006) may include a home application (1036), a contact application (1030), a browser application (1032), a book reader application (1034), a location application (1042), a media application (1044), a messaging application (1046), a game application (1048), and a wide variety of other applications, such as a third-party application (1040). Applications (1006) are programs that execute functions defined in programs. Various programming languages ​​may be used to create one or more of the applications (1006) configured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, a third-party application (1040) (e.g., an application developed by an entity other than a vendor of a specific platform using an ANDROID™ or IOS™ SDK (software development kit)) may be mobile software running on a mobile operating system, such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In such an example, the third-party application (1040) may invoke API calls (1050) provided by the operating system (1012) to facilitate the function described herein.

[0084] It will be understood that the terms and expressions used herein have the general meanings given to them in relation to their respective corresponding areas of inquiry and study, except where specific meanings are otherwise presented herein. Relational terms such as First, Second, etc., may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “to have,” “having,” “to include,” “including,” or any other variation thereof are intended to cover a non-exclusive inclusion so that a process, method, article, or device having or including a list of elements or steps may include not only these elements or steps but also other elements or steps that are not unique to or not explicitly listed in such process, method, article, or device. A singular element does not exclude, without additional restrictions, the presence of additional identical elements in a process, method, article, or device including the element.

[0085] Unless otherwise stated, any and all measurements, values, grades, positions, sizes, dimensions, and other specifications described herein, including the following claims, are not exact but approximate. Such quantities are intended to have a reasonable range consistent with custom in the functions to which they relate and in the relevant technology to which they relate. For example, unless otherwise explicitly stated, parameter values, etc., may vary by ± 10% from the stated quantities, whether quantified in terms of degrees (e.g., approximately, substantially, or about).

[0086] The examples exemplified herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other examples may be used or derived therefrom so that structural and logical substitutions and modifications may be made without departing from the scope of the disclosure. Accordingly, the detailed description should not be taken in a limiting sense, and the scope of the various examples is defined only by such claims, together with the full scope equivalents for which the appended claims have rights.

Claims

Claim 1 A method for identifying emerging viewing patterns for visual media viewed by groups of users, comprising: identifying visual tags for each of a plurality of visual media viewed by a first group among the groups of users; selecting a subset of the identified visual tags by applying a taxonomy-based filter; generating pattern candidates from the selected subset — each of the pattern candidates includes two or more of the identified visual tags from the subset of the identified visual tags —; evaluating consumption metrics of the visual media by the first group of users for each of the generated pattern candidates; and ranking the generated pattern candidates in response to the evaluated consumption metrics to identify emerging viewing patterns for the first group of users. Claim 2 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of identifying the visual tags includes the step of applying a machine learning model to the plurality of visual media. Claim 3 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of selecting a subset of the identified visual tags comprises selecting a subset of the identified visual tags from the identified visual tags located at one or more specific levels in the classification system of the classification system-based filter. Claim 4 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of selecting a subset of the identified visual tags in claim 3 includes the step of selecting a subset of the identified visual tags from the leaf nodes of the classification system. Claim 5 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of generating pattern candidates comprises the step of generating all combinations of the identified visual tags in a subset of the identified visual tags having n visual tags, wherein n is 2 or more. Claim 6 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of generating pattern candidates further comprises: identifying prohibited combinations within the generated combinations having a parent-child relationship within the classification system of the classification system-based filter; and omitting the prohibited combinations from the pattern candidates. Claim 7 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of evaluating the consumption metrics comprises: receiving one or more consumption metrics from each user of a first group of users; and scoring each of the generated pattern candidates in response to the received consumption metrics. Claim 8 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the visual media are videos, each of the videos has an associated time length, and one or more consumption metrics include total viewing time, and the scoring step comprises: calculating the ratio of the average time percentage to the associated time length for which the first group watched each of the videos; and for each pattern candidate, aggregating the ratio of the average time percentage to the associated time length for which the first group watched each of the videos. Claim 9 A method for identifying emerging viewing patterns for visual media viewed by groups of users, further comprising: a step of generating a graph of the ranking of the generated pattern candidates in claim 1; and a step of visually presenting the graph on a display. Claim 10 A method for identifying emerging viewing patterns for visual media viewed by groups of users, further comprising: repeating each of the steps of claim 1 for at least one other group of users; generating a graph of rankings of the generated pattern candidates for each of the first group of users and for at least another group of users; and visually presenting the graph on a display. Claim 11 A method for identifying emerging viewing patterns for visual media viewed by groups of users, wherein the step of evaluating the consumption metrics comprises applying multi-fold cross-validation consumption metrics of the visual media by a first group of users to each of the generated pattern candidates. Claim 12 A system for identifying emerging viewing patterns of visual media viewed by groups of users, comprising: a memory storing instructions; and a processor, wherein the processor comprises instructions for performing: identifying visual tags for each of a plurality of visual media viewed by a first group of the groups of users; selecting a subset of the identified visual tags by applying a classification system-based filter; generating pattern candidates from the selected subset — each of the pattern candidates includes two or more of the identified visual tags from the subset of the identified visual tags —; evaluating consumption metrics of the visual media by the first group of users for each of the generated pattern candidates; and ranking the generated pattern candidates in response to the evaluated consumption metrics to identify emerging viewing patterns for the first group of users. Claim 13 In claim 12, the processor is a system for identifying emerging viewing patterns of visual media viewed by groups of users, wherein the processor applies a machine learning model to the plurality of visual media to identify visual tags. Claim 14 In claim 12, the processor selects a subset of identified visual tags from one or more specific levels in the classification system of the classification system-based filter, thereby identifying an emerging viewing pattern of visual media viewed by groups of users. Claim 15 In claim 12, the processor generates pattern candidates by generating all combinations of the identified visual tags in a subset of the identified visual tags having n visual tags; a system for identifying emerging viewing patterns of visual media viewed by groups of users, wherein n is 2 or more. Claim 16 In claim 15, to generate the pattern candidates, the processor additionally identifies prohibited combinations within the generated combinations having a parent-child relationship within the classification system of the classification system-based filter; and a system for identifying emerging viewing patterns of visual media viewed by groups of users, which omits the prohibited combinations from the pattern candidates. Claim 17 A system for identifying emerging viewing patterns of visual media viewed by groups of users, wherein the processor receives one or more consumption metrics from each user of a first group of users; and evaluates the consumption metrics by scoring each of the generated pattern candidates in response to the received consumption metrics. Claim 18 A system for identifying emerging viewing patterns of visual media viewed by groups of users, wherein, in claim 17, the visual media are videos, each of the videos has an associated time length, and one or more consumption metrics include total viewing time, and the scoring comprises calculating the ratio of the average time percentage to the associated time length for which the first group watched each of the videos; and for each pattern candidate, aggregating the ratio of the average time percentage to the associated time length for which the first group watched each of the videos. Claim 19 A system for identifying emerging viewing patterns of visual media viewed by groups of users, wherein, in claim 12, the system further comprises a display; and the processor is further configured by commands for further performing the operation of generating a graph of the ranking of the generated pattern candidates; and the operation of visually presenting the graph on the display. Claim 20 A non-transient processor-readable storage medium storing processor-executable instructions for identifying emerging viewing patterns of visual media viewed by groups of users, wherein the processor-executable instructions, when executed by a processor of a machine, cause the machine to perform operations, the operations include: identifying visual tags for each of a plurality of visual media viewed by a first group of groups of users; selecting a subset of the identified visual tags by applying a classification-based filter; generating pattern candidates from the selected subset — each of the pattern candidates includes two or more of the identified visual tags from the subset of the identified visual tags —; evaluating consumption metrics of the visual media by the first group of users for each of the generated pattern candidates; and ranking the generated pattern candidates in response to the evaluated consumption metrics to identify emerging viewing patterns for the first group of users.

Images