Enhanced tessellation analysis and quality control for content distribution
Through collaborative filtering and semantic filtering technologies, user interest assessment problems are solved after third-party cookies are abandoned, and the efficient distribution of related content is achieved, improving user experience and privacy protection.
Patent Information
- Application Number
- CN202380018317.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-13
- Publication Date
- 2024-11-15
AI Technical Summary
As third-party cookies are gradually abandoned, it is difficult for existing technologies to effectively evaluate user interests and distribute relevant content without using user identification information, resulting in a decline in user online experience and waste of resources.
Using collaborative filtering and semantic filtering techniques, evaluate user interest segments and assign them to videos or channels. Based on user segment viewing scores, semantic similarity and quality scores, select and distribute digital content related to users to ensure the content Relevance and accuracy.
Without receiving user identification information, the relevance and user experience of content distribution are improved, the calculation and resource waste of irrelevant content are reduced, and user privacy protection is enhanced.
Smart Images

Figure CN118975260A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to data processing and digital content distribution. Background Art
[0002] In systems and devices connected to public networks such as the Internet, data security and user privacy are of paramount importance. The enhancement of user privacy has led many developers to change the way they handle user data. For example, some browsers are planning to deprecate the use of third-party cookies. Summary of the invention
[0003] This document relates to techniques for optimizing content distribution in a manner that protects and enhances user data privacy. The techniques may include using collaborative filtering and semantic filtering to evaluate user interest segments and video content, and assigning user interest segments that meet quality assurance criteria to videos or groups of videos (e.g., channels). The assignments may then be used to distribute content relevant to the user watching the video without using information identifying the user or any other user-specific information. In general, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following operations: identifying, for a video, a set of user segments to which users who have viewed the video are assigned, wherein each user segment has a corresponding topic; for each user segment in the set of user segments, determining a segment viewing score based on the number of users who have viewed the video assigned to the user segment relative to the total number of users who have viewed the video; determining a semantic similarity score representing a measure of semantic similarity between the video and the corresponding topic of the user segment; and determining one or more quality scores based on a level of accuracy corresponding to the assignment of the user segment to the video; selecting one or more user segments of the video from the set of user segments based on the segment viewing score, the semantic similarity score, and the one or more quality scores of each user segment; and distributing a digital component to a client device for display with the video based on the corresponding topic of the selected one or more user segments. Other implementations of this aspect include corresponding devices, systems, and computer programs configured to perform various aspects of the method and encoded on a computer storage device.
[0004] These and other implementations can each optionally include one or more of the following features. In some aspects, the segment viewing score for each user segment is proportional to the number of users assigned to the user segment who viewed the video divided by the total number of users who viewed the video.
[0005] In some aspects, the segment viewing score for each user segment is also based on a segment metric based on the number of users assigned to the user segment relative to the total number of known users.
[0006] In some aspects, the segment viewing score for each user segment is proportional to the number of users assigned to the user segment who viewed the video divided by the product of (i) the total number of users who viewed the video and (ii) the segment metric.
[0007] In some aspects, selecting one or more user segments for a video from the set of user segments includes: identifying a proper subset of the subset of user segments based on the segment viewing score of each user segment; and filtering one or more user segments having semantic scores that do not satisfy a first threshold from the proper subset of user segments.
[0008] In some aspects, selecting one or more user segments for the video from the set of user segments includes filtering one or more additional user segments from the proper subset of user segments having at least one quality score that does not satisfy a corresponding threshold for the at least one quality score.
[0009] In some aspects, a corresponding threshold for each quality score is determined based on a number of user segments whose quality scores meet the threshold.
[0010] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. A combination of several techniques including collaborative filtering, semantic filtering, and quality assurance filtering for assigning user interest segments to videos or channels including multiple videos enables the distribution and display of content relevant to these users without knowing who the users watching the videos are or receiving any identifying information about these users. Providing relevant content enhances the user's online experience while reducing the amount of wasted computing and network resources used to provision irrelevant content that the user would ignore. The combination of techniques provides a synergistic effect of providing such relevant content in a manner that enhances user privacy, such as by not requiring the transmission of user identification information or third-party cookies from the client device to the online platform that selects and distributes content.
[0011] Historically, third-party cookies (e.g., cookies from a domain different from the resource rendered by the client device) have been used to collect data from client devices across the Internet. However, some browsers and device platforms block the use of third-party cookies, and third-party cookies are increasingly being removed from use, thereby preventing the use of third-party cookies to collect data. This presents challenges when attempting to use the collected data to make inferences, segment the data, or otherwise use the data to enhance the online browsing experience (e.g., by selecting content relevant to the user based on data collected using third-party cookies). In other words, without the use of third-party cookies, much of the data previously collected will no longer be available, making it impossible for the computing system to use the data to predict the interests or attributes of the user based on the activities performed by the user at a particular web page or other resource, thereby failing to enhance the user's online experience and / or display relevant content to the user.
[0012] The techniques described herein address barriers that may arise from eliminating third-party cookies. For example, the techniques described in this document evaluate historical data related to video viewers and their assigned user segments to assign user segments to videos so that these assigned segments are used to select relevant content for the user without receiving data identifying the user watching the video. The collaborative filtering, semantic filtering, and quality assurance filtering techniques described ensure that the user segments are of high quality and that relevant content is displayed with the video.
[0013] By pre-assigning user segments to videos, content such as digital components can be selected faster than techniques that analyze user information and / or other signals at the time of request. For example, pre-assigned user segments can be mapped to specific digital components to speed up the selection of digital components for display with the video. For example, delays in providing content such as digital components in response to requests can result in page loading errors at the client device or cause portions of the electronic resource to remain unpopulated even after other portions of the electronic resource are displayed at the client device. Moreover, as the delay in providing the digital component to the client device increases, it is more likely that the electronic resource will no longer be displayed at the client device when the digital component is delivered to the client device, thereby negatively affecting the user's experience of the electronic resource. In addition, for example, if the electronic resource is no longer displayed at the client device when the digital component is provided, the delay in providing the digital component can cause the delivery of the digital component to fail.
[0014] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a block diagram of an example environment in which user segments are assigned to videos and used to distribute digital components for display with the videos.
[0016] Figure 2 is a flow diagram of an example process for assigning user segments to videos.
[0017] Figure 3 is a flow chart of an example process for selecting digital components to display with a video at a client device.
[0018] Figure 4 is a block diagram of an example computer system.
[0019] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0020] In general, this document describes systems and techniques for evaluating user interest segments and video content using collaborative filtering and semantic filtering and assigning user interest segments that meet quality assurance criteria to videos. The assignments can then be used to distribute digital content relevant to the user watching the video.
[0021] Figure 1 1 is a block diagram of an example environment 100 in which user segments are assigned to videos and used to distribute digital components for display with the videos. The environment 100 includes a data communication network 105, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. The data communication network 105 connects the client device 110 with the content platform 130. Although Figure 1 Not shown, the network 105 may also connect the content platform 130 with the video rating system 120 .
[0022] The client device 110 is an electronic device capable of communicating over the network 105. Example client devices 110 include personal computers, server computers, mobile communication devices (e.g., mobile phones and / or tablet computers), and other devices that can send and receive data over the network 105. The client device may also include a digital assistant device that accepts audio input through a microphone and outputs audio output through a speaker. When the digital assistant detects a "hot word" or "hot phrase" that activates the microphone to accept audio input, the digital assistant may be placed in a listening mode (e.g., ready to accept audio input). The digital assistant device may also include a camera and / or a display to capture images and display information visually. The digital assistant may be implemented in different forms of hardware devices, including wearable devices (e.g., watches or glasses), smart phones, speaker devices, tablet devices, or other hardware devices. The client device may also include a digital media device, such as a streaming device that is plugged into a television or other display to stream video to a television, a gaming device, or a virtual reality system.
[0023] A game device is a device that enables a user to participate in a game application, for example, in which a user controls one or more characters, avatars, or other rendered contents displayed in a game application. A game device typically includes a computer processor, a memory device, and a controller interface (physical or visually rendered) that enables a user to control the contents rendered by the game application. The game device can store and execute game applications locally, or execute game applications (e.g., online game applications) that are at least partially stored and / or provided by a cloud server. Similarly, the game device can be docked with a game server that executes the game application and "streams" the game application to the game device. The game device can be a tablet device, a mobile telecommunications device, a computer, or another device that performs other functions in addition to executing the game application.
[0024] The client device 110 may include applications 112, such as a web browser and / or native applications, to facilitate sending and receiving data over the network 105. Native applications are applications developed for a specific platform or a specific device (e.g., a mobile device with a specific operating system). Operations described as being performed by the client device 110 may be performed by the application 112, and operations described as being performed by the application 112 may be performed by another component of the client device 110. The client device 110 may include many different types of applications.
[0025] Application 112 may present, for example, display an electronic resource, such as a web page, application page, or other application content, to a user of client device 110. The electronic resource may include a digital component slot for displaying a digital component with the content of the electronic resource. A digital component slot is an area of an electronic resource (e.g., a web page or application page) for displaying a digital component. A digital component slot may also refer to a portion of an audio stream and / or a video stream (which is another example of an electronic resource) for playing a digital component.
[0026] Electronic resources are also referred to herein as resources. For the purposes of this document, a resource may refer to a web page, an application page, application content displayed by a native application, an electronic document, an audio stream, a video stream, or other appropriate types of electronic resources that can be used to display a digital component.
[0027] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another unit of content). A digital component may be electronically stored in a physical memory device as a single file or in a collection of files, and a digital component may take the form of a video file, an audio file, a multimedia file, an image file, or a text file and include advertising information, such that an advertisement is a type of digital component. For example, a digital component may be content that is intended to supplement the content of a web page or other resource displayed by application 112. More specifically, a digital component may include digital content that is related to resource content (e.g., a digital component may relate to the same subject as the web page content, or relate to a related subject). The provision of a digital component may therefore supplement and generally enhance the web page or application content.
[0028] When application 112 loads a resource including a digital component slot, application 112 may generate a digital component request that requests a digital component for display in the digital component slot. In some implementations, the digital component slot and / or the resource may include code (e.g., a script) that enables application 112 to request a digital component. Application 112 may send the digital component request to content platform 130.
[0029] The digital component request sent by the application 112 may include contextual data. The contextual data may describe the environment in which the selected digital component will be displayed. The contextual data may include, for example, a resource locator of a resource (e.g., a website or native application) that will be used to display the selected digital component, coarse location information indicating the approximate location of the client device 110 that sent the digital component request (e.g., the country or state in which the client device 110 is located), the type of client device 110 (e.g., a laptop, a smart phone, a gaming device, etc.), the spoken language setting of the application 112 or the client device 110, the number of digital component slots in which the digital component will be displayed with the resource, the type of the digital component slot, and other appropriate contextual information. The resource locator may be in the form of a uniform resource locator (URL), a uniform resource identifier (URI), a network address, a domain name, or other appropriate resource locator.
[0030] Some applications 112 may include a video player for streaming or otherwise playing a video requested from a content platform 130 or another video source. When an application 112 sends a digital component request to have the digital component displayed with a video, such as in a break in the video or near the video player, the application 112 may include data identifying the video, such as in the context data. The data identifying the video may include a unique identifier for the video, a title for the video, a resource locator of a resource from which the video is streamed or downloaded, and / or other suitable information for identifying the video.
[0031] The digital component request may not include data that can be used to identify the user. For example, the digital component request may not include cookies (e.g., third-party cookies), user identifiers, IP addresses, or other identifying information.
[0032] Content platform 130, which may be implemented as one or more computers in one or more locations, is configured to distribute digital components to client devices 110, for example, in response to digital component requests received from client devices 110. Content platform 130 includes digital component selection system 134, which may be implemented as one or more computers in one or more locations.
[0033] Digital component selection system 134 is configured to select digital components for display at client device 110, for example, based on context data included in the digital component request. For example, digital component selection system 134 may select one or more digital components for display with a video based on the video itself that the user is watching. In a specific example, digital component selection system 134 may select digital components based on the subject matter of the video, the subject matter of the channel that includes the video, and / or other appropriate information.
[0034] The digital component selection system 134 may also be configured to select digital components for a video based on a user segment assigned to the video and / or a user segment assigned to a channel (or another video group) that includes the video. Generally speaking, a user segment may include a group or group of users that have been determined to be interested in a particular topic. Each user segment may have a corresponding topic, and the users assigned to the user segment may be users that have been determined to be interested in the topic. Example topics may include topics of interest (e.g., cats, books, travel, etc.) or an image that represents a user's long-term interests, such as an outdoor enthusiast, a sports fan, a gardener, etc.
[0035] As described in more detail below, the video rating system 120 is configured to assign user segments to videos and / or channels. The digital component selection system 134 can use the assignments to select digital components for display with the video. In some implementations, digital component providers who wish to have their digital components displayed with videos related to a particular topic or to users interested in a particular topic can assign topics to digital components as distribution criteria.
[0036] In general, the distribution criteria of a digital component defines the circumstances under which the digital component is eligible or ineligible for display at a user's client device 110. The distribution criteria of a digital component may include topics for which the digital component is eligible for display and / or topics for which the digital component is ineligible for display. If the digital component request includes data identifying a video that has been assigned a user segment with one of the eligible topics, then the digital component is eligible for selection and display with the video. If the digital component request includes data identifying a video that has been assigned a user segment with one of the ineligible topics, then the digital component is ineligible for selection or display with the video.
[0037] For example, the distribution criteria for a digital component having content related to a Hawaii event may include "Hawaii" and "travel to Hawaii" as eligible topics. If a digital component request causes the digital component to be displayed with a video that has been assigned to a user segment with the topic "Hawaii," the digital component for the Hawaii event will be eligible for selection and distribution to the client device 110 from which the digital component request was received.
[0038] The digital component selection system 134 may use other information to select the digital component, such as other contextual data included in the digital component request (and corresponding distribution criteria for the digital component) and selection parameters for the digital component. The selection parameters may specify an amount that a digital component provider is willing to provide to a publisher (e.g., a publisher of a video) in exchange for displaying the digital component with the video.
[0039] The video evaluation system 120 evaluates user segments and videos and assigns user segments and their corresponding topics to videos based on the evaluation. The video evaluation system 120 can be implemented as one or more computers in one or more locations. In general, the video evaluation system 120 includes several evaluation stages to identify user segments with the following topics for each video: (i) topics that users who have watched the video are interested in, (ii) topics that are semantically related to the video content, and (iii) topics that have high quality of relevance to the video. The video evaluation system 120 includes a collaborative filtering engine 122, a semantic evaluation engine 124, and a quality evaluation engine 126 corresponding to these stages.
[0040] The collaborative filtering engine 122 is configured to identify a set of user segments for a video based on videos viewed by the user and user segments assigned to the users who viewed the video. The collaborative filtering engine 122 may identify user segments for a video based on user data stored in the user data storage unit 123 and video data stored in the video data storage unit 125. The data storage units 123 and 125 may include a database, a table, or other appropriate data structures for storing data. Each user segment may have a unique segment identifier (ID), and each video may have a unique video ID.
[0041] For each user in the user set, the user data may include data identifying videos viewed by the user and the user segments to which the user is assigned. In general, users may be assigned to user segments based on a user profile (which may include their identified interests), the subject matter of videos viewed by the user, another online activity (e.g., a query submitted to a video platform), and / or other appropriate data. For each video in the video set, the video data may include data identifying the video title, the publisher of the video, the channel that includes the video, one or more subjects of the video content, and / or any user segments that have been assigned to the video.
[0042] The user data storage unit 123 may include user data of users who have opted in to such data collection and are logged into a publisher's website or application, for example, a user logged into a video platform that streams or otherwise provides videos to the client device 110 for display to the user of the client device 110. For example, a publisher may store user data for users who have consented to the collection and use of their data. The users whose user data is collected and stored may be a subset of the total number of unique users who may request videos from the publisher and / or may receive digital component requests from them.
[0043] In addition to the description throughout this document, controls (e.g., user interface elements with which the user can interact) may be provided to the user to allow the user to make choices about whether and when the systems, programs, or features described herein may allow the collection of user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location in which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.
[0044] The collaborative filtering engine 122 may use collaborative filtering techniques to identify a set of user segments for each video. Generally speaking, these collaborative filtering techniques include identifying a user segment for a video based on the user segments of users who have previously viewed the video. For example, if many users who watch a video about racing cars have been assigned to the "car enthusiast" user segment, the collaborative filtering engine 122 may assign the "car enthusiast" user segment to the video.
[0045] The collaborative filtering engine 122 may aggregate the user data and the video data and perform collaborative filtering techniques using the aggregated data. For each video, the collaborative filtering engine 122 may determine a segmented viewing score for each user segment or at least some of the user segments. The segmented viewing scores for videos and user segments may be based on the number of unique users assigned to the user segment who viewed the video relative to the total number of unique users who viewed the video. For example, the segmented viewing scores for videos and user segments may be calculated using equation (1):
[0046] In equation (1), Count (video ID, segment ID) represents the count of the number of unique users assigned to a user segment with a specific segment ID who have watched a video with a specific video ID, and Count (video ID) represents the count of the total number of unique users who have watched a video with a specific video ID. Therefore, in this example, the segment viewing score is equal to the number of unique users assigned to the user segment who have watched the video divided by the total number of unique users who have watched the video. The quotient can be multiplied by another factor to obtain the segment viewing score, so that the segment viewing score is proportional to the number of unique users assigned to the user segment who have watched the video divided by the total number of unique users who have watched the video. When a greater number of users who have watched a video are assigned to a user segment, these example segment viewing scores of the user segment have higher values. The segment viewing scores of the user segment and a given video calculated in this way are normalized scores among all user segments to capture the conditional probability of cross-keys (e.g., video ID, segment ID) given a primary key (video ID).
[0047] In some implementations, the segmented viewing score for a video can be based on a segmentation metric based on the number of users assigned to a user segment relative to the total number of known users, e.g., a ratio between the two numbers. The total number of known users can include all users who can request a video, including users who have not opted in to data collection. The segmented viewing score can be calculated using equation (2):
[0048] In Equation 2, Count(Segment ID) represents the count of the number of unique users assigned to the user segment with a particular segment ID, and Total Count represents the total number of known users.
[0049] The segmented viewing score for a user segment and a video may be calculated based on the segmented metric for the user segment using Equation 3:
[0050] In Equation 3, Count(Video ID, Segment ID) represents a count of the number of unique users assigned to a user segment with a specific segment ID who have viewed a video with a specific video ID, Count(Video ID) represents a count of the total number of unique users who have viewed a video with a specific video ID, and Segment Metric(Segment ID) represents a segment metric for the user segment with a specific segment ID. This way of calculating a segment viewing score introduces a ratio of the number of users in a user segment to the total number of known users.
[0051] Similar equations can be used to calculate similar scores for video channels. For example, the segmented viewing scores for user segments and video channels can be calculated using Equation 4 or Equation 5:
[0052] In Equations 4 and 5, Count (channel ID, segment ID) represents a count of the number of unique users assigned to a user segment with a specific segment identifier (ID) who have viewed a channel with a specific channel ID, and Count (channel ID) represents a count of the total number of unique users who have viewed a channel with a specific channel ID. The segment metric of Equation 5 for a user segment with a specific segment ID may be calculated using Equation 2.
[0053] To select a user segment for a video, the collaborative filtering engine 122 may determine the segment viewing score for each user segment and the video, or a combination of the two scores (e.g., the product, sum, or average of the two scores) using Equation 1 or Equation 3. One or both of Equation 1 and Equation 3 may be used depending on the application and / or available data. Equation 1 measures the count by segment ID for user segments and videos, while Equation 3 measures the lift of the user segment of the video relative to all video traffic, and may be more accurate in some applications.
[0054] The collaborative filtering engine 122 may then use the segment viewing scores of the user segments to select user segments for the video. For example, the collaborative filtering engine 122 may select each user segment whose segment viewing score meets a threshold score or a specified number of user segments with the highest segment viewing scores. For example, the collaborative filtering engine 122 may select user segments in order from highest to lowest until a specified maximum number of user segments for the video is reached.
[0055] The collaborative filtering engine 122 may use a similar process to select user segments for a channel. For example, the collaborative filtering engine 122 may determine the segment viewing scores for each user segment and channel using Equation 4 or Equation 5. The collaborative filtering engine 122 may select each user segment whose segment viewing score meets a threshold score or a specified number of user segments having the highest segment viewing scores.
[0056] In some implementations, the collaborative filtering engine 122 may apply some user privacy enhancement conditions to the user segments being assigned to videos or channels. For example, the collaborative filtering engine 122 may apply a k-anonymity condition to each user segment to ensure that at least a minimum number of users are assigned to the user segment before the user segment is assigned to a video. Applying this k-anonymity condition ensures that the user data of the users assigned to the user segment will not be leaked or disclosed by the video rating system 120.
[0057] The collaborative filtering engine 122 may provide the user segment set for each video and / or the user segment set for each channel to the semantic evaluation engine 124. In general, the semantic evaluation engine 124 evaluates the semantic relevance between each user segment in the user segment set and the video or channel, and filters out user segments that do not have sufficient semantic relevance to the video or channel from the user segment set.
[0058] In some implementations, user segments can be built hierarchically using broad parent segments and more specific sub-segments. For example, the highest level segment may be "Media and Entertainment," with some sub-segments being "Music Lovers," "Movie Lovers," and "Book Lovers." Sub-segments can also have their own sub-segments. For example, the segment "Music Lovers" may have sub-segments "Country Music Lovers" and "Pop Music Lovers."
[0059] The semantic evaluation engine 124 may execute segment aggregation logic that assigns higher-level segments to videos to which their child or grandchild segments have been assigned. For example, if a video is assigned a "pop music lover" user segment by the collaborative filtering engine 122, the semantic evaluation engine 124 may also assign "music lover" and "media and entertainment" user segments to the video.
[0060] The semantic filtering engine 124 may apply a semantic filter to the user segment set of a video channel to ensure a sufficient level of semantic relevance between the corresponding topic of each user segment and the content of the video or channel. In some implementations, the semantic filtering engine 124 determines the semantic similarity score of each user segment in the user segment set of a video or channel. The semantic similarity score represents the similarity between the topic of the user segment and the content of the video or channel. Then, the semantic filtering engine 124 may filter any user segment whose semantic similarity score does not meet the threshold from the user segment set. For example, the semantic filtering engine 124 may compare each semantic similarity score with a threshold score. If the semantic similarity score of the user segment is less than the threshold score, the semantic filtering engine 124 may remove the user segment from the user segment. After filtering, the user segment set of a video or channel may only include user segments with semantic similarity scores that meet (e.g., reach or exceed) the threshold score.
[0061] The semantic similarity scores of the video or channel and the user segment can be determined based on the theme or category assigned to the video or channel and the theme or category assigned to each user segment. The semantic evaluation engine 124 can calculate the semantic similarity score based on the cosine similarity between the theme or category assigned to the video and the theme or category assigned to the user segment. In other words, the semantic similarity score of the video or channel and the user segment can be based on the similarity between the theme or category assigned to the video or channel and the theme or category assigned to the user segment. For example, the more matching (or similar) themes shared by the video or channel and the user segment, the higher the semantic similarity score of the video or channel and the user segment.
[0062] In some implementations, the semantic evaluation engine 124 (or another component of the video evaluation system 120) trains a semantic relevance model based on the co-occurrence data of the videos. For example, if a user who watched one of the videos also watched the other video, the two videos may be considered semantically similar. The semantic evaluation engine 124 may train a machine learning model (e.g., a neural network, a regression model, a decision tree, etc.) based on the user data and video data stored in the data storage units 123 and 125. The semantic relevance model may be trained to output semantic similarity scores for videos or channels and user segments.
[0063] The semantic evaluation engine 124 may provide the subset of user segments remaining after semantic filtering to the quality evaluation engine 126. The quality evaluation engine 126 may evaluate the quality of the user segments and remove any user segments of insufficient quality from the subset of user segments remaining after semantic filtering.
[0064] The quality assessment engine 126 may determine one or more quality scores for each user segment with respect to the video or channel. One example quality score is an accuracy score. The accuracy metric may be based on the user segment assigned to the user viewing the video. The accuracy metric may be determined based on user feedback, which may take the form of a survey provided to a set of users. For example, a survey for a user may request the user to identify whether the user is interested in a topic corresponding to the user segment. In a specific example, the survey may ask, "Are you a food lover?" for the food lover user segment.
[0065] The precision score for a user segment and a video may be based on the number of true positive responses for the user segment and the videos assigned to the user segment and the total number of false positive survey responses for the user segment and the video. For example, the precision score for a user segment and a video may be based on the ratio between the number of true positive responses for the user segment and the videos assigned to the user segment and the total number of false positive survey responses for the user segment and the video. A true positive response for a user segment and a video is a view of the video by a surveyed user that indicates in the survey results that the user is a member of the user segment. A false positive response for a user segment and a video is a view of the video by a surveyed user that indicates in the survey that the user is not a member of the user segment.
[0066] For example, the accuracy score can be calculated using equation (6):
[0067] Another example quality score for a user segment and a video is a combined score based on a precision score and a lift score. The lift score for a user segment and a video can be based on the precision score for the user segment and the video and a prior score for the user segment that is independent of the video. The prior score can be based on the number of positive survey responses, the number of negative survey responses, and the number of videos viewed by users with positive survey responses and the number of videos viewed by users with negative survey responses. For example, the prior score for a user segment can be calculated using equation (7):
[0068] In equation (7), PSRs represents the number of positive survey responses for the user segment during the specified time period (e.g., indicating that the user is a member of the user segment or the number of users interested in the topic corresponding to the user segment), Views1 represents the number of videos viewed by users with positive survey responses (e.g., indicating that the user is a member of the user segment or the number of users interested in the topic corresponding to the user segment) during the specified time period, Views2 represents the number of videos viewed by users with negative survey responses (e.g., indicating that the user is not a member of the user segment or the number of users not interested in the topic corresponding to the user segment) during the specified time period, and NSRs represents the number of negative survey responses for the user segment during the specified time period (e.g., indicating that the user is not a member of the user segment).
[0069] For example, consider the user segment "Food Lovers". Assume there are 5 users who are food lovers, each of whom has watched 7 videos, and 2 users who are not food lovers, each of whom has watched 2 videos. In this example, the PSR is 5 (users who are food lovers), the NSR is 2 (users who are not food lovers), View1 is 35 (a total of 35 videos watched by 5 food lovers), and View2 is 4 (a total of 4 videos watched by 2 non-food lovers). Therefore, the prior score in this example is (5*35) / ((5*35) + (2*4)) = 0.0956.
[0070] The prior score represents the overall distribution of users in the user segment as to whether they have seen the video. The prior score combined with the precision score provides a deeper understanding of the relevance of the user segment to the video than using the precision score alone. For example, combining the two scores can indicate the lift of users of the user segment to watch the video, which is called the lift score. The lift score indicates how much more likely users of the user segment are to watch the video compared to all users.
[0071] The lift score can be calculated using equation (8):
[0072] In equation (8), precision represents the precision score for the user segment and the video (e.g., calculated using equation (6)), and prior represents the prior score for the user segment (e.g., calculated using equation 7). In this example, a lift score of 1.0 indicates a neutral reaction of the users of the user segment to the video, because a lift score of 1.0 indicates that the rate at which the users watch the video matches the distribution of users who generally watch videos. A lift score greater than 1.0 indicates that the users of the user segment are more likely to watch the video than the overall viewing rate of all users, and the higher the lift score, the more likely it is that the users in the user segment are likely to watch the video. A lift score less than 1.0 indicates that the users in the user segment are less likely to watch the video than all users, and the lower the lift score, the less likely it is that the users in the user segment are likely to watch the video.
[0073] The quality assessment engine 126 may calculate the combined score as the product of the precision score and the lift score. The quality assessment engine 126 may compare each quality score (e.g., the precision score and / or the combined score) of the user segment to a corresponding threshold. In some implementations, a user segment must pass two quality checks to be assigned to a video. For example, for a user segment to be assigned to a video, the precision score may have to meet (e.g., meet or exceed) a first threshold, and the combined score may have to meet (e.g., meet or exceed) a second threshold. In another example, a user segment may only have to pass one of the quality checks, or only one of the quality checks may be used to check quality.
[0074] The quality assessment engine 126 may filter each user segment that did not pass the quality check from the subset of user segments received from the semantic assessment engine 124. The quality assessment engine 126 may generate a final set of user segments 127 for the video or channel. The final set of user segments 127 may include user segments that were included in the original set by the collaborative filtering engine 122 and were not filtered by the semantic assessment engine 124 or the quality assessment engine 126. In other words, the final set of user segments 127 are user segments that passed both the semantic relevance assessment and the quality assessment.
[0075] Video rating system 120 may send end-user segments 127 for each video and / or channel to content platform 130. Digital component selection system 134 may use the end-user segments assigned to a video or channel to distribute digital components to client devices 110 for display with the video or channel, as described herein.
[0076] The video assessment system 120 may be configured to fine-tune the thresholds for selecting and / or filtering user segments for videos or channels. For example, the video assessment system 120 may be configured to calibrate the threshold scores so that at least a specified number of user segments may meet the threshold and be assigned to each video.
[0077] In another example, the video evaluation system 120 may be configured to fine-tune a threshold for determining whether an unknown user is a member of a user segment based on the video being watched by the unknown user. For example, the digital component selection system 134 may be configured to determine the likelihood that the user of the client device 110 from which the digital component request is received is a member of each of the one or more user segments. The digital component selection system 134 may determine this likelihood based on the user segment assigned to the video, for example using a trained machine learning model that is trained to output a likelihood score based on the user segment assigned to the video and the context signal of the digital component request (e.g., a rough geographic location, device type, etc.). When a digital component request related to video viewing is received by the digital component selection system 134, the digital component selection system 134 may provide the user segment assigned to the video (and / or its corresponding topic) and / or the context signal of the digital component request as input to the machine learning model, and receive the likelihood score of each of the one or more user segments as an output of the machine learning model. The digital component selection system 134 may then compare the likelihood score of each user segment with the corresponding threshold for the user segment. If the likelihood score satisfies a threshold (eg, meets or exceeds a threshold), the user may be considered a member of the user segment. The video assessment system 120 may be configured to adjust these thresholds for the user segment to improve the quality of the prediction results.
[0078] Figure 2 is a flow chart of an example process 200 for assigning user segments to videos. The operations of process 200 may be performed by a video evaluation system such as Figure 1 The operations of process 200 may also be implemented as instructions stored on one or more computer-readable media that may be non-transitory, and execution of the instructions by one or more data processing devices may cause the one or more data processing devices to perform the operations of process 200. Although example process 200 is described in terms of assigning user segments to videos, a similar process may be used to assign user segments to channels or other video groups.
[0079] Identifying User Segments of the Video (210). As described above, the video assessment system 120 can use collaborative filtering to identify user segments of the video.
[0080] Determine a segment viewing score for each user segment (220). The collaborative filtering engine 122 can determine the segment viewing scores for the user segment and the video based on, for example, the number of users assigned to the user segment who viewed the video relative to the total number of users who viewed the video. For example, the collaborative filtering engine 122 can determine the segment viewing score for each user segment in the set of user segments using Equation 1 or Equation 3 above.
[0081] A semantic similarity score for each user segment is determined (230). The semantic similarity score for the user segment relative to the video may represent a measure of semantic similarity between the video and the corresponding topic of the user segment.
[0082] One or more quality scores are determined for each user segment (240).For example, quality assessment engine 126 may determine the accuracy score and / or the combined score for each user segment as the quality score using Equations 6-8 above.
[0083] One or more user segments for the video are selected based on the scores (250). In some implementations, the video assessment system 120 can compare each score of each user segment to a threshold. The video assessment system 120 can then select each user segment whose score meets (e.g., meets or exceeds) its threshold.
[0084] In another example, the video assessment system 120 can combine the scores (eg, by determining the product of the scores or normalizing the scores and determining a sum or central tendency measure of the normalized scores for each user segment) and select the user segment with the highest combined score.
[0085] In another example, the video assessment system 120 may select a set of user segments based on the segment viewing scores of the user segments. For example, the video assessment system 120 may select a specified number of user segments with the highest segment viewing scores or each user segment with a segment viewing score that satisfies a threshold score. The video assessment system 120 may then filter each user segment from the set of user segments whose semantic similarity score or quality score fails to meet the corresponding threshold. The video assessment system 120 may then assign to the video a proper subset (e.g., at least one but less than the entire set) of the user segments remaining after filtering.
[0086] Based on the themes assigned to the user segments of the video, the digital components are distributed to the client devices for display at the client devices (260). For example, the digital components may be displayed using Figure 3 The process 300 distributes the digital components to client devices.
[0087] Figure 3 is a flow chart of an example process 300 for selecting digital components to be displayed with a video at a client device. The operations of process 300 may be performed by a digital component selection system such as Figure 1 The digital component selection system 130 of the process 300 is performed. The operations of the process 300 may also be implemented as instructions stored on one or more computer-readable media that may be non-transitory, and execution of the instructions by one or more data processing devices may cause the one or more data processing devices to perform the operations of the process 300.
[0088] A digital component request is received (310). For example, digital component selection system 134 may receive a digital component request from client device 110. The digital component request may identify a video that application 112 running on client device 110 is displaying or is about to display.
[0089] The user segments assigned to the video are identified (320). As described above, a set of user segments for a video may be selected and assigned to the video by the video assessment system 120. The digital component selection system 134 may identify the user segments and their corresponding topics.
[0090] A set of digital components is identified based on themes corresponding to the user segments assigned to the video (330). As described above, the digital components may have distribution criteria that identify themes for which the digital components are eligible and / or themes for which the digital components are not eligible. The digital component selection system 134 may compare the themes of the user segments assigned to the video with the themes of the distribution criteria of the digital components to identify a set of eligible digital components that are eligible to be provided to the client device 110 in response to the digital component request.
[0091] A digital component is selected from the set of digital components (340). The digital component selection system 134 may select the digital component based on selection parameters of each qualifying digital component, a predicted performance measure of each digital component, and / or other appropriate information.
[0092] The selected digital component is provided to the client device 110 from which the digital component request was received (350). For example, the digital component selection system 134 may send the digital component or a resource locator of the digital component to the client device 110. Where appropriate, the client device 110 may use the resource locator to download the digital component from a network computer for display to a user of the client device 110.
[0093] Figure 4 4 is a block diagram of an example computer system 400 that can be used to perform the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In some implementations, processor 410 is a single-threaded processor. In another implementation, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored in memory 420 or on storage device 430.
[0094] Memory 420 stores information within system 400. In one implementation, memory 420 is a computer readable medium. In some implementations, memory 420 is a volatile memory unit. In another implementation, memory 420 is a non-volatile memory unit.
[0095] The storage device 430 can provide mass storage for the system 400. In some implementations, the storage device 430 is a computer-readable medium. In various implementations, the storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.
[0096] The input / output device 440 provides input / output operations for the system 400. In some implementations, the input / output device 440 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., and an 802.11 card). In another implementation, the input / output device may include a driver device configured to receive input data and send output data to an external device 460 (e.g., a keyboard, a printer, and a display device). However, other implementations may also be used, such as a mobile computing device, a mobile communication device, a set-top TV client device, etc.
[0097] Despite Figure 4 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuit systems or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.
[0098] The subject matter and the embodiments of the operations described in this specification may be implemented in a digital electronic circuit system or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium (or multiple computer storage media) for data processing equipment to execute or control the operation of data processing equipment. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for data processing equipment to execute. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or be included in them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).
[0099] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
[0100] The term "data processing device" covers all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality of the foregoing or a combination of the foregoing. The device may include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The device and the execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.
[0101] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.
[0102] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may also be implemented as, a special purpose logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
[0103] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors. Generally speaking, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally speaking, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or operatively coupled to receive data from the one or more mass storage devices or transfer data to the one or more mass storage devices or both. However, a computer does not have to have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
[0104] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same one or more computers.
[0105] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.
[0106] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
[0107] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs that run on respective computers and have a client-server relationship to each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server.
[0108] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that are unique to specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve changes in sub-combinations or sub-combinations.
[0109] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0110] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions set forth in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.
Claims
1. A computer-implemented method, include: identifying, for a video, a set of user segments to which users who have viewed the video are assigned, wherein each user segment has a corresponding topic; For each user segment in the user segment set, determining a segment viewing score based on a number of users assigned to the user segment who viewed the video relative to a total number of users who viewed the video; determining a semantic similarity score, the semantic similarity score representing a measure of semantic similarity between the video and the corresponding topic of the user segment; as well as determining one or more quality scores, the one or more quality scores being based on a level of accuracy corresponding to assigning the user segment to the video; selecting one or more user segments for the video from the set of user segments based on the segment viewing score, the semantic similarity score, and the one or more quality scores for each user segment; as well as A digital component is distributed to a client device for display with the video based on the corresponding theme of the selected one or more user segments.
2. The computer-implemented method of claim 1, wherein the segment viewing score for each user segment is proportional to the number of users assigned to the user segment who viewed the video divided by the total number of users who viewed the video.
3. The computer-implemented method of claim 1 or 2, wherein the segment viewing score for each user segment is further based on a segment metric based on the number of users assigned to the user segment relative to a total number of known users.
4. A computer-implemented method as described in claim 3, wherein the segment viewing score for each user segment is proportional to the number of users assigned to the user segment who viewed the video divided by the product of: (i) the total number of users who viewed the video and (ii) the segment metric.
5. The computer-implemented method of any preceding claim, wherein one or more user segments of the video are selected from the set of user segments. include: identifying a proper subset of the subset of user segments based on the segment viewing score of each user segment; as well as One or more user segments having semantic scores that do not satisfy a first threshold are filtered from the proper subset of user segments.
6. The computer-implemented method of claim 5, wherein one or more user segments of the video are selected from the set of user segments. include: One or more additional user segments having at least one quality score that does not satisfy a corresponding threshold for at least one quality score are filtered from the proper subset of user segments.
7. The computer-implemented method of any preceding claim, further comprising determining the respective threshold for each quality score based on a number of user segments whose quality scores satisfy the threshold.
8. A system, include: one or more processors; and One or more storage devices storing instructions which, when executed by the one or more processors, cause the one or more processors to perform a method as claimed in any preceding claim.
9. A computer readable medium carrying instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 7.
10. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.
