Model training system and method based on feature fusion of multiple data types

Abstract

Systems, methods, and computer-readable storage media for training a model based on combined common features of two or more different data types. A method includes: receiving a plurality of first data elements of a first data type and a plurality of second data elements of a second data type; identifying first features of each of the plurality of first data elements; identifying second features of each of the plurality of second data elements; generating combined features by combining a first feature of a first feature of each of the plurality of first data elements with a second feature of a second feature of one of the plurality of second data elements, wherein the first feature and the second feature each represent a common feature; and training a model based on the combined features and at least a portion of the first and second features.

Description

Background Art

[0001] The present disclosure generally relates to model training based on data of multiple modalities. Some content items can include multiple pieces of content, such as images and text. However, images and text belong to different data modalities. A model for classifying content items of multiple data modalities can be trained based on data from the data modalities.

[0002] Some training methods for training classification models can assume that data points across multiple modalities are directly linked (e.g., captions of videos or clinical annotations of laboratory reports) to utilize zero-shot learning. Alternatively, certain training methods can co-train multiple types of content data (e.g., images, videos, HTML5, application data, etc.) in the same embedding space. These training methods require the existence of a large amount of labeled multimedia data (e.g., image and / or video data), or multimedia implicitly labeled by proximity to other content types. It may not be possible to obtain such a large amount of data, especially in the case of image-based content, which may require a large number of users to view and manually classify. In addition, video-based content may take a large amount of user inspection time to classify, and thus increase the cost. Summary of the Invention

[0003] Generally, an innovative aspect of the subject matter described in this specification can be implemented in a method including the following: receiving, by one or more processing circuits, a plurality of first data elements of a first data type and a plurality of second data elements of a second data type, where the first data type is text data and the second data type is at least one of image data or video data. The method includes identifying, by one or more processing circuits, a first feature of each of the plurality of first data elements, and identifying, by one or more processing circuits, a second feature of each of the plurality of second data elements. The method includes generating, by one or more processing circuits, a combined feature by combining a first feature of a first feature of each of the plurality of first data elements with a second feature of a second feature of one of the plurality of second data elements, where the first feature and the second feature each represent a common feature; training, by one or more processing circuits, a model based on the common feature and at least a portion of the first feature and the second feature, and classifying a content item based on the model, where the content item includes at least one of content text and content image or content video. In some embodiments, the content item includes image data features fused with video data features (rather than text features, or in addition to text features).

[0004] In general, another aspect of the subject matter described in this specification can be found in a system that includes one or more storage devices configured to store instructions thereon that, when executed by one or more processors, cause the one or more processors to: receive a plurality of first data elements of a first data type and a plurality of second data elements of a second data type, identify a first feature of each of the plurality of first data elements, and identify a second feature of each of the plurality of second data elements. The instructions cause the one or more processors to generate a combined feature by combining the first feature of each of the plurality of first data elements with the second feature of one of the plurality of second data elements, where the first feature and the second feature each represent a common feature and a training model based on the common feature and at least a portion of the first feature and the second feature.

[0005] In general, another aspect of the subject matter described in this specification can be implemented in one or more computer-readable storage media configured to store instructions thereon that, when executed by one or more processors, cause the one or more processors to: receive a plurality of first data elements of a first data type and a plurality of second data elements of a second data type, identify a first feature of each of the plurality of first data elements, and identify a second feature of each of the plurality of second data elements. The instructions cause the one or more processors to generate a combined feature by combining the first feature of each of the plurality of first data elements with the second feature of one of the plurality of second data elements, where the first feature and the second feature each represent a common feature and a training model based on the common feature and at least a portion of the first feature and the second feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0007] Figure 1 is a block diagram of an analysis system including a model manager and a merge function according to an illustrative implementation, the model manager and the merge function implementing feature merging of multiple data types for model training.

[0008] Figure 2 is shown in more detail according to an illustrative implementation Figure 1 of the model manager.

[0009] Figure 3 is a block diagram of the merge function according to an illustrative implementation showing the generation of training data for training a model Figure 1 of.

[0010] Figure 4 is a block diagram of a merging function showing the generation of combined features for inferring classification using a model trained as shown Figure 3 as shown. Figure 1

[0011] Figure 5 is a flowchart of a process for training a model with combined features generated by the merging function of Figure 1 as described in an illustrative embodiment. Figure 1

[0012] Figure 6 is a flowchart of a process for generating a model output using the model of Figure 1 as described in an illustrative embodiment, where the merging function provides the combined features to the model.

[0013] Figure 7 is a block diagram of a computing system as described in an illustrative embodiment. DETAILED DESCRIPTION

[0014] In general, with reference to the accompanying drawings, various illustrative systems and methods are provided that can be used for a machine learning (ML) classifier for classifying content items of multiple content types including, for example, images, text, audio, video, HTML5 data, application data, etc. For example, a content item may include a text description of a product as well as a video or image of the product. Additionally, the present disclosure more particularly relates to training a classifier when there is more labeled training data for one content type than for another content type, e.g., when there is more labeled text data than image data and / or video data. Although there may be a large amount of text-based training data available, it may be difficult to obtain image and / or video training data because this may require a person to manually view the images and / or videos.

[0015] In this regard, when there is not a large amount of training data for another content type (e.g., an image or a video), the labeled data of one content type (e.g., text) can be utilized to develop a classifier for content items including multiple content types. The systems described herein are configured to quickly establish a classifier for content items including multiple content types without collecting a large amount of manually labeled training data for each content type.

[0016] ​​The present system and method utilize a common feature space formed by data from different domains (alternatively referred to as data types or modalities). The system can be configured to incorporate features from different domains in the early stages of training. The system can use the combined features to train a joint model for classifying content items that include multiple content types. Features can be extracted from multiple content types and combined to form a common feature space. This combination can be referred to as "early fusion" because the features are combined early in the machine learning training pipeline. The system can further utilize only text and only image features, where the uncombined features are unique to text or image respectively. The classifier trained with early fusion outperforms models trained based on only text data or only image data.

[0017] The system can be configured to train a classifier model by obtaining data from both images and text to build the classifier model. The system can be configured to extract common features from different content types. These common features can be plain text (e.g., overlaid text on an image, text derived from speech recognition on a video, etc.), categorical features, or numerical features calculated using a machine learning model. Figure 1-6 The training and inference of a classifier model trained with early fusion are shown.

[0018] The system can be configured to receive content items from a content provider and can be configured to apply a trained classifier trained with early fusion to classify the content items. In some embodiments, the output of the classifier model can determine whether the content item should be served to a user, withheld from the user, or restricted on which users or websites the content item is available.

[0019] The data used to train the classifier model can be content item data for serving end users, and a model for extracting features from the content item data can be trained based on search data. The corpus size of the training model for extracting features from content items can be one or more orders of magnitude larger than the total number of content items used to train the classifier model. Although the content item corpus is smaller, it can be important because the content item corpus belongs to the domain for which the classifier is trained, that is, they are real content items. Additionally, these content items can have classification labels (e.g., policy labels) assigned by human reviewers. The models shown can only be trained on image and text content. Features fed to the policy classification model can be extracted by a model trained on search data. Early fusion is useful for any task that requires classifying data from multiple domains with a shared (or partially shared) feature set.

[0020] Now refer to Figure 1, A block diagram of an analysis system 120 and a related environment 100 is shown according to an illustrative embodiment. One or more user devices 104 can be used by a user to perform various actions and / or access various types of content, some of which can be provided via a network 102 (e.g., the Internet, LAN, WAN, etc.). As used herein, a "user" or "entity" can refer to an individual operating user device 104, interacting with resources or content items via user device 104, etc. User device 104 can be used to access websites (e.g., using an Internet browser), media files, and / or any other type of content. A content management system 108 can be configured to select content to be displayed to a user within a resource (e.g., a web page, an application, etc.) and provide content items to user device 104 via network 102 for display within the resource. The content from which content management system 108 selects items can be provided by one or more content providers using one or more content provider devices 106 via network 102.

[0021] In some embodiments, content management system 108 can select content items from content providers for display on user device 104. In such embodiments, content management system 108 can determine the content to be published in the content interface of one or more resources (e.g., web pages, applications, etc.). Content management system 108 can be configured to conduct a content auction among third-party content providers to determine which third-party content will be provided to user device 104. The auction winner can be determined based on the bid amount and a quality score (i.e., a measure of the likelihood that a user of user device 104 will click on the content). In some embodiments, content management system 108 allows content providers to create content campaigns. A campaign can include any number of parameters, such as minimum and maximum bid amounts, a target bid amount, and / or one or more budget amounts (e.g., a daily budget, a weekly budget, a total budget, etc.).

[0022] Analysis system 120 can include one or more processors (e.g., any general or dedicated processor) and can include and / or be operatively coupled to one or more temporary and / or non-temporary storage media and / or storage devices (e.g., any computer-readable storage medium, such as magnetic storage, optical storage, flash memory, RAM, etc.). In various embodiments, analysis system 120 and content management system 108 can be implemented as separate systems or integrated within a single system (e.g., content management system 108 can be configured to incorporate some or all of the functions / capabilities of analysis system 120).

[0023] The analytics system 120 can be communicatively and operably coupled to the analytics database 126. The analytics system 120 can be configured to query information in the analytics database 126 and store information in the analytics database 126. In various embodiments, the analytics database 126 includes various temporary and / or non-temporary storage media. The storage media can include, but are not limited to, magnetic storage, optical storage, flash memory, RAM, etc. The analytics database 126 and / or the analytics system 120 can use various APIs to perform database functions (i.e., manage the data stored in the database 126). These APIs can be, but are not limited to, SQL, ODBC, JDBC, etc.

[0024] The analytics system 120 can be configured to communicate with any device or system shown in the environment 100 via the network 102. The analytics system 120 can be configured to receive information from the network 102. The information can include browsing history, cookie logs, television advertising data, print publication advertising data, radio advertising data, and / or online advertising campaign data. The analytics system 120 can be configured to receive and / or collect interactions that the user device 104 has on the network 102.

[0025] The analytics system 120 can be configured to send information and / or notifications related to various metrics or models that it determines, generates, or that are suitable for the content provider device 106. This can allow a user of one of the content provider devices 106 to review the various metrics or models determined by the analytics system 120. Additionally, the analytics system 120 can use various metrics to identify an appropriate time to contact a user or an appropriate amount to spend on various media channels (e.g., television advertising, internet advertising, radio advertising, etc.) (e.g., optimal mixed media spend). The analytics system 120 can cause a message to be sent to the content management system 108 and / or the content provider device 106 that indicates that the content management system 108 should contact a particular user and / or a content campaign with certain parameters at a specific time. This can cause the content management system 108 to manage content auctions and / or identify various system loads accordingly.

[0026] The analysis system 120 may include one or more modules (i.e., computer-readable instructions executable by a processor) and / or circuitry (i.e., ASIC, processor-memory combination, logic circuitry, etc.) configured to perform the various functions of the analysis system 120. In some embodiments, the module may be or include a model manager 122 and a merging function 124. The model manager 122 may be configured to train a model 128 stored in the analysis database 126 based on training data 132. The training data 132 includes data of a first data type 134 and a second data type 136. The first data type 134 and the second data type 136 may each be one of image data, text data, video data, audio data, etc. In some embodiments, the first data type 134 and the second data type 136 are separate data types. For example, the first data type 134 is image data and the second data type 136 is text data. In some embodiments, the first data type 134 and the second data type 136 are the same data type, but are received from separate data sources. For example, an image received from a product manufacturer and a second image received from a website selling the product.

[0027] In some embodiments, the model manager 122 is configured to perform early fusion. The model manager 122 may be configured to merge features of multiple data modalities to create a single common feature space for training the model 128. Features shared by multiple data modalities may be merged into a single feature. The data may be data from the raw text of a text post, captions derived from image data points or video, audio data, and / or any other data. Additionally, features specific to certain data modalities may be applied to the model 128 rather than being combined (e.g., image-specific embeddings will not appear in text data). In some embodiments, the data modalities and the label source may be jointly trained.

[0028] The model manager 122 may be configured to apply a feature extraction model 130 to the training data 132 to extract features of the first data type 134 and the second data type 136. The merging function may be configured to identify common features between the first data type 134 and the second data type 136 and merge the common features to generate merged features. The model manager 122 may be configured to apply the merged features and / or unique features of the first data type 134 and the second data type 136 to the training data to train the model 128.

[0029] Using the trained model 128, the model manager 122 can be configured to classify the content item 138. The content item 138 includes different data types, data of a first data type 140 and a second data type 142. The first data type 140 and the second data type 142 can each be one of image data, text data, video data, audio data, etc. The model manager 122 can be configured to apply the feature extraction model 130 to the content item 138 to extract features from the first data type 140 and the second data type 142.

[0030] The content item 138 can be the same as or similar to the content item 112. In some embodiments, the content provider device 106 and / or the user device 104 can provide the content item 138 to the analysis system 120 for serving the user. However, before the content item 138 can be added to the content database 110 for providing to the user, the analysis system 120 can determine a policy classification for the content item 138. The policy classification can identify restrictive information for the content item 138. The restrictive information can identify what types of users the content item 138 should serve, what types of web pages the content item 138 should serve, and / or any other restrictive information.

[0031] The merging function 124 can be configured to merge the common features extracted by the feature extraction model 130 to generate merged features. The model manager 122 can apply the unique features of the first data type 140, the unique features of the second data type 142, and the merged features to the model 128 to generate a classification of the content item 138. The result of applying the content item 138 to the model 128 can be a classification of the content item 138. The policy classification can be an indication of the rules for serving the content item 138. The rules can indicate certain types of users or web pages that can be served the content item 138.

[0032] In some embodiments, the model 128 is a neural network. The neural network can be a recurrent neural network, a convolutional neural network, a long short-term memory neural network, a gated recurrent unit neural network, an autoencoder neural network, a variational autoencoder neural network, and / or any other type or combination of neural network types. In some embodiments, the model 128 is a non-linear support vector machine, a random forest, a gradient boosting tree, a decision tree, a Bayesian network, a hidden Markov model, and / or any other type of model.

[0033] The feature extraction model 130 can be various types of models that extract features from the training data 132 and / or the content item 138. For example, the feature extraction model 130 can be a speech recognition model, an image embedding model, a video embedding model, an object recognition model, optical character recognition (OCR), and / or any other type of feature recognition model. The model manager 122 can train the feature extraction model 130 on the training data 132.

[0034] Now referring to Figure 2 , the model manager 122 of Figure 1 is shown in more detail according to the illustrative embodiment. The model manager 122 receives the training data 132. The model manager 122 can be configured to train the model 128 based on the training data 132. In addition, the model manager 122 receives the content item 138. The model manager 122 can cause the model 128 to generate a classification 218 for the content item 138, where the model 128 is trained based on the training data 132.

[0035] The model manager 122 includes a feature extraction manager 214. The feature extraction manager 214 can be configured to extract features of the first data element 202, the second data element 204, the first data type 140, and / or the second data type 142. In some embodiments, the feature extraction manager 214 can identify common features between the first data element 202 and the second data element 204. In some embodiments, the feature extraction manager 214 can identify common features between the first data type 140 and the second data type 142.

[0036] In addition, the feature extraction manager 214 can extract unique features from the training data 132 and / or the content item 138. Unique features can be features that appear only in one of the first data element 202 or the second data element 204 or the first data type 140 or the second data type 142. In this regard, features that appear only in one data type without the other can be applied to the model trainer 216 to train the model 128 or alternatively applied to the model 128 for inferring the classification 218.

[0037] The feature extraction manager 214 can be configured to compare the features extracted from the first data element 202 and the second data element 204 to determine whether any of the features represent the same feature, i.e., whether they are common features. For example, the feature extraction manager 214 can extract text features representing a golf club in a content item of a golf device. The extraction manager 214 can extract image features from an image of the content item. For example, the content item can include various images of golf devices, such as golf clubs, golf balls, and golf gloves. The feature extraction manager 214 can identify image features of golf clubs, golf balls, and / or golf gloves. The feature extraction manager 214 can be configured to identify common features, i.e., image-based features of a golf club and text features of a golf club, and provide the common features to the merging function 124 to generate merged features.

[0038] The feature extraction manager 214 can be configured to train a feature extraction model (e.g., the feature extraction model 130) that extracts features from the training data 132 and / or the content item 138. For example, the feature extraction manager 214 can include an object feature extraction model, such as a convolutional neural network that extracts features from images, an audio processing model that identifies audio features, an optical character recognition model that extracts characters from images, a text processing feature that identifies text features from text data, etc. The model can be trained by the feature extraction manager 214 based on the feature extraction training data 212.

[0039] The merging function 124 can be configured to merge the common features identified by the feature extraction manager 214. The merging function 124 can combine the common features. For example, each feature can have a metric or other value associated with the feature. For example, for a golf club feature, the metric can indicate the likelihood that the identified golf club is a golf club. The merging function 124 can combine the metrics associated with each common feature by applying a mathematical operation. For example, the operation can include summation, subtraction, multiplication, averaging, determining the median, etc. The output of the merging function can be the metric output by applying the mathematical operation.

[0040] The model manager 122 includes a model trainer 216. The model trainer 216 can be configured to train the model 128 based on the merged features received from the merging function 124 and / or the unique features extracted by the feature extraction manager 214. The model trainer 216 can be configured to perform gradient descent, conjugate gradient, Newton's method, quasi-Newton, Levenberg-Marquardt, etc. In some embodiments, the training data 132 includes a policy classification for each of the first data element 202 and / or the second data element 204. Based on the classification and the first data element 202 and the second data element 204, the model trainer 216 can train the model 128.

[0041] With the trained model 128, the model manager 122 can apply the content item 138 to the model 128 to generate a classification 218. The feature extraction manager 214 can be configured to extract common features from the first data type 140 and the second data type 142. Additionally, the merging function 124 can merge the common features and apply the merged features to the model 128 to infer a policy classification 218. Additionally, the feature extraction manager 214 can extract unique features from the first data type 140 and the second data type 142 and apply the unique features to the model 128 to generate a policy classification 218.

[0042] Now referring Figure 3 , a system 300 for the merging function is shown according to an illustrative embodiment, which generates training data for training a model. The system 300 includes an image-based content item 302 and a text-based content item 304. The model manager 122 can generate training data 306 and train the model 128 based on the training data 306. The model manager 122 can be configured to extract image-only features 308 from the image-based content item 302 and text-only features 314 from the text-based content item 304. The image-only features 308 can be features that appear only in the image-based content item 302 and not in the text-based content item 304.

[0043] For example, the image-only features 308 can be the colors used in the image-based content item 302, certain shapes or objects that appear in the image-based content item 302, and / or indications of any other information that does not appear in the text-based content item 302. The text-only features 314 can be indications of product prices, the number of products sold together, product usage instructions, or other product details that are not indicated in the image-based content item 302. Additionally, the model manager 122 can extract common features 310 and 312 from the image-based content item 302 and the text-based content item 304.

[0044] The merging function 124 can merge the common features 310 and 312 to generate a merged feature 316. The merged feature 316 can be included in the training data 306 to train the model 128. The merged feature 316 can be generated from the common features 310 and 312. For example, for the common features of the image-based content item 302 and the common features of the text-based content item 304, the merging function 124 can apply a mathematical operation to the values associated with each content item. The values can be the confidence or probability of the features identified in the data. For example, if a golf club feature is associated with a probability of 0.9 from an image-based content item and the golf club feature is associated with a probability of 0.5 from a text-based content item, the merging function 124 can apply a mathematical operation to the probabilities to generate a merged function for the golf club. For example, the merging function 124 can average the two probability values to generate a probability of 0.7.

[0045] Based on the training data 306 formed by only the image features 308, the merged features 316, and only the text features 314, the model manager 122 can be configured to train the model 128. More specifically, the model trainer 216 can train the model 128 based on the training data 306. By training the model 128 as Figure 3 shown, inferences can be determined as Figure 4 shown.

[0046] Now referring to Figure 4 , according to an illustrative embodiment, the system 400 includes a merging function 124 that generates merged features for inferring classifications with a model trained as Figure 3 shown. The content item 402 is applied to the model 128. The content item 402 can be the same as or similar to the content item 138. The content item 402 includes image-based information 404 and text-based information 406. In some embodiments, the image-based information 404 includes one or more images or video information of a product. In some embodiments, the text-based information 406 includes a text description of the product. The only image features 408, the common features 410, the common features 412, and / or only the text features 414 can be similar to the only image features 308, the common features 310, the common features 312, and only the text features 314.

[0047] Feature extraction manager 214 can be configured to extract image-only features 408, common features 410, common features 412, and / or text-only features 414. Common features 410 and 412 can be applied to merge function 124. Merge function 124 can combine common features 410 and 412 to generate merged features 416. Image-only features 408, merged features 416, and text-only features 414 can be applied to model 128 to generate classification 418. Classification 418 can be a classification of content item 402. Classification 418 can be a strategy for providing content item 402 to a user.

[0048] Now refer to Figure 5 , which is according to an illustrative embodiment by Figure 1 1. In some embodiments, the process 500 is performed by the model manager 122. In some embodiments, the process 500 is performed by the computer system 700. In some embodiments, any computing system as described herein can be configured to perform the process 500.

[0049] In step 502, the model manager 122 receives a first data element of a first data type and a second data element of a second data type. In some embodiments, the first data element and the second data element are training data 132. The first data type and the second data type can each be a different data type of at least one of image data, video data, text data, audio data, etc.

[0050] In step 504, the model manager 122 identifies a first feature for each of the first data elements. In some embodiments, the model manager 122 applies the feature extraction manager 214 to the first data elements of step 502 to generate common features and unique features for the first data elements. In step 506, the model manager 122 identifies a second feature for each of the second data elements. In some embodiments, the model manager 122 applies the feature extraction manager 214 to the second data elements of step 502 to generate common features and unique features for the second data elements.

[0051] In step 508, the model manager 122 generates a merged feature by combining the first feature of the first feature of the first data element with the second feature of the second feature of the second data element. The first feature and the second feature may represent a common feature. The merge function 124 may merge the common features to generate the merged feature. In some embodiments, the merge function 124 may apply a mathematical operation, such as addition, averaging, median, subtraction, etc., on the common features to generate the merged feature.

[0052] In step 510, the model manager 122 trains a model based on the combined features and unique features of the first data element and the second data element. For example, the model manager 122 may train the model 128 based on the training data 306. In some embodiments, the model trainer 216 is configured to perform the training.

[0053] Now refer to Figure 6 , which is a block diagram of a flowchart of a process for generating a model output using a model according to an illustrative embodiment, where a merging function provides combined features to the model. In some embodiments, the process 600 is performed by the model manager 122. In some embodiments, the process 600 is performed by the computer system 700. In some embodiments, any computing system as described herein may be configured to perform the process 600.

[0054] In step 602, the model manager 122 receives a data element that includes a first data element of a first data type and a second data element of a second data type. The data element may be a content item 402 that includes image-based information 404 and text-based information 406.

[0055] In step 604, the model manager 122 extracts one or more first features of the first data element and one or more second features of the second data element. The first features and the second features may be common features and may be extracted by the feature extraction manager 213. In step 608, the model manager 122 may generate one or more combined features by combining one or more first features with one or more second features. In some embodiments, the merging function 124 may receive one or more first features and one or more second features and merge the features to generate combined features. For example, in some embodiments, the model manager 122 may apply the merging function 124 to merge common features to generate combined features.

[0056] In step 608, the model manager 122 may extract one or more unique first features of the first data element and one or more unique second features of the second data element. In some embodiments, the unique features each appear in one of the first data element and the second data element and do not appear in the other data element and are not common. In step 610, the model manager 122 may generate a model output by inputting the one or more combined features determined in step 606 and the unique features of step 608. In some embodiments, the model manager 122 applies the common features and the unique features to the model 128 to generate the policy classification 218.

[0057] Now refer to Figure 7, shows a computer system 700, which can be used, for example, to implement the exemplary user equipment 104, the exemplary content management system 108, the exemplary content provider equipment 106, the exemplary analysis system 150, and / or various other exemplary systems described in the present disclosure. The computing system 700 includes a bus 705 or other communication components for communicating information, and a processor 710 coupled to the bus 705 for processing information. The computing system 700 also includes a main memory 715, such as a random access memory (RAM) or other dynamic storage device, which is coupled to the bus 705 for storing information and instructions to be executed by the processor 710. The main memory 715 can also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 710. The computing system 700 may also include a read-only memory (ROM) 720 or other static storage device coupled to the bus 705 for storing static information and instructions for the processor 710. A storage device 725, such as a solid-state device, a magnetic disk, or an optical disk, is coupled to the bus 705 for persistently storing information and instructions.

[0058] The computing system 700 can be coupled to a display 735, such as a liquid crystal display or an active matrix display, via the bus 705 to display information to a user. An input device 730, such as a keyboard including alphanumeric and other keys, can be coupled to the bus 705 for communicating information and command selections to the processor 710. In another embodiment, the input device 730 has a touchscreen display 735. The input device 730 can include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 710 and controlling the movement of a cursor on the display 735.

[0059] In some embodiments, the computing system 700 can include a communication adapter 740, such as a network adapter. The communication adapter 740 can be coupled to the bus 705 and can be configured to enable communication with a computing or communication network 745 and / or other computing systems. In various illustrative embodiments, any type of network configuration can be implemented using the communication adapter 740, such as wired (e.g., via Ethernet), wireless (e.g., via WiFi, Bluetooth, etc.), pre-configured, ad-hoc, LAN, WAN, etc.

[0060] In accordance with various embodiments, in response to a processor 710 executing an arrangement of instructions contained in a main memory 715, a computing system 700 may implement a process for completing the illustrative embodiments described herein. These instructions may be read from another computer-readable medium, such as a storage device 725, into the main memory 715. Execution of the arrangement of instructions contained in the main memory 715 causes the computing system 700 to perform the illustrative processes described herein. One or more processors in a multiprocessing arrangement may also be employed to execute the instructions contained in the main memory 715. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the exemplary embodiments. Accordingly, embodiments are not limited to any specific combination of hardware circuitry and software.

[0061] Although example processing systems are described in Figure 7 , embodiments of the subject matter and functional operations described in this specification may be performed using other types of digital electronic circuitry or in computer software, firmware, or hardware, or in combinations of one or more of them that include the structures disclosed in this specification and their equivalents.

[0062] Embodiments of the subject matter and operations described in this specification may be performed using digital electronic circuitry or in computer software, firmware, or hardware, or in combinations of one or more of them that include the structures disclosed in this specification and their equivalents embodied on a tangible medium. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus. A computer-readable storage medium may be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more of their combinations. Moreover, although a computer storage medium is not a propagated signal, a computer storage medium may be the source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium may also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices) or be included in one or more separate components or media. Thus, a computer storage medium is tangible and non-transitory.

[0063] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0064] The term "data processing apparatus" or "computing device" encompasses a variety of devices, equipment, and machines for processing data, e.g., including programmable processors, computers, system-on-chips, or multiple or combinations thereof as above. The apparatus may include dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program being discussed, e.g., code that constitutes processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or a combination of one or more of them. The apparatus and the execution environment may implement various different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0065] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts in a markup language document), in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0066] The processes and logical flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processing and logical flows can also be performed by dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the apparatus can also be implemented as dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit).

[0067] By way of example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions in accordance with instructions and one or more storage devices for storing instructions and data. Generally, a computer will also include or be operatively coupled to receive data from and transfer data to one or more mass storage devices for storing data (e.g., magnetic, magneto-optical or optical disks); however, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including by way of example semiconductor storage devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0068] In order to provide interaction with a user, a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and an indicating device, such as a mouse or a trackball, by which the user can provide input to the computer may be used to perform embodiments of the subject matter described in this specification. Other kinds of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback or tactile feedback; and input from the user may be received in any form, including acoustic, speech or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on a user client device in response to a request received by the web browser.

[0069] Embodiments of the subject matter described in this specification may be implemented using a computing system that includes, for example, backend components such as a data server, or middleware components such as an application server, or frontend components such as a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, such as, for example, a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (such as the Internet), and peer-to-peer networks (such as ad hoc peer-to-peer networks).

[0070] The computing system may include a client and a server. The client and the server are typically remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a client device (e.g., for the purpose of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the result of a user interaction) may be received at the server from the client device.

[0071] In some illustrative embodiments, the features disclosed herein may be implemented on a smart TV module (or connected TV module, hybrid TV module, etc.), which may include processing circuitry configured to integrate an Internet connection with more traditional TV program sources (e.g., via cable, satellite, over-the-air, or other signal reception). The smart TV module may be physically incorporated into a television set or may include a separate device, such as a set-top box, Blu-ray or other digital media player, gaming console, hotel TV system, and other companion devices. The smart TV module may be configured to allow a viewer to search for and locate videos, movies, photos, and other content on the Internet, on local cable TV channels, on satellite TV channels, or stored on a local hard drive. A set-top box (STB) or set-top unit (STU) may include an information application device, which may include a tuner and be connected to a television set and an external signal source, convert the signal into content, and then display it on a TV screen or other display device. The smart TV module may be configured to provide a home screen or top-level screen that includes icons for a plurality of different applications, such as a web browser and a plurality of streaming services (e.g., Netflix, Vudu, Hulu, etc.), connected cable or satellite media sources, other Internet “channels,” etc. The smart TV module may further be configured to provide an electronic program guide to a user. Companion applications for the smart TV module may run on a mobile computing device to provide additional information to the user about available programs, to allow the user to control the smart TV module, etc. In alternative embodiments, the features may be implemented on a laptop computer or other personal computer, smartphone, other mobile phone, palmtop computer, tablet computer, or other computing device.

[0072] Although this specification contains many specific implementation details, these details should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that are specific to particular embodiments of a particular invention. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination or in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Moreover, although the features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination may be excised from the combination, and the claimed combination may cover a sub-combination or a variation of a sub-combination. Alternatively, features described with respect to a particular heading may be used in relation to and / or in combination with illustrative embodiments described under other headings. Headings, if provided, are included only for readability purposes and should not be construed as limiting any functionality provided under such headings.

[0073] Similarly, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products embodied on a tangible medium.

[0074] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the particular order shown or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for classifying content items, comprising: Receiving, by one or more processing circuits, a plurality of first data elements of a first data type and a plurality of second data elements of a second data type; Identifying, by the one or more processing circuits, a plurality of first features of each of the plurality of first data elements; Identifying, by the one or more processing circuits, a plurality of second features of each of the plurality of second data elements; Generating, by the one or more processing circuits, a combined feature by combining a first feature among the plurality of first features of each of the plurality of first data elements with a second feature among the plurality of second features of one of the plurality of second data elements, wherein the first feature and the second feature each represent a common feature of the plurality of first data elements and the plurality of second data elements; Training, by the one or more processing circuits, a model based on the combined feature and at least a portion of the plurality of first features and the plurality of second features, wherein the portion of the plurality of first features and the plurality of second features includes features unique to the first data type of the plurality of first data elements and features unique to the second data type of the plurality of second data elements; and Classifying a content item based on the model, wherein the content item includes at least one of content text and a content image or a content video.

2. The method according to claim 1, wherein Each of the plurality of first data elements is associated with one of the plurality of second data elements; Wherein generating the combined feature by the one or more processing circuits includes: combining a first feature among the plurality of first features of each of the plurality of first data elements with a second feature among the plurality of second features of one of the plurality of second data elements associated with each of the plurality of first data elements.

3. The method according to claim 1, wherein, Identifying the plurality of first features and the plurality of second features by the one or more processing circuits includes: applying one or more models to the plurality of first data elements and the plurality of second data elements, wherein the one or more models extract the plurality of first features from the plurality of first data elements and extract the plurality of second features from the plurality of second data elements.

4. The method according to claim 3, wherein The one or more models include at least one of an image embedding model, a video embedding model, an object recognition model, an audio translation model, and an optical character recognition model.

5. The method according to claim 1, wherein Combining the first feature with the second feature includes: performing an operation on a first value representing a first confidence of the first feature and a second value representing a second confidence of the second feature.

6. The method according to claim 5, wherein, The operation is at least one of the following: A maximum operation of selecting the maximum value of the first value and the second value; A summation operation of adding the first value and the second value; A median operation of determining the median of the first value and the second value; and A minimum operation of selecting the minimum value of the first value and the second value.

7. The method according to claim 1, further comprising: Receiving, via the one or more processing circuits, data elements including a first data element of the first data type and a second data element of the second data type; Extracting, via the one or more processing circuits, a first inferred feature of the first data element and a second inferred feature of the second data element; Generating, via the one or more processing circuits, one or more combined features by combining one or more of the first inferred features with one or more of the second inferred features, wherein each of the one or more of the first inferred features is a specific common feature of one of the one or more of the second inferred features; Identifying, via the one or more processing circuits, a unique first classification feature of the first inferred features unique to the first data type; Identifying, via the one or more processing circuits, a unique second classification feature of the second inferred features unique to the second data type; And Generating, via the one or more processing circuits, a model output of the model by applying the one or more combined features, the unique first classification feature, and the unique second classification feature as inputs to the model.

8. The method according to claim 1, wherein The first data type is a text-based data type, and the second data type is at least one of an image data type or a video data type.

9. The method according to claim 8, wherein At least a first portion of the plurality of first data elements is associated with a first data element label, and at least a second portion of the plurality of second data elements is associated with a second data element label, wherein a first quantity of the first data element labels is greater than a second quantity of the second data element labels; wherein the model is further trained by the one or more processing circuits based on the first data element labels and the second data element labels.

10. The method according to claim 8, wherein, At least a first portion of the plurality of first data elements is associated with a first data element label, and none of the plurality of second data elements is associated with a second data element label; wherein the model is further trained by the one or more processing circuits based on the first data element labels.

11. A system including one or more memory devices configured to store instructions thereon that, when executed by one or more processors, cause the one or more processors to: Receive a plurality of first data elements of a first data type and a plurality of second data elements of a second data type; Identify a plurality of first features of each of the plurality of first data elements; Identify a plurality of second features of each of the plurality of second data elements; Generate a combined feature by combining a first feature of the plurality of first features of each of the plurality of first data elements with a second feature of the plurality of second features of one of the plurality of second data elements, wherein the first feature and the second feature each represent a common feature of the plurality of first data elements and the plurality of second data elements; and Train a model based on the combined features and at least a portion of the plurality of first features and the plurality of second features, where the portion of the plurality of first features and the plurality of second features includes features unique to the first data type of the plurality of first data elements and features unique to the second data type of the plurality of second data elements.

12. The system according to claim 11, wherein, Each of the plurality of first data elements is associated with one of the plurality of second data elements; Wherein the instructions cause the one or more processors to generate the combined features includes: combining a first feature of the plurality of first features of each of the plurality of first data elements with a second feature of the plurality of second features of one of the plurality of second data elements associated with each of the plurality of first data elements.

13. The system according to claim 11, wherein, The instructions cause the one or more processors to identify the plurality of first features and the plurality of second features includes: applying one or more models to the plurality of first data elements and the plurality of second data elements, where the one or more models extract the plurality of first features from the plurality of first data elements and extract the plurality of second features from the plurality of second data elements.

14. The system according to claim 11, wherein Combining the first feature and the second feature includes: performing an operation on a first value representing a first confidence level of the first feature and a second value representing a second confidence level of the second feature.

15. The system according to claim 11, wherein, The instructions cause the one or more processors to: Receive a data element including a first data element of the first data type and a second data element of the second data type; Extract a first inferred feature of the first data element and a second inferred feature of the second data element; Generate one or more combined features by combining one or more of the first inferred features with one or more of the second inferred features, where each of the one or more of the first inferred features is a specific common feature of one of the one or more of the second inferred features; Identify a unique first classification feature of the first inferred features unique to the first data type; Identify a unique second classification feature of the second inferred features unique to the second data type; And Generate a model output of the model by applying the one or more combined features, the unique first classification feature, and the unique second classification feature as inputs to the model.

16. The system according to claim 15, wherein, The data element is a content item including multiple content types, where the first data element is text data and the second data element is at least one of image data or video data.

17. The system according to claim 11, wherein The first data type is a text-based data type, and the second data type is at least one of an image data type or a video data type.

18. The system according to claim 17, wherein, At least a first portion of the plurality of first data elements is associated with a first data element label, and at least a second portion of the plurality of second data elements is associated with a second data element label, Wherein, a first quantity of the first data element labels is greater than a second quantity of the second data element labels; wherein the instruction causes the one or more processors to further train the model based on the first data element tag and the second data element tag.

19. The system according to claim 17, wherein, At least a first portion of the plurality of first data elements is associated with a first data element tag, and none of the plurality of second data elements is associated with a second data element tag; wherein the instruction causes the one or more processors to further train the model based on the first data element tag.

20. One or more computer-readable storage media configured to store instructions thereon, the instructions, when executed by one or more processors, cause the one or more processors to: Receive a plurality of first data elements of a first data type and a plurality of second data elements of a second data type; Identify a plurality of first features of each of the plurality of first data elements; Identify a plurality of second features of each of the plurality of second data elements; Generate combined features by combining a first feature among the plurality of first features of each of the plurality of first data elements with a second feature among the plurality of second features of one of the plurality of second data elements, wherein the first feature and the second feature each represent a common feature of the plurality of first data elements and the plurality of second data elements; and Train a model based on the combined features and at least a portion of the plurality of first features and the plurality of second features, wherein the portion of the plurality of first features and the plurality of second features includes features unique to the first data type of the plurality of first data elements and features unique to the second data type of the plurality of second data elements.

Images