Product identification in media items

Abstract

A method includes obtaining first data including a first identifier of a first product determined based on first metadata of a content item being associated with the content item. The method further includes obtaining a first confidence value associated with the first product and the content item. The method further includes obtaining second data including a second identifier of the first product and a second confidence value. The method further includes providing the first data and the second data to a trained machine learning model. The method further includes obtaining, from the trained machine learning model, a third confidence value associated with the first product. The method further includes adjusting second metadata of the content item in view of the third confidence value.

Description

Technical Field

[0001] This disclosure relates to aspects and implementations of methods and systems for facilitating the pairing of media items and associated objects, and more specifically to systems for identifying pairings of media items and products. Background Technology

[0002] A platform (e.g., a content sharing platform) can transmit (e.g., stream) media items to client devices connected to the platform via a network. Different types of client devices can be optimized for different tasks, preferred by users for different tasks, etc. Media items may contain references to one or more products. Summary of the Invention

[0003] The following summary is a simplified summary of this disclosure in order to provide a basic understanding of some aspects of this disclosure. This summary is not a comprehensive overview of this disclosure. It is not intended to identify key or essential elements of this disclosure, nor is it intended to indicate any scope of any particular implementation of this disclosure or any scope of the claims. Its sole purpose is to present some concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0004] Systems and methods for facilitating the identification of one or more products associated with a media item are disclosed. In some implementations, one method includes obtaining first data. The first data includes a first identifier of a first product determined in association with a content item based on first metadata of the content item. The first data further includes a first confidence value associated with the first product and the content item. The method further includes obtaining second data. The second data includes a second identifier of the first product determined in association with the content item based on first image data of the content item. The second data further includes a second confidence value associated with the first product and the content item. The method further includes providing the first data and the second data to a trained machine learning model. The method further includes obtaining a third confidence value associated with the first product from the trained machine learning model. The method further includes adjusting the second metadata associated with the content item based on the third confidence value.

[0005] In some embodiments, the method further includes providing first metadata of the content item as input to the second model. The method may include obtaining first data as output of the second model. In some embodiments, the method further includes providing first image data of the content item as input to the second model. The method may further include obtaining first dimension-reduced data as output of the second model. The method may further include obtaining second dimension-reduced data associated with the first product from a data repository. The second dimension-reduced data may be obtained from the data repository in response to obtaining the first data. The second data may be generated based at least on the first dimension-reduced data and the second dimension-reduced data.

[0006] The method may further include providing second image data to a third model. The method may further include obtaining a third identifier for the first product from the third model. The second dimensionally reduced data may be obtained from a data repository in response to obtaining the third identifier for the first product. The second data may be generated based at least on the first dimensionally reduced data and the second dimensionally reduced data.

[0007] In some embodiments, the content item is a video. The first data may further include indications of timestamps for one or more frames of the video associated with the product. Adjusting the second metadata may include including the indications of the first product and the timestamps in the second metadata.

[0008] In some embodiments, metadata may include a title for a content item. Metadata may include a description of the content item. Metadata may include explanatory text associated with the content item. Adjusting second metadata may include adjusting explanatory text associated with the product.

[0009] In some embodiments, the method further includes training a machine learning model to generate a trained machine learning model. Training the machine learning model may include receiving image-based product data associated with a plurality of content items. The image-based product data may include indications of one or more products detected in an image and one or more product image confidence values. Training may further include receiving metadata-based product data associated with the plurality of content items. The metadata-based product data may include indications of one or more products detected in text and one or more confidence values. Training may further include receiving data indicating products included in the plurality of content items. Training may further include providing the image-based product data and the metadata-based product data as training inputs to the machine learning model. Training may further include providing the data indicating products included in the plurality of content items as the target output to the machine learning model.

[0010] In some embodiments, the method further includes receiving third data, which includes a third identifier of a first product category associated with a content item. The method may further include providing the third data to a trained machine learning model. A third confidence value may be based on the first data, the second data, and the third data.

[0011] In another aspect, a method includes obtaining first metadata associated with a content item. The method further includes providing the first metadata to a first model. The method further includes obtaining a first product identifier based on the first metadata and a first confidence value associated with the first product identifier as output of the first model. The method further includes obtaining image data of the content item. The method further includes providing the image data to a second model. The method further includes obtaining a second product identifier based on the image data and a second confidence value associated with the second product identifier as output of the second model. The method further includes providing data including the first product identifier, the first confidence value, the second product identifier, and the second confidence value as input to a third model. The method further includes obtaining a third product identifier and a third confidence value as output of the third model. The method further includes adjusting the metadata associated with the content item based on the third product identifier and the third confidence value.

[0012] In some embodiments, generating a second confidence value includes reducing the dimensionality of the image data to generate first dimensionally reduced data. Generating the second confidence value may further include obtaining the second dimensionally reduced data from a data repository. The second dimensionally reduced data may be associated with a product indicated by a second product identifier. Generating the second confidence value may further include performing one or more operations to generate the second confidence value. The second confidence value may be based on one or more differences between the first dimensionally reduced data and the second dimensionally reduced data.

[0013] In some embodiments, each of the first product identifier, the second product identifier, and the third product identifier identifies the first product.

[0014] In some embodiments, the method further includes obtaining a timestamp associated with the image data and the content item. Adjusting the second metadata associated with the content item may include adjusting the second metadata to include an indication that the product identified by the second product identifier is associated with the timestamp and the content item.

[0015] In some embodiments, the second metadata includes machine-generated explanatory text. A first product identifier may be associated with a product. The language associated with the product may have been incorrectly transcribed during the generation of the machine-generated explanatory text. Updating the second metadata associated with a content item may include replacing the product-associated portion of the machine-generated explanatory text with the product's text identifier.

[0016] In some embodiments, the method further includes providing a fourth product identifier and a fourth confidence value to a third model. The method may further include obtaining a fifth product identifier as the output of the third model. The third product identifier is associated with a first product, and the fifth product identifier may be associated with a second product.

[0017] In another aspect, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform an operation including obtaining first data. The first data includes a first identifier of a first product determined in association with a content item based on first metadata of the content item. The first data further includes a first confidence value associated with the first product and the content item. The operation further includes obtaining second data, the second data including a second identifier of the first product determined in association with the content item based on first image data of the content item. The second data further includes a second confidence value associated with the first product and the content item. The operation further includes providing the first data and the second data to a trained machine learning model. The operation further includes obtaining a third confidence value associated with the first product from the trained machine learning model. The operation further includes adjusting the second metadata associated with the content item based on the third confidence value.

[0018] In some embodiments, the operation further includes providing first image data of the content item as input to the second model. The operation may further include obtaining first dimension-reduced data as output of the second model. The operation may further include obtaining second dimension-reduced data associated with the first product from a data repository. Data from the data repository may be obtained in response to obtaining the first data. The second data may be based at least on the first dimension-reduced data and the second dimension-reduced data.

[0019] In some embodiments, the content item is a video. The first data may further include indications of timestamps for one or more frames of the video associated with the product. Adjusting the second metadata may include including the indications of the first product and the timestamps in the second metadata.

[0020] In some embodiments, the operation further includes receiving third data, the third data including a third identifier of a first product category associated with the content item. The operation may further include providing the third data to a trained machine learning model. A third confidence value may be generated based on the first data, the second data, and the third data.

[0021] Where appropriate, optional features of one aspect can be combined with other aspects. Attached Figure Description

[0022] The aspects and implementations of this disclosure will become more fully understood from the detailed description given below and from the accompanying drawings of the various aspects and implementations of this disclosure; however, these should not be construed as limiting this disclosure to any particular aspect or implementation, but are intended for explanation and understanding only.

[0023] Figure 1 An example system architecture for providing relevant and associated product information, according to some embodiments, is shown.

[0024] Figure 2 This is a block diagram of a system 200, according to some embodiments, including a dataset generator for creating datasets for one or more models.

[0025] Figure 3A This is a block diagram illustrating a system for generating output data, such as associations between products and content items, according to some embodiments.

[0026] Figure 3B This is a block diagram of an example system for generating data describing the association between content items and one or more products, according to some embodiments.

[0027] Figure 4A An apparatus for presenting an example user interface (UI) according to some embodiments is depicted, the example user interface including UI elements indicating a product associated with a content item.

[0028] Figure 4B An apparatus for presenting an example UI according to some embodiments is depicted, the example UI including UI elements indicating products associated with content items.

[0029] Figure 4C An apparatus for presenting an example UI according to some embodiments is described, the example UI including UI elements for presenting content items and UI elements for presenting information about products associated with the content items.

[0030] Figure 4D An apparatus for presenting an example UI according to some embodiments is described, the example UI including UI elements for presenting content items and UI elements for facilitating transactions associated with the product.

[0031] Figure 4E An example device with UI elements overlaid on content presentation elements, according to some embodiments, is depicted.

[0032] Figure 5A This is a flowchart of a method for generating a dataset for a machine learning model, according to some embodiments.

[0033] Figure 5B This is a flowchart of a method for updating metadata for content items according to some embodiments.

[0034] Figure 5C This is a flowchart of a method for training a machine learning model associated with content items and product pairings, according to some embodiments.

[0035] Figure 5D This is a flowchart of a method for adjusting metadata associated with content items, according to some embodiments.

[0036] Figure 5E This is a flowchart of a method for presenting UI elements associated with one or more products, according to some embodiments.

[0037] Figure 5F This is a flowchart of a method for instructing a device to present one or more UI elements associated with a product, according to some embodiments.

[0038] Figure 6 This is a block diagram illustrating an exemplary computer system according to an implementation of the present disclosure. Detailed Implementation

[0039] This disclosure relates to methods and systems for facilitating the pairing of media items (e.g., content items, content, etc.) with associated products. A platform (e.g., a content sharing platform, etc.) may enable users to access (e.g., via a client device connected to the platform) media items (e.g., video items, audio items, etc.) hosted by the platform. The platform may provide access to media items to a user's client device via a network (e.g., the Internet) (e.g., by transmitting the media item to the user's client device, etc.). Media / content items may have one or more additional associated activities that can enhance user interaction with the content item. For example, interactive activities may include a comment section for the content item, live chat associated with the content item, etc. Some content items may be associated with one or more products. For example, a product may be displayed as part of a content item, a product may be rated within the content item, and the content item may be associated with one or more products (e.g., sponsored by a company associated with the one or more products), etc.

[0040] In conventional systems, identifying the products associated with content items can be difficult, inconvenient, or time-consuming. Content platforms (e.g., platforms that provide content for users) may struggle to identify the products associated with content items. In some systems, content creators can identify one or more products associated with content items, content channels, lists of content items, etc. In some systems, content creators can include information about one or more products in content items, such as photos of the item, items appearing in videos, etc. In some systems, content creators can include information about one or more products in one or more fields associated with the content item, such as the content item title, content item description, content item comments (e.g., pinned comments), etc.

[0041] In a conventional system, users may not be able to easily determine whether a content item has one or more associated products. The content item may not have indicators of associated products. Users may need to be presented with content items (e.g., watching a video) to determine whether the content item has one or more associated products.

[0042] In conventional systems, users may struggle to identify the products associated with content items. There may be no direct indication of an associated product for a content item (e.g., when selecting a content item to consume or view). Product information may be difficult to find, for example, scattered across different fields such as content item titles, descriptions, and comment sections. Product information can be included within the content item itself; for example, a video might include audio describing one or more products, and an image item might include images of one or more products. Extracting this information from content items can be difficult, time-consuming, and error-prone.

[0043] In conventional systems, receiving additional information about a product that is a feature within a content item can be difficult, time-consuming, etc. In some systems, the product may appear within the content item (e.g., within a video). Users may not be given additional information (e.g., product name, merchant name, etc.) and can perform searches independently of the content platform to learn more about the product. The name and / or merchant associated with the product may be provided (e.g., in the description or title of the content item). Users can perform searches (e.g., independently of the content platform) to learn more about the product, such as product changes, related products, availability, and price. Guidance may be provided to facilitate further understanding of the product (e.g., user guidance, guidance to the processing device in the form of links to the merchant's website, etc.). Users may receive additional information from a separate source independent of the content platform (e.g., a website).

[0044] In a typical system, information about the products included in a content item can become outdated. For example, within a content item or related fields (e.g., content item title, content item description, content item reviews, etc.), the content creator may include additional information about one or more products, such as pricing, merchant information, availability information, alternative versions, or changes. Some information included in a content item or related fields may not be updated with changes to that information, but may depend on updates by the content creator.

[0045] In conventional systems, there may be obstacles that make it difficult, time-consuming, or cumbersome for users to purchase one or more items associated with a content item. Users can search for products, search for merchants that store or sell products, etc. In some embodiments, a content item or associated field may include instructions for purchasing the item (e.g., the description of the content item may include one or more links to products associated with the content item). Users may be directed to a separate platform independent of the content platform to complete the purchase of one or more products.

[0046] Users may spend significant time and computing resources searching for information about the products covered by a content item. For example, videos can be very long, and products of interest may not be shown until the very end. Users may spend considerable time consuming videos to obtain accurate information about products of interest, leading to increased usage of the client device's computing resources. Furthermore, the computing resources on the client device that enable users to consume media items may not be available for other processes, which can reduce overall efficiency and increase the overall latency of the client device.

[0047] The aspects of this disclosure can address one or more of these drawbacks of conventional methods. In some embodiments, aspects of this disclosure can enable automated identification of products displayed and / or included in content items. In some embodiments, aspects of this disclosure can enable the use of a model to identify products from text associated with a content item. The text may include a title of the content item, a description of the content item, explanatory text associated with the content item, etc. The explanatory text may be machine-generated explanatory text, such as explanatory text generated for video or audio content items by a speech-to-text model. The model may be a machine learning model. The model may output a confidence value indicating the likelihood that the content item includes a product.

[0048] In some embodiments, aspects of this disclosure can implement the use of a model to identify a product from an image of a content item. The content item may be an image or include images. The content item may be or include video. One or more images of the content item may be provided to a model configured to identify a product from the images. The model may reduce the dimensionality of the images. The model may search for similar images associated with the product. The model may search for similar dimensionality-reduced images in the reduced-dimensional space. The model may be a machine learning model. The model may output a confidence value indicating the probability that the content item includes the product.

[0049] In some embodiments, aspects of this disclosure implement the use of a model (e.g., a fusion model) to determine whether a product is included in the content item. The fusion model may receive indications of one or more products detected by a model that receives text associated with the content item as input. The fusion model may receive indications of confidence values ​​indicating that one or more products are included in the text. The fusion model may receive indications of one or more products detected by a model that receives an image of the content item as input. The fusion model may receive indications of confidence values ​​indicating that one or more products are included in the image. The fusion model may determine the confidence that one or more products appear in the content item and associated data (e.g., title, description, etc.). The fusion model may be a machine learning model.

[0050] In some embodiments, product detection can be used to improve content or associated information (e.g., descriptions, captions). Data associated with content items can be provided to the model. For example, machine-generated captions for content items can be provided to the model. The model can identify one or more captions that could be misrepresentations of a product name (e.g., machine-generated captions may include the closest English equivalent to a spoken product name). One or more images of content items can be provided to the model. The model can determine the likelihood that one or more products are included in the images. Information associated with content items (e.g., metadata, descriptions, captions, etc.) can be updated based on one or more detected products.

[0051] In some embodiments, aspects of this disclosure implement indicators for content items having one or more associated products. The list of content items may include one or more indicators among the content items in the list that include one or more associated products. Indicators may include visual indicators (e.g., a "shopping" symbol or text indicating one or more products displayed in association with the content item), additional fields (e.g., a panel including product information), etc. In some embodiments, the list of content items may be presented to a user via a user interface (UI). UI elements may be associated with a content platform (e.g., may be presented via an application associated with a content delivery platform). The UI may include elements indicating that a content item is associated with one or more products. In some embodiments, user interaction with UI elements may cause additional UI elements with additional information, such as a list of products associated with the content item, to be presented.

[0052] In some embodiments, aspects of this disclosure enable a user to identify products associated with content items. In some embodiments, the list of content items presented to the user may include a list of products associated with one or more content items. For example, the list of content items may be displayed to the user via a UI. The UI may include one or more UI elements that present one or more products associated with the content items to the user. The UI elements may be provided via an application associated with a content delivery platform for the content items. In some embodiments, UI elements listing products associated with a content item may be presented in response to detecting user interaction with a UI element indicating that the content item has associated products. In some embodiments, user interaction with products in the product list may cause the UI to present additional information to the user.

[0053] In some embodiments, aspects of this disclosure enable the provision of additional information to a user about one or more products associated with a content item. UI elements may be provided to the user that provide additional information about the product. The UI elements may provide information such as product variations (e.g., color variations, size variations, etc.), related products, availability (e.g., associated with one or more merchants), and / or price. The UI elements may be provided via an application associated with the content item, a content delivery platform, etc. UI elements displaying additional product information may be presented in response to user interaction with another UI element, such as selecting a content item to view, selecting a UI element indicating one or more associated products, etc. In some embodiments, further UI elements may be provided after user interaction with a UI element, for example, to encourage product purchase.

[0054] In some embodiments, aspects of this disclosure enable automatic updates of information connected to one or more products associated with a content item. In some embodiments, the content providing platform associated with a content item may include one or more storage devices containing product data, communicating with one or more storage devices containing product data, or having a communication connection with one or more storage devices containing product data. For example, the content platform may maintain and update a database of product-related information, and changes to the database may be reflected in UI elements presented to the user.

[0055] In some embodiments, aspects of this disclosure implement a simplified purchase process for users. Upon receiving an indication of a user's intention to purchase a product (e.g., after user interaction with UI elements associated with the product), UI elements designed to facilitate the purchase can be presented to the user. In some embodiments, users can purchase products via applications associated with content items, content delivery platforms, etc. In some embodiments, users may be directed to one or more external merchants (e.g., merchant applications, merchant websites, etc.) to purchase products.

[0056] In some embodiments, product-related UI elements may be provided in response to various operations of the application (e.g., an application associated with a content platform). Product-related UI elements may be provided as part of a list of content items customized for the user—e.g., customized for a user account associated with the user. Product-related UI elements may be provided as part of a list of content items related to previously presented content items, such as a "watch next" list, a recommended video list, etc. Product-related UI elements may be provided as part of a list of content items generated in response to a user search. Product-related UI elements may be provided as part of a product-specific list, such as a shopping area in an application associated with a content platform. The inclusion of product-related UI elements, product-related content items, the number of product-related UI elements or content items presented, etc., may be based on multiple metrics. Metrics may include user viewing history, user search history, etc.

[0057] In some embodiments, aspects of this disclosure can enable quick access to one or more portions of a product-related content item. For example, a UI element may include a list of products associated with a content item. One or more products in the list may be associated with a portion of the content item—for example, a timestamp of a video content item. After interaction with the product-related portion of the UI element, the content item may render the relevant portion of the content item (e.g., a video may begin playing the portion of the video associated with the content item). In some embodiments, the list of products associated with a content item may be updated as the content item is rendered. For example, the list of products associated with a video may be rearranged as the video is played. For example, the product currently highlighted in the video may be at the top of the product list, and products currently on the screen may be grouped together in the product rendering UI element, etc.

[0058] The aspects of this disclosure can provide technical advantages over previous solutions. These aspects enable automated product detection in content items. This can improve the content creator's experience by automatically associating one or more products with content items (e.g., removing the burden of associating products with content items from the creator). The automatically generated product associations can have improved accuracy due to the use of multiple sources (e.g., text and image searches for products), fusion models, etc. Model-based product detection can be used to improve content items, associated data, etc. For example, object detection can improve machine-generated captions or descriptions of content items. More accurate captions can improve the user experience when consuming content items. These improvements can reduce the time required for content creators to generate accurate content, reduce the time users spend discovering content items and / or products of interest, and increase the accuracy of machine-generated information associated with content items, etc. Therefore, computational resources at client devices associated with content creators, content viewers, and / or platforms are reduced and can be used for other processes, increasing overall efficiency and reducing overall system latency.

[0059] This disclosure improves the user experience when searching for content items, viewing content items, scrolling through lists of content items, etc. Users may be able to identify a content item from a list of content items, for example, without being shown a content item with associated products. A seamless way can be provided to increase user interaction with products associated with content items; for example, interaction with UI elements indicating the presence of products associated with a content item can lead to the presentation of content items with further information about one or more products, and further interaction can encourage the purchase of one or more products, etc. The presentation of one or more UI elements can streamline the user experience. Users may be able to easily retrieve additional information, for example, within an application associated with the content platform / content item. Users may be able to more easily purchase products associated with content items. Users can be directed to relevant sections of content items based on their expressed interest in products associated with them. Users may be able to easily view price information, availability information, product changes, related products, etc., within a single application context. Such implementations can save users time and effort, simplify the shopping and / or purchasing process, and streamline product research, evaluation, and / or selection processes, etc.

[0060] Figure 1 An example system architecture 100 for providing content and associated product information, according to some embodiments, is illustrated. System architecture 100 includes a client device 110, one or more networks 105, a content platform system 102, and a product identification system 175. Content platform system 102 includes one or more server machines 106, one or more data repositories 140, and may include various platforms designed to perform tasks (e.g., tasks associated with content delivery). The platform of content platform system 102 may be hosted by one or more server machines 106. The platform of content platform system 102 may include and / or be hosted on one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.) and one or more data repositories (e.g., hard disks, storage, and databases), and may be coupled to one or more networks 105. In some embodiments, components of content platform system 102 (e.g., server machines 106, data repositories 140, hardware associated with one or more platforms, etc.) may be directly connected to one or more networks 105. In some embodiments, one or more components of the content platform system 102 may access the network 105 via another device, such as a hub, switch, etc. In some embodiments, one or more components of the content platform system 102 may directly connect to... Figure 1The system communicates with other components depicted, such as components of the product identification system 175, such as server machines 170 and / or 180. A data repository 140 may be included in one or more server machines 106, including external data storage, etc. The content platform system 102 may include an advertising platform 165, a social networking platform 160, a recommendation platform 157, a search platform 145, and a content delivery platform 120. The product identification system 175 includes server machines 170 and 180 and a set of models 190. The product identification system 175 may include further devices such as a data repository, further servers, etc. Multiple operations of the product identification system 175 may be performed by a single physical or virtual device.

[0061] One or more networks 105 may include one or more public networks (e.g., the Internet), one or more private networks (e.g., local area networks (LANs), wide area networks (WANs), one or more wired networks (e.g., Ethernet networks), one or more wireless networks (e.g., 802.11 networks), one or more cellular networks (e.g., LTE networks), routers, hubs, switches, server computers, and / or combinations thereof. In one implementation, some components of architecture 100 are not directly connected to each other. In one implementation, system architecture 100 includes separate networks 105.

[0062] One or more data repositories 140 may reside in memory (e.g., random access memory), cache, drive (e.g., hard disk drive), flash drive, etc., and may be part of one or more database systems, one or more file systems, or another type of component or device capable of storing data. One or more data repositories 140 may include or span multiple storage components (e.g., multiple drives or multiple databases) across multiple computing devices (e.g., multiple server computers). The data repository may be persistent storage capable of storing data. Persistent storage may be a local or remote storage unit, an electronic storage unit (e.g., main memory), or a similar storage unit. Persistent storage may be a monolithic device or a distributed set of devices.

[0063] Content items 121A-121C (e.g., media content items) may be stored on one or more data repositories. A data repository may be part of one or more platforms. Examples of content items 121 may include, but are not limited to, digital videos, digital films, animated images, digital photographs, digital music, digital audio, digital video games, collaborative media content presentations, website content, social media updates, ebooks, e-journals, digital audiobooks, web blogs, software applications, etc. Content items 121A-121C may also be referred to as media items. Content items 121A-121C may be pre-recorded or live. For brevity and simplicity, video may be used as an example of content item 121 (e.g., content item 121A) throughout this document. Videos may include pre-recorded videos, live videos, short videos, etc.

[0064] Content items 121A-121C can be provided by a content provider. The content provider can be a user, company, organization, etc. The content provider can provide content item 121 as a video (e.g., content item 121A). The content provider can provide content item 121 that includes live content; for example, content item 121 may include live video, live chat associated with the video, etc.

[0065] Client device 110 may include devices such as televisions, smartphones, personal digital assistants, portable media players, laptops, e-book readers, tablets, desktop computers, game consoles, set-top boxes, etc.

[0066] Client device 110 may include communication application 115. A user can consume content item 121 (e.g., content item 121A) via communication application 115. For example, communication application 115 may access one or more networks 105 (e.g., the Internet) via the hardware of client device 110 to provide content item 121 to the user. As used herein, “media,” “media item,” “online media item,” “digital media,” “content,” “media content item,” and “content item” can include electronic files that can be executed or loaded using software, firmware, and / or hardware configured to render content items. In one implementation, communication application 115 may be an application that allows a user to compose, send, and receive content item 121 (e.g., video) via a platform (e.g., content delivery platform 120, recommendation platform 157, social networking platform 160, and / or search platform 145) and / or a combination of platforms and / or networks.

[0067] In some embodiments, the communication application 115 may be a social networking application, a video sharing application, a video streaming application, a video game streaming application, a photo sharing application, a chat application, or a combination of such applications (or aspects thereof). The communication application 115 associated with the client device 110 may render, display, present, and / or play one or more content items 121 to one or more users. For example, the communication application 115 may provide a user interface 116 (e.g., a graphical user interface) to be displayed on the endpoint device 110 for receiving and / or playing video content. In some embodiments, the communication application 115 is associated with (and managed by) a content platform 102 or a content delivery platform 120.

[0068] In some embodiments, the communication application 115 may include a content viewer 113 and an associated product component 114. A user interface 116 (UI) may display the content viewer 113 and the associated product component 114. The associated product component 114 may be used to display UI elements that display information about one or more products (e.g., one or more products associated with a content item). The associated product component 114 may display UI elements for informing the user that a content item has one or more associated products (e.g., the associated product component 114 may cause the display of a "shopping" symbol near the content viewer 113, or cause elements to be displayed near elements for selecting content items from a list of content items and indicating that the content item has one or more associated products, etc.). The associated product component 114 may cause the UI to display information about one or more products associated with a content item (e.g., the associated product component 114 may cause the UI to display a list of products associated with the content item, images of products associated with the content item, price information, or information linking products to the content item, such as timestamps of portions of videos associated with the product, etc.). The related product component 114 can cause UI elements to display additional information about one or more products, such as product variations (e.g., color variations, size variations, etc.), related products, recommended products, etc. The related product component 114 can also cause UI elements to display one or more options for purchasing one or more products. In some embodiments, users may be able to navigate between different views provided by the related product component 114. This can be combined with... Figures 4A to 4EFurther description of example UI elements associated with the product related to the content item. In some embodiments, for example, if more than one content item with an associated product is displayed, the related product component 114 may cause more than one UI element to be displayed. In some embodiments, the related product component 114 may cause UI elements indicating related products not to be displayed, for example, based on user settings, user preferences, user history, etc. In some embodiments, multiple content viewers 113 and / or related product components 114 may be associated with a user interface, communication application, client device, etc. For example, multiple content items may be displayed to the user at one time. In some implementations, the communication application 115 is a web browser that can access, retrieve, render, and / or navigate content (e.g., web pages, such as Hypertext Markup Language (HTML) pages, digital media items, etc.) and may include the related product component 114 and the content viewer 113, the content viewer being an embedded media player embedded in a user interface 116 (e.g., a web page associated with the viewed content) provided by the content delivery platform 120. Alternatively, application 115 is not a web browser, but a standalone application (e.g., mobile application, desktop application, game console application, TV application, etc.) downloaded from or pre-installed on client device 110 from a platform (e.g., content delivery platform 120, recommendation platform 157, social networking platform 160, or search platform 145). Standalone application 115 can provide a user interface 116 including a content viewer 113 (e.g., an embedded media player) and related product components 114.

[0069] In some embodiments, content platform system 102 may include (e.g., hosted by server machine 106) a product information platform 161. Product information platform 161 may store, retrieve, provide, receive, and otherwise associate data with one or more products related to one or more content items. Content platform system 102 may provide data to relevant product components 114 of client device 110. Product information platform 161 may include information provided by content creators, such as a list of products associated with content items provided to content platform system 102 by the content creator. Product information platform 161 may include information provided by one or more users, such as identifying products associated with content items, for example, in response to having content items presented to them. Product information platform 161 may include information provided by product identification system 175, such as using one or more machine learning models to identify products displayed in content items and provide indications of associated products and content items to product information platform 161.

[0070] In some embodiments, a communication application 115 installed on a client device 110 may be associated with a user account; for example, a user may log in to an account on the client device 110. In some embodiments, multiple client devices 110 may be associated with the same client account. In some embodiments, providing information about the association of a product with one or more content items may be performed based on the user account—for example, account settings, account history (e.g., history of interactions with UI elements including associated product information).

[0071] In some embodiments, client device 110 may include one or more data repositories. The data repository may include commands (e.g., instructions causing operation when executed by a processing device) for rendering a UI (e.g., user interface 116). Instructions may include instructions for rendering interactive components, such as UI elements that a user can interact with to present additional information about one or more products associated with a content item. In some embodiments, the instructions may cause the processing device to render UI elements that present information about one or more products associated with one or more content items (e.g., multiple videos reviewing a product may be presented along with UI elements that present more information about a product that the user is interested in).

[0072] In some embodiments, one or more server machines 106 may include computing devices such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc., and may be coupled to one or more networks 105. Server machine 106 may be a standalone device or part of any platform (e.g., content delivery platform 120, social networking platform 160, etc.).

[0073] Social networking platform 160 can provide online social networking services. Social networking platform 160 can provide communication application 115 for users to create profiles and perform activities using their profiles. Activities may include updating profiles, exchanging messages with other users, evaluating (e.g., liking, commenting, sharing, recommending) status updates, photos, videos, etc., and receiving notifications related to the activities of other users. In some embodiments, users can share additional product information (e.g., as provided by product information platform 161) with one or more additional users via social networking platform 160.

[0074] Recommendation platform 157 can be used to generate and provide content recommendations (e.g., articles, videos, posts, news, games, etc.). Recommendations can be based on search history, content consumption history, followed / subscribed channel content, linked profiles (e.g., friend lists), trending content, etc. Recommendation platform 157 can be used to generate, for example, user homepage feeds, user viewing lists, user playlists, etc. One or more UI elements, such as indicating associated products, displaying lists of associated products, presenting product information, presenting one or more options for purchasing products, etc., can be presented in conjunction with, as part of, associated with, or accessible from homepage feeds, viewing lists, playlists, watch-next-view lists, etc. The presentation of one or more UI elements can be performed based on user history, user settings, data indicating user preferences (e.g., demographic data), etc.

[0075] Search platform 145 can be used to allow users to query one or more data repositories 140 and / or one or more platforms and receive query results. Users can use search platform 145 to search for content items, search topics, etc. For example, users can use search platform 145 to search for content items with one or more associated products. Search platform 145 can be used to search for content items related to a certain type of product (e.g., headphone review videos). Search platform 145 can be used to search for content items related to a certain product (e.g., a specific brand and / or model of headphones). In response to receiving a search query, one or more UI elements can be displayed to the user. The type, style, etc., of the displayed UI elements can be based on the content of the user's search. For example, in response to a search for content related to a certain type of product (e.g., headphones), UI elements can be displayed to indicate that the suggested content items have one or more associated products in light of the search. As a further example, in response to a search for content items related to a more specific product (e.g., "best headphones for podcasting"), different UI elements can be displayed to provide information about the products associated with the content items. As a further example, in response to a search for content items related to a specific product (e.g., a specific brand and / or model), different UI elements can be displayed to provide specific information about the searched product, thereby indicating that the product is associated with content items recommended in light of the search.

[0076] Content delivery platform 120 can be used to provide access to content item 121 to one or more users and / or to provide content item 121 to one or more users. For example, content delivery platform 120 can allow users to consume, upload, download, and / or search for content item 121. In another example, content delivery platform 120 can allow users to evaluate content item 121, such as by approving (“like”), disapproving, recommending, sharing, rating, and / or commenting on content item 121. In another example, content delivery platform 120 can allow users to edit content item 121. Content delivery platform 120 may also include websites (e.g., one or more web pages) and / or one or more applications (e.g., communication application 115) that can be used to provide access to content item 121 to one or more users. For example, client device 110 can use communication application 115 to access content item 121. Content delivery platform 120 may include any type of content delivery network that provides access to content item 121.

[0077] Content delivery platform 120 may include multiple channels (e.g., channel A 125, channel B 126, etc.). A channel may be a collection of content available from a common source, a collection of content with a common topic or theme, etc. Data content may be user-selected digital content, user-provided digital content, user-uploaded digital content, digital content selected by a content provider, digital content selected by a broadcaster, etc. For example, channel A 125 may include two videos (e.g., content items 121A-121B). A channel may be associated with an owner, who may be a user capable of performing actions on that channel. Content may be one or more content items 121. The data content of a channel may be pre-recorded content, live content, etc. Although a channel is described as one implementation of a content delivery platform, the implementations disclosed herein are not limited to a content sharing platform that provides content items 121 through a channel model.

[0078] Product identification system 175, server machine 170, and server machine 180 may each include one or more computing devices, such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, graphics processing units (GPUs), application-specific integrated circuits (ASICs) for accelerators (e.g., tensor processing units (TPUs)). The operation of predictive servers 112, server machines 170 and 180, data storage 140, etc., can be performed by cloud computing services, cloud data storage services, etc.

[0079] Product recognition system 175 may include one or more models 190. The models 190 included in product recognition system 175 can perform tasks related to recognizing one or more products from content items. One or more models among the models 190 may be trained machine learning models. Figure 3A and Figure 5C Describes the operations for generating a trained machine learning model, including training, validating, and testing the model.

[0080] Model 190 may include one or more text parsing models 191. Text parsing models 191 may be configured to receive text as input and generate one or more indications of products associated with that text as output. For example, a first model in text parsing model 191 may be configured to predict associated products from the title of a content item, a second model in text parsing model 191 may be configured to predict associated products from the (e.g., written) description of a content item, a third model may be configured to predict associated products from the description of a content item (e.g., automatically generated descriptions, machine-generated descriptions, user-provided descriptions, etc.), and so on. In some embodiments, all operations in text parsing model 191 may be performed by a single model. In some embodiments, models in text parsing model 191 may be configured to generate product context information—e.g., information indicating that a content item is associated with one or more products—as output. For example, product context information may indicate that a content item includes a category of products, such as a group of different products (e.g., product type, product trademark, product category—such as "electronic device," etc.).

[0081] Model 190 may include one or more image resolution models 192. Image resolution models 192 may be configured to identify products from one or more images. Image resolution models 192 may include one or more models for identifying products in images, models configured to isolate product images from content item images (e.g., remove background elements), models configured to determine the identity of products in content item images, etc. Operations of image resolution models 192 may be performed by a single model. Image resolution models 192 may include one or more models configured to provide images to product recognition models. For example, image resolution models 192 may include models configured to extract a portion of a still image of a content item, models configured to extract one or more frames of a video content item, etc. Models in image resolution models 192 may be provided with one or more frames of a video, one or more portions of one or more frames of a video, etc., and generate one or more products and one or more confidence values ​​associated with one or more products as output. For example, models in image resolution models 192 may receive one or more frames of a video content item as input and generate a list of products with confidence values ​​indicating the probability that a product is included in the image of a content item as output. Image parsing model 192 may include one or more models that determine which images to use from the content items. For example, image parsing model 192 may include one or more models that select frames for image recognition from video content items.

[0082] Image resolution model 192 may include one or more models configured to reduce the dimensionality of an image. For example, an image may be simplified to a value vector. In some embodiments, one or more models in image resolution model 192 may be configured to reduce the dimensionality of an image in such a way that similar images (e.g., images of the same product) can be represented similarly (e.g., by similar vectors) after dimensionality reduction. One or more models in image resolution model 192 may be configured to compare a reduced-dimensional image from a content item (e.g., a value vector generated from one or more frames of a content item video) with a reduced-dimensional image of a known product (e.g., via product information platform 161).

[0083] Model 190 may include text correction model 193. Text correction model 193 may be configured to provide correction to text associated with a content item. Text correction model 193 may be configured to adjust the text associated with the content item to include one or more products referenced in the content item. One or more models in text correction model 193 may be configured to adjust computer-generated text, machine-generated text, automatically generated text, etc., associated with the content item. One or more models in text correction model 193 may be configured to update the video narration (e.g., incorrect narration) to include one or more products. In some embodiments, the machine-generated text (e.g., narration) associated with the content item may be incorrect. For example, when generating the narration, the product name may be replaced with an approximate term (e.g., the product name may not be a word in the language of the narration, the product name may be a word in a language different from the language of the narration, etc.). Models in text correction model 193 may be configured to identify potentially incorrect portions of the text and recommend correction, perform correction, warn the user or another system, etc. For example, the model in text correction model 193 can receive machine-generated narration of a video, identify parts of the narration that may incorrectly substitute for words used for product names in the language, and provide data indicating potentially incorrect text to another model, system, user, etc.

[0084] Model 190 may include a fusion model 194. Fusion model 194 may receive one or more indications of products associated with a content item as input. In some embodiments, fusion model 194 may receive outputs from one or more other models (e.g., text parsing model 191, image parsing model 192, etc.) as input. Fusion model 194 may receive one or more indications of products associated with a content item and one or more indications of confidence values. For example, fusion model 194 may receive indications of one or more products detected in the title of a content item and confidence values ​​associated with those products. Fusion model 194 may further receive indications of one or more products detected in the description of a content item and confidence values ​​associated with those products. Fusion model 194 may further receive indications of one or more products detected in the description of a content item and confidence values ​​associated with those products. Fusion model 195 may generate one or more products detected in association with a content item as output. The fusion model 195 can further generate confidence values ​​associated with the confidence that the product appears in the content item and that the product is associated with the content item. Further operations can be performed based on the output of the fusion model 195 (e.g., product identity information and confidence values) (e.g., UI elements describing the product associated with the content item can be rendered).

[0085] One type of machine learning model that can be used to perform some or all of the tasks mentioned above is an artificial neural network, such as a deep neural network. Artificial neural networks typically include a feature representation component with classifier or regression layers that map features to a desired output space. Convolutional neural networks (CNNs), for example, host multiple convolutional filter layers. Pooling is performed at lower layers and can solve non-linear problems; on top of these lower layers, multilayer perceptrons are typically attached, mapping the top-level features extracted by the convolutional layers to a decision (e.g., a classification output).

[0086] Recurrent Neural Networks (RNNs) are another type of machine learning model. RNN models are designed to interpret a series of inputs that are inherently correlated with each other, such as time-tracking data, sequential data, etc. The output of the RNN's perceptron is fed back into the perceptron as input to generate the next output.

[0087] Deep learning is a class of machine learning algorithms that use cascades of multiple layers of non-linear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure of layers, where different layers learn different levels of representations corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in image recognition applications, the original input might be a matrix of pixels; the first representation layer might abstract the pixels and encode edges; the second layer might encode the arrangement of edges; the third layer might encode higher-level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer might identify the character of the scan. Notably, the deep learning process can learn on its own which features are best placed at which level. The "depth" in "deep learning" refers to the number of layers through which data is transformed. More precisely, deep learning systems have a considerable credit assignment path (CAP) depth. A CAP is a chain of transformations from input to output. CAP describes the underlying causal relationship between the input and output. For feedforward neural networks, the depth of CAP can be the depth of the network and can be the number of hidden layers plus one. For recurrent neural networks where signals can propagate through a layer more than once, the depth of CAP is potentially unlimited.

[0088] In some embodiments, the product identification system 175 further includes server machine 170 and server machine 180. Server machine 170 includes a dataset generator 172 capable of generating datasets (e.g., a set of data inputs and a set of target outputs) for training, validating, and / or testing a model 190 comprising one or more machine learning models. The following is about... Figure 2 and Figure 5A The following describes some operations of the dataset generator 172 in detail. In some embodiments, the dataset generator 172 may partition historical data (e.g., pre-existing content item data, content items with one or more specified associated products, content items with product assignments provided by one or more users, etc.) into a training set (e.g., 60 percent of the historical data), a validation set (e.g., 20 percent of the historical data), and a test set (e.g., 20 percent of the historical data).

[0089] In some embodiments, components of the product identification system 175 can generate multiple sets of features. For example, features may be a rearrangement of input data, a combination of input data, a reduction in the dimensionality of input data, a subgroup of input data, etc. One or more datasets can be generated based on one or more features of the input data.

[0090] Server machine 180 includes a training engine 182, a validation engine 184, a selection engine 185, and / or a testing engine 186. Engines (e.g., training engine 182, validation engine 184, selection engine 185, and testing engine 186) can refer to hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, processing device, etc.), software (such as instructions that run on a processing device, general-purpose computer system, or dedicated machine), firmware, microcode, or a combination thereof. Training engine 182 may be able to train one or more models 190 using one or more sets of features associated with a training set from dataset generator 172. Training engine 182 can generate multiple trained models 190, each corresponding to a distinct set of features from the training set. The dataset generator 172 can receive the output of a trained model (e.g., the fusion model 194 can be trained based on the output of the text parsing model 191 and / or the image parsing model 192), collect the data into training, validation and testing datasets, and use the datasets to train a second model (e.g., the fusion model 194).

[0091] The validation engine 184 may validate the trained model 190 using a corresponding set of features from the validation set generated by the dataset generator 172. For example, a first trained machine learning model 190 trained using a first set of features from the training set may be validated using a first set of features from the validation set. The validation engine 184 may determine the accuracy of each of the trained models 190 based on the corresponding set of features from the validation set. The validation engine 184 may discard trained models 190 with accuracy that does not meet a threshold accuracy. In some embodiments, the selection engine 185 may select one or more trained models 190 with accuracy that meets a threshold accuracy. In some embodiments, the selection engine 185 may select the trained model 190 with the highest accuracy among the trained models 190.

[0092] The testing engine 186 may be able to test the trained model 190 using a corresponding set of features from the test set generated by the dataset generator 172. For example, a first trained machine learning model 190 trained using the first set of features from the training set can be tested using the first set of features from the test set. The testing engine 186 can determine the trained model 190 with the highest accuracy among all trained models based on the test set.

[0093] In the case of a machine learning model, model 190 can refer to a model artifact created by training engine 182 using a training set, which includes data inputs and corresponding target outputs (correct answers to the corresponding training inputs). Patterns in the dataset that map data inputs to target outputs (correct answers) can be found, and mappings that capture these patterns are provided for machine learning model 190. Machine learning model 190 can use one or more of the following: Support Vector Machine (SVM), Radial Basis Function (RBF), clustering, supervised machine learning, semi-supervised machine learning, unsupervised machine learning, k-nearest neighbor algorithm (k-NN), linear regression, random forest, decision forest, neural network (e.g., artificial neural network, recurrent neural network), linear model, function-based model (e.g., NG3 model), etc. Synthetic data generator 174 can include one or more machine learning models, which can include one or more models of the same type (e.g., artificial neural networks).

[0094] Automated (e.g., model-based) detection of products from content items and associated data offers significant technical advantages over other methods. In some embodiments, content items showcasing a product (e.g., product review videos reviewing the product) can become linked to or associated with the product without the attention, actions, time, etc. of the content creator (e.g., data linking the product to the content item can be generated). In some embodiments, content items advertising a product (e.g., content items that may be sponsored or promote one or more products) can be linked to or associated with the product. Model-based detection of products in content items can generate product associations for products that are not specifically shown in the content item but exist in the content item (e.g., products that a user may be interested in purchasing may be in video content items on the screen). Model-based detection of products in content items can generate product associations for products advertised in the content item. By providing users with UI elements (e.g., indicating that a product is associated with a content item), users can be directed to products present in the content item based on model-based detection.

[0095] One or more models 190 can be run on the input to generate one or more outputs. The models can determine (e.g., extract) confidence data from the output, which indicates the level of confidence that the model's output accurately describes the content item. For example, the model can determine that a first product is associated with the content item and determine the confidence that the model has correctly identified the first product in the content item. One or more components of the product identification system 175 can use the confidence data to determine whether to update data associated with the content item, such as whether to associate one or more products with the content item, or whether to update one or more descriptive terms for the content item.

[0096] Confidence data may include or indicate the confidence level that the model's output (e.g., one or more products) is an accurate indication of the product associated with the content item. For example, the confidence level output by the model (e.g., associated with a product identified in the content item) may be a real number between 0 and 1 (inclusive). 0 may indicate that the predicted product is associated with the content item with zero confidence, and 1 may indicate that the predicted product is associated with the content item with absolute confidence. In response to confidence data indicating that the confidence level for a predetermined number of instances (e.g., percentage of instances, frequency of instances, total number of instances, etc.) is below a threshold level, the product identification system 175 may cause one or more trained models 190 (e.g., based on updated and / or new data used for training, validation, testing, etc.) to be retrained. Retraining may include (e.g., via dataset generator 172) generating one or more datasets.

[0097] For illustrative purposes and not for limitation, aspects of this disclosure describe training one or more machine learning models 190 using historical data, and feeding current data (e.g., newly updated content items, content items not previously associated with a product, etc.) into one or more trained machine learning models to determine outputs indicating content item-product associations. In other embodiments, heuristic models, physics-based models, or rule-based models are used to determine the association of one or more products with content items (e.g., without using trained machine learning models). In some embodiments, historical data may be used to train such models. In some embodiments, these models may be retrained using historical data. In heuristic, physics-based, or rule-based models, information about... Figure 2 Data input 210 describes any information.

[0098] In some embodiments, the functionality of client device 110, product identification system 175, content platform system 102, server machine 170, server machine 180, and server machine 106 can be provided by a fewer number of machines. For example, in some embodiments, server machines 170 and 180 can be integrated into a single machine, while in other embodiments, server machines 170, 180, and 106 can be integrated into a single machine. In some embodiments, client device 110 and server machine 106 can be integrated into a single machine. In some embodiments, the functionality of client device 110, server machine 106, server machine 170, server machine 180, and data repository 140 can be performed by cloud-based services.

[0099] Generally, if appropriate, functions described in one embodiment as being performed by client device 110, server machine 106, server machine 170, and server machine 180 may also be performed on server machine 106 in other embodiments. Additionally, functionality attributable to a particular component may be performed by different or multiple components operating together. For example, in some embodiments, product identification system 175 may determine the association between a product and a content item. In another example, content platform system 102 may determine the association between a content item and one or more products.

[0100] Furthermore, the functionality of a specific component can be performed by different or multiple components operating together. One or more of server machines 106, 170, or 180 can be accessed as services provided to other systems or devices through appropriate application programming interfaces (APIs).

[0101] In implementations of this disclosure, "user" can refer to a single individual. However, other implementations of this disclosure encompass "user" as an entity controlled by a group of users and / or automated sources. For example, a group of individual users united as a community in a social network can be considered a "user." In another example, an automated consumer can be an automated ingestion pipeline of one or more platforms, one or more content items, such as a topic channel. In addition to the above description, users may be provided with controls that allow them to choose whether and when the system, program, or feature described herein can collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current location) and whether to send content or communications to the user from a server. Furthermore, before storing or using certain data, it can be processed in one or more ways to remove personally identifiable information. For example, the user's identity can be processed to make it impossible to determine any personally identifiable information about the user, or, in the case of location information, the user's geographic location can be generalized (e.g., generalized to the city, zip code, or state level) to make it impossible to determine the user's specific location. Thus, users can control what information about themselves is collected, how that information is used, and what information is provided to them.

[0102] Figure 2This is a block diagram of a system 200, according to some embodiments, including a dataset generator 272 for creating datasets for one or more models. The dataset generator 272 can use historical data to create datasets (e.g., data input 210, target output 220). Unsupervised machine learning models can be trained using dataset generators similar to dataset generator 272; for example, the target output 220 may not be generated by dataset generator 272. Semi-supervised machine learning models can be trained using dataset generators similar to dataset generator 272; for example, the target output 220 corresponding to a subgroup of data input 210 may be generated by dataset generator 272.

[0103] Dataset generator 272 can generate datasets for training, testing, and validating models. Dataset generator 272 can generate datasets for machine learning models. System 200 can generate datasets for training, testing, and / or validating fusion models, such as those used to determine the probability of one or more products appearing in a content item. Systems similar to system 200 can generate datasets for training, testing, and / or validating models with different functionalities, having corresponding modifications to the input and / or output data included in the dataset. Models for parsing text (e.g., extracting one or more references to products from text associated with a content item), parsing images (e.g., extracting one or more references to products from images associated with a content item), correcting text (e.g., including one or more references to products in machine-generated text associated with a content item), etc., can have datasets for training, testing, and / or validating models generated by a dataset generator similar to dataset generator 272.

[0104] In some embodiments, a dataset generator, such as dataset generator 272, may be associated with two or more separate models (e.g., the dataset can be used to train an ensemble model). For example, an input dataset may be provided to a first model, the output of the first model may be provided to a second model, and a target output may be provided to the second model to train, test, and / or validate the first and second models (e.g., an ensemble model).

[0105] Dataset generator 272 can generate one or more datasets for use, for example, during training, validation, and / or testing operations, to be provided to the model. Multiple sets of historical data can be provided to the machine learning model. Multiple sets of historical text parsing data 264A-264Z can be provided as data input to the machine learning model (e.g., a fusion model). The text parsing data can be provided by the machine learning model, and may include, for example, one or more product and confidence values ​​identified by the machine learning model in text associated with content items. Multiple sets of historical image parsing data 265A-265Z can be provided as data input to the machine learning model. The image parsing data can be provided by a trained machine learning model, and may include, for example, one or more product and confidence values ​​identified by the machine learning model in one or more images associated with content items.

[0106] In some embodiments, dataset generator 272 can be configured to generate datasets for training, testing, validating, and other purposes of fusion models. A dataset generator similar to dataset generator 272 can generate multiple sets of text data (e.g., content item title text, content item description text, content item explanatory text, etc.) as data input to train a machine learning model for identifying one or more products associated with a content item. A dataset generator similar to dataset generator 272 can generate multiple sets of image data (e.g., one or more frames from a video content item, portions of one or more frames from a video content item, etc.) as data input to train a machine learning model for identifying one or more products associated with a content item.

[0107] In some embodiments, the dataset generator 272 generates a dataset (e.g., training set, validation set, test set) that includes one or more data inputs 210 (e.g., training input, validation input, test input). The data inputs 210 may be provided to... Figure 1 The training engine 182, validation engine 184, or testing engine 186 are provided. The dataset can be used to train, validate, or test the model (e.g., a fusion model, a text parsing model, an image parsing model, etc.). The dataset generator 272 can generate datasets (e.g., training set, validation set, test set) that include one or more data inputs 210. The data inputs 210 can be referred to as "features," "attributes," "vectors," or "information."

[0108] In some embodiments, the dataset generator 272 can generate a first data input corresponding to a first set of historical text parsing data 264A and / or a first set of historical image parsing data 265A, to train, validate, or test a first machine learning model. The dataset generator 272 can generate a second data input corresponding to a second set of historical econometric data 264B and / or a second set of design rule data 265B, to train, validate, or test a second machine learning model. About Figure 5A Further descriptions are provided of some implementation examples for generating training sets, test sets, validation sets, etc.

[0109] In some embodiments, dataset generator 272 can generate a target output 220 to be provided for training, testing, validation, etc., of one or more machine learning models. Dataset generator 272 can generate product association data 268 as the target output 220. Product association data 268 may include identifiers (e.g., manually tagged product associations) of one or more products associated with content items. Product association data 268 may include input-output mappings, e.g., a set of historical text parsing data 264A may be associated with a first set of product association data 268, etc. The machine learning model can be updated (e.g., trained) by providing input data, generating an output, and comparing that output with a provided target output (e.g., "correct answer"). Various weights, biases, etc., of the model are then updated to better align the model with the training data. This process can be repeated many times to generate a model that provides accurate outputs for a threshold portion of the provided input. The target output 220 may share one or more features of the data input 210; for example, the target output 220 may be organized into attributes or vectors, the target output 220 may be organized into groups A, Z, etc.

[0110] In some embodiments, a dataset generator similar to dataset generator 272 can be used in conjunction with a text parsing model and configured to determine associations with one or more products associated with a content item. Product associations may include contextual associations, such as product trademarks, product types, product categories, etc. The dataset generator can generate a list of products associated with the content item, product types, product trademarks, etc., as the target output, which are associated with the text of the content item. A dataset generator similar to dataset generator 272 can be used in conjunction with an image parsing model and configured to determine associations with one or more products associated with a content item. The dataset generator can generate a list of products associated with the image input, product categories, product types, etc., as the target output, which are associated with the image input. A dataset generator similar to dataset generator 272 can be used in conjunction with a text correction model. The text correction model can be configured to identify machine-generated text that incorrectly provides one or more words in the target language to replace product names. One or more sets of machine-generated text can be provided to the text correction model as input data, and products associated with the text (e.g., products not correctly captured by the machine generation of the text) as the target output.

[0111] In some embodiments, after generating a dataset and using that dataset to train, validate, or test a machine learning model, the model can be further trained, validated, tested, or tuned (e.g., adjusting weights or parameters associated with the model's input data, such as connection weights in a neural network). The model can be tuned and / or retrained based on data different from the original training operation, such as data generated after the model's training, validation, and / or testing.

[0112] Figure 3A This is a block diagram illustrating a system 300A for generating output data (e.g., product / content item association data) according to some embodiments. System 300A can be used in conjunction with a fusion model to generate product / content item association data and confidence data based on potential product / content item associations generated by other models (e.g., a text-based model that detects products in text associated with content items, an image-based model that detects products in images associated with content items, etc.). Systems similar to System 300A can be used to generate output data from other types of models, such as text parsing models, image parsing models, text correction models, etc.

[0113] At box 310, system 300A (e.g., Figure 1 The product identification system 175 (components) uses data (e.g., via...) to train, validate, and / or test machine learning models. Figure 2The dataset generator 272) performs data partitioning. In some embodiments, training data 364 includes historical data, such as historical associations between products and text-based content items, historical associations between products and image-based content items, etc. In some embodiments, for example, when system 300A generates output from a fusion model, training data 364 may include data generated by one or more trained machine learning models—for example, models configured to detect products in text or images associated with content items. Training data 364 may undergo data partitioning at box 310 to generate training set 302, validation set 304, and test set 306. For example, the training set may be 60% of the training data, the validation set may be 20% of the training data, and the test set may be 20% of the training data.

[0114] The generation of training set 302, validation set 304, and test set 306 can be customized for a specific application. For example, the training set can be 60% of the training data, the validation set can be 20% of the training data, and the test set can be 20% of the training data. System 300A can generate multiple sets of features for each of the training, validation, and test sets. For example, if training data 364 includes product associations extracted from text data from more than one text source (e.g., titles associated with content items and descriptions associated with content items), the input training data can be partitioned into a first set of features including products identified from text from a first source, and a second set of features including products identified from text from a second source. The target input can be partitioned into groups, the target output can be partitioned into groups, both can be partitioned into groups, or neither can be partitioned into groups. Multiple models can be trained on different sets of data.

[0115] At box 312, system 300A uses training set 302 (e.g., via...). Figure 1 The training engine (182) performs model training. Training a machine learning model can be implemented in a supervised learning manner, which involves: providing the model with a training dataset including labeled inputs, observing its outputs, defining the error (by measuring the difference between the output and the labeled values), and tuning the model's weights using techniques such as deep gradient descent and backpropagation to minimize the error. In many applications, this process is repeated across many labeled inputs in the training dataset to produce a model that can produce correct outputs when presented with inputs different from those present in the training dataset. In some embodiments, training a machine learning model can be implemented unsupervised, for example, without providing labels or classifications during training. Unsupervised models can be configured to perform anomaly detection, result clustering, etc.

[0116] For each training data item in the training dataset, that training data item can be fed into the model (e.g., into a machine learning model). The model can then process the input training data item (e.g., indications of one or more products detected in combination with the content item, and associated confidence values, etc.) to generate an output. The output may include, for example, a list of products that can be associated with the content item and their corresponding confidence values. The output can be compared with the labels of the training data items (e.g., a set of manually labeled products associated with the content item).

[0117] The processing logic then compares the generated output (e.g., predicted product / content item associations) with labels already included in the training data items (e.g., a manually generated list of product / content item associations). The processing logic determines the error (i.e., classification error) based on the difference between the output and the labels. Based on this error, the processing logic adjusts one or more weights and / or values ​​of the model.

[0118] In training a neural network, an error term or increment can be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more parameters (weights of one or more inputs to a node) of its nodes. Parameters can be updated via backpropagation, such that the nodes in the highest layer are updated first, followed by the nodes in the next layer, and so on. An artificial neural network contains multiple layers of "neurons," where each layer receives values ​​from neurons in the previous layer as input. The parameters of each neuron include weights associated with the values ​​received from each neuron in the previous layer. Therefore, adjusting the parameters can include adjusting the weights of each input to one or more neurons in one or more layers of the artificial neural network.

[0119] System 300A can use multiple sets of features from training set 302 (e.g., a first set of features from training set 302, a second set of features from training set 302, etc.) to train multiple models. For example, system 300A can use a first set of features from the training set (e.g., subgroups of training data 364, such as data associated only with a subgroup of models configured to generate product / content items, etc.) to train a model to generate a first trained model, and use a second set of features from the training set to train a model to generate a second trained model. In some embodiments, the first trained model and the second trained model can be combined to generate a third trained model (e.g., the third trained model may be better than either the first or second trained model alone). In some embodiments, the multiple sets of features used when comparing models can overlap (e.g., the first set of features is based on the product, content item title, description, and some images, and the second set of features is based on the content item description, a different set of images of the content item, and the detected context of the content item (e.g., the type of product associated with the content item)). In some embodiments, hundreds of models can be generated, including models and combinations of models with various feature arrangements.

[0120] At box 314, system 300A uses verification set 304 (e.g., via...). Figure 1 The system 300A uses a validation engine 184 to perform model validation. The system 300A can use a corresponding set of features from the validation set 304 to validate each model in the trained models. For example, the system 300A can use a first set of features from the validation set to validate a first trained model and a second set of features from the validation set to validate a second trained model. In some embodiments, the system 300A can validate hundreds of models generated at box 312 (e.g., models with various feature permutations, model combinations, etc.). At box 314, the system 300A can (e.g., via model validation) determine the accuracy of each of one or more trained models and can determine whether one or more trained models have an accuracy that meets a threshold accuracy. In response to determining that no trained model has an accuracy that meets the threshold accuracy, the flow returns to box 312, where the system 300A performs model training using different sets of features from the training set, an updated or expanded training set provided by the dataset generator, etc. In response to determining that one or more trained models have an accuracy that meets the threshold accuracy, the flow continues to box 316. System 300A can (e.g., based on a validation set) discard trained models with accuracy below a threshold.

[0121] At box 316, system 300A (e.g., via...) Figure 1The selection engine 185 performs model selection to determine which of one or more trained models that meet a threshold accuracy has the highest accuracy (e.g., the selected model 308 based on the validation in box 314). In response to determining that two or more trained models that meet the threshold accuracy have the same accuracy, the process can return to box 312, where system 300A performs model training using a further refined training set corresponding to a further refined set of features to determine the trained model with the highest accuracy.

[0122] At box 318, system 300A uses test set 306 (e.g., via...). Figure 1 The system 300A uses a test engine 186 to perform model testing to test the selected model 308. The system 300A can use a first set of features from the test set to test the first trained model to determine if the first trained model meets a threshold accuracy (e.g., based on the first set of features from test set 306). In response to the selected model 308's accuracy not meeting the threshold accuracy (e.g., the selected model 308 is overfitted to training set 302 and / or validation set 304 and is not suitable for other datasets such as test set 306), the process continues to box 312, where the system 300A performs model training (e.g., retraining) using different training sets corresponding to different sets of features, different content items, etc. In response to determining that the selected model 308 has accuracy meeting the threshold accuracy based on test set 306, the process continues to box 320. At least in box 312, the model can learn patterns in the training data to make predictions, and in box 318, the system 300A can apply the model to the remaining data (e.g., test set 306) to test the predictions.

[0123] At box 320, system 300A uses a trained model (e.g., selected model 308) to receive current data 322 (e.g., newly uploaded content items, newly created content items, content items not included in the training, testing, or validation sets of selected model 308, etc.) and determines (e.g., extracts) output data 324 (e.g., product / content item associations and corresponding confidence values) from the output of the trained model. Corrective actions associated with the content item and / or associated data can be performed based on the output data 324. For example, instructions can be updated to include presenting UI elements along with the content item indicating that the specified content item includes one or more associated products, instructions can be updated to include presenting UI elements along with the content item containing additional information about the associated products, etc. Instructions can depend on additional factors of the content item, such as user preferences, presentation context (e.g., search page, homepage, etc.). In some embodiments, current data 322 may correspond to the same type of features in historical data used to train the machine learning model. In some embodiments, the current data 322 corresponds to a subgroup of features of the type used in the historical data for training the selected model 308 (e.g., a machine learning model may be trained using product association and / or contextual information and confidence values ​​from multiple sources such as text-based and image-based sources, and the subgroup of such data is provided as the current data 322).

[0124] In some embodiments, the performance of the machine learning model (e.g., model 308 of selection) can be tuned, improved, and / or updated over time. For example, additional training data can be provided to the model to improve its ability to correctly classify product associations with content items. In some embodiments, a portion of the current data 322 can be provided (e.g., via...) Figure 1 The training engine 182) can be used to retrain the model. This portion of the current data 322 can be labeled (e.g., manually labeled), and the label can be provided to retrain the model (e.g., as the current target output data 346). The current data 322 and the current target output data 346 can be used to periodically and continuously update and / or improve the selected model 308.

[0125] In some embodiments, one or more of actions 310-320 may occur in various orders and / or occur together with other actions not presented and described herein. In some embodiments, one or more of actions 310-320 may not be performed. For example, in some embodiments, one or more of the following may not be performed: data partitioning of block 310, model validation of block 314, model selection of block 316, or model testing of block 318.

[0126] System 300A has described a fusion model. The fusion model takes one or more indications and confidence values ​​(e.g., confidence that the product is indeed referenced in the content item) of a product detected in association with a content item (e.g., from text associated with the content item, from an image associated with the content item, etc.), and generates an overall probability as output that the product is referenced in the content item based on multiple inputs. Other machine learning-based tasks can be performed using systems similar to System 300A; for example, text or image parsing models—whose outputs are fed as input to the fusion model—can be operated on using appropriate substitutions for the input data, output data, etc., in a manner similar to that described in conjunction with System 300A.

[0127] Figure 3B This is a block diagram of an example system 300B for generating associations between content items and one or more products, according to some embodiments. System 300B may include multiple modules, such as image recognition 330, image verification 340, text recognition 350, fusion 360, etc. In some embodiments, multiple modules of system 300B may operate together to identify products from content items. For example, the product detected in an image of the content item and the metadata of the content item (e.g., title, description, captions, etc.) may be provided to a fusion model to determine the likelihood that one or more products are associated with the content item based on multiple input channels. Actions may be taken in response to the likelihood that a product appears in the content item, as determined by the fusion model (e.g., updating the metadata of the content item to include an indication of the product).

[0128] Image recognition module 330 can be used to identify one or more products from images associated with the model. For example, images from a content item can be compared with images in a database of products (e.g., thousands of products) to identify products present in the video. Present products may include specific topics of the content item (e.g., products evaluated in the content item), products included in the content item (e.g., products that appear incidentally, products that appear without being specifically highlighted, etc.), etc. Text recognition module 350 can identify one or more products associated with a content item from metadata / text data associated with the content item (e.g., from text including the title of the content item, the description of the content item, narration associated with the content item, etc.). Image verification module 340 can use one or more images to verify products identified in the content item. For example, image verification module 340 can operate similarly to image recognition module 330, but can be used (e.g., by comparing images of potential products with a more limited range of product images provided by another module) to confirm the existence of one or more products identified by a single module. The fusion module 360 ​​can receive candidate products included in the content item and their associated confidence values, and determine the probability of one or more products appearing in the content item based on various inputs.

[0129] Image recognition 330 can be used to determine products associated with content items—such as videos—that have visual components. Image recognition 330 may include frame selection 332. Frame selection 332 can be used to select one or more frames of an image from the video for searching for a product. Frame selection 332 can occur via random sampling, periodic sampling, smart sampling methods, etc. For example, the content item (e.g., video) can be fed into a machine learning model, and the machine learning model can be trained to predict frames in the video that may include one or more products.

[0130] One or more frames can be provided to the object detection model 334. The object detection model 334 can extract predicted objects from the one or more frames. For example, the object detection model 334 can isolate potential products from image data of content items such as people, animals, and backgrounds. The object detection model 334 can be or include a machine learning model.

[0131] Images of detected objects can be fed into embedding 336. Embedding 336 may include converting one or more images to a lower dimension. Embedding 336 may include providing one or more images to a dimensionality reduction model. This dimensionality reduction model may be a machine learning model. The dimensionality reduction model can be configured to reduce the dimension of similar images in a similar manner. For example, embedding 336 may receive images as input and generate value vectors as output. Embedding 336 can be configured, trained, etc., such that similar images (e.g., images of the same or similar products) are similarly represented in a reduced-dimensional vector space (e.g., by Cartesian distance, by cosine distance, by another distance metric, etc.). Embedding 336 can generate dimensionality-reduced data.

[0132] Reduced-dimensional image data can be provided to product recognition 338. Product recognition 338 can identify one or more products associated with the reduced-dimensional representation provided by embedding 336. Product recognition 338 can compare the reduced-dimensional image data (e.g., provided by embedding 336) with reduced-dimensional image data of products included in product image index 339 (e.g., generated from product images by the same machine learning model used by embedding 336). Product image index 339 can be stored as part of a data repository. Product image index 339 can include, for example, many products (e.g., hundreds of products, thousands of products, or more). Product image index 339 can include associations between stored image data (e.g., reduced-dimensional image data) and product identifiers, product indicators, etc. Product image index 339 can be segmented; for example, the stored reduced-dimensional data can be categorized into one or more categories, classifications, etc. For example, product recognition 338 can compare data received from embedding 336 with products of a specific category, type, classification, etc. In some embodiments, categories, types, classifications, etc., may be provided by one or more users, one or more content creators, or may be automatically detected (e.g., by one or more machine learning models). Content items or one or more products associated with content items may be associated with categories (e.g., general categories such as electronic devices, more limited categories such as screen devices, product classifications such as tablet computers, brands or trademarks, models, etc.). Product recognition 338 may generate one or more indications of products detected in the image of the content item (e.g., a list of products that can match the products represented in the product image index 339) and one or more indications of confidence values ​​(e.g., the confidence level that each product in the list has been accurately detected). The output of image recognition 330 may be used to update the metadata of the content item (e.g., to update to include associations with one or more products, to update to include one or more product identifiers or indicators, etc.). The output of image recognition 330 may be provided to image verification 340, for example, to verify the presence of an image in the content item recognized by image recognition 330. The output of image recognition 330 may be provided to fusion 360, for example, to generate a holistic and / or multi-input determination of the products included in the content item via fusion model 366. The output of image recognition 330 can be provided to text recognition 350 (not shown), for example, to limit the space of products queried, searched, compared, etc. by text recognition module 350. In some embodiments, image recognition 330 is used to identify products found in one or more frames of a video content item. For example, image recognition module 330 can be configured to generate a list of all products detected in any selected frame and provide a confidence value for each product in each selected frame.The image recognition module 330 can generate image-based product data, such as one or more identifiers of a product that is identified based on an image of a content item.

[0133] Image verification module 340 can be configured to verify the existence of an identified product of a content item using one or more images of the content item. For example, image verification module 340 may include a model configured to confirm the existence of a product identified by another model. Image verification module 340 may include secondary recognition 345. Secondary recognition 345 may include components similar to image recognition 330. In some embodiments, instead of communicating with product candidate image index 344, or otherwise, image recognition module 330 may communicate directly with product candidate image index 344. In some embodiments, secondary recognition 345 may perform a similar role to image recognition 330, but may include a different model, a model trained using different training data, a model configured to select different frames or detect different objects, etc.

[0134] Image verification 340 may include a synthesis model 341. The synthesis model 341 may receive indications of products identified by image recognition module 330, text recognition module 350, secondary recognition 345 (data stream not shown), etc. The synthesis model 341 may select an object detection model 342 to which data is to be provided (e.g., an object detection model specifically configured for product category or classification). The synthesis model 341 may include image selection; for example, it may provide one or more images to object detection 342, and may select one or more frames to be provided to object detection 334, etc. For example, the synthesis model 341 may provide one or more frames that may include products to object detection 342 based on data received from image recognition 330 and text recognition 350. The object detection model 342 may perform functions similar to those of object detection model 334, for example, modified by the functionality of the synthesis model 341. Embedding 343 may perform functions similar to embedding 336, for example, to reduce the dimensionality of the image of the detected product. In some embodiments, the product candidate image index 344 may include reduced-dimensional image data (e.g., value vectors) detected by other modules (e.g., image recognition 330, text recognition 350, etc.). Secondary recognition 345 may compare the reduced-dimensional image data (e.g., embedded image data) with candidate data in the product candidate image index 344 (e.g., products identified by modules other than image verification module 340) to confirm the presence of a product in the content item.

[0135] Text recognition 350 can be configured to identify one or more products from text data (e.g., metadata) associated with content items. Text recognition 350 can generate metadata-based product data, such as one or more product identifiers based on the metadata of the content item. Text recognition 350 can generate text-based product data, such as one or more product identifiers based on text data associated with content items. Text recognition 350 can identify products from one or more of the following: content item title, content item description, explanatory text associated with the content item (e.g., machine-generated explanatory text), comments associated with the content item, and / or other text data or metadata of the content item. Text data (e.g., metadata) associated with content items can be provided to text parsing model 352. Text parsing model 352 can be a machine learning model. Text parsing model 352 can be configured to detect or predict products from the text data associated with content items. Text parsing model 352 can be configured to detect one or more products having product identifiers stored in product identifier 354. Text parsing model 352 can provide output (e.g., a list of detected candidate products, associated confidence values, etc.) to image verification module 340. Text parsing model 352 can also provide output to synthesis model 341. Text parsing model 352 can provide output affecting products in image verification; for example, the output of text parsing model 352 can cause products detected by text recognition module 350 to be added to product candidate image index 344. Image verification module 340 can query the index (e.g., product candidate image index 344) which includes products detected by other modules such as image recognition module 330, text recognition module 350, etc.

[0136] The fusion module 360 ​​can receive output data (e.g., detected products, associated confidence values) from one or more sources (e.g., image recognition module 330, image verification module 340, text recognition module 350, etc.). The fusion module 360 ​​can further receive data from other sources, such as context term extraction 362 or additional feature extraction 363. Context term extraction 362 can, for example, provide context to potential products of a content item, such as detecting categories or topics associated with some products. Context term extraction 362 can be performed by one or more machine learning models. Context term extraction 362 can detect context information from text associated with the content item, metadata associated with the content item, etc. Additional feature extraction 363 can provide additional details that can be used to determine whether one or more products appear in the content item. Additional features may include video embeddings. Additional features may include other metadata about the content item, such as the date the content item was uploaded to the content delivery platform (e.g., compared to the product’s release date), the category of the content item (e.g., shopping or product review videos are more likely to include products than other types of videos), etc.

[0137] Data from multiple sources can be provided to the fusion model 366. The fusion model 366 can be configured to receive data, for example, including one or more products with confidence values, and to determine one or more products using the confidence values ​​that indicate the likelihood of a product appearing in a content item. In some embodiments, the content item can be a video. In some embodiments, the content item can be a live feed, such as a live video feed (e.g., a product review stream, unboxing stream, etc.). The content item can be a short video.

[0138] Figures 4A to 4E An example UI presented on a user device according to some embodiments is depicted, the example UI including UI elements indicating an associated product. Figures 4A to 4E This may include a UI provided as part of an application for devices 400A-400E, such as a web browser application, a mobile application associated with / provided by a content platform, etc. Figures 4A to 4EUser interaction with various elements of the UI can cause changes to the presented UI elements. For example, interacting with a UI element that indicates an associated product of a content item can cause a second UI element (e.g., replacing the first UI element, expanding the first UI element, etc.) to be displayed, presenting additional information (e.g., about the associated product). UI elements can include elements that, when interacted with, cause the UI to display less information about the associated product (e.g., collapsing a panel describing one or more associated products). Interacting with UI elements associated with one or more products can result in different effects, such as a transition to the UI environment used to present the content item. The new UI environment can include one or more UI elements associated with one or more products of the content item. Interacting with UI elements associated with products can cause UI elements that facilitate the transaction (e.g., purchase) of the product to be displayed. Figures 4A to 4E Various interactions between the UI and UI elements presented in the disclosure are possible (e.g., interacting with an element of a first UI layout may cause a transition to a second UI layout), and any transformation between sample UIs, similar UIs, inclusion of similar UI elements, etc., is within the scope of this disclosure. Figures 4A to 4E This is a description combined with video content items; other types of content items (e.g., image content, text content, audio content, etc.) can be presented in similar UIs. Where appropriate, it can be combined with... Figures 4A to 4E Any optional features, elements, etc., presented in relation to one or more graphs in a system can be included in a system similar to another graph in those graphs.

[0139] Figure 4A Device 400A is depicted presenting an example UI 402 according to some embodiments, the example UI including UI elements 404 indicating one or more associated products. As part of operations for presenting one or more content items for user selection, device 400A may present a UI including elements of UI 402 and / or similar UI elements. UI element 404 in Figure 4A The text describes it as being in a collapsed state, for example, the collapsed default state.

[0140] UI 402 includes a first content item selector 406 (e.g., video thumbnails) and a second content item selector 408. In some embodiments, more or fewer content items may be selectable, and UI 402 can be scrolled to view additional content items, etc. UI element 404 is associated with the content item indicated by content item selector 406. UI element 404 (and...) Figures 4A to 4DOther UI elements related to the product may be displayed above, beside, overlapping, inside, or below the content item or content item selector associated with UI element 404. In some embodiments, product information (e.g., product / content item association) may be provided by the content creator. Product information may be provided by one or more users. Product information may be provided by an administrator. Product information may be retrieved from content items, for example, via one or more machine learning models, or via a system such as system 300B.

[0141] In some embodiments, a user can interact with UI element 404 to be presented with replaced UI elements, updated UI elements, etc. For example, a user can interact with extended element 410 to display more information about the product associated with the content item. In some embodiments, extended UI element 404 can open a panel that includes additional information about one or more products associated with the content item. Extended UI element 404, interacting with UI element 404, etc., can adjust the presentation of UI 402 to, for example, include... Figures 4B to 4D The elements depicted in the text.

[0142] UI element 404 may include, for example, extended elements 410, indications of associated products (e.g., how many products are associated with a content item), and visual indications of products 412 (e.g., visual indications of available deals or purchases, visual indications of links to merchants offering products, etc.). User interaction with one or more components of UI element 404 may cause device 400A to modify the presentation of UI element 404; for example, a user selection of extended element 410 may cause the presentation of UI element 404 to be modified to an extended state.

[0143] In response to device 400A providing content to a platform (e.g., Figure 1The content delivery platform 120 sends a request for a content item, which can present a UI including UI element 404. UI element 404 can be presented in a homepage feed (e.g., a list of suggested content items for a user or user account). UI element 404 can be presented in suggested feeds (e.g., a list of suggested content items based on one or more recently presented content items). UI element 404 can be presented as part of a playlist (e.g., a list of content items to be presented, filled in by the user, the creator, etc.). UI element 404 can be presented in a search feed (e.g., in response to a search query generated by a user and sent to the content delivery platform by device 400A). For example, UI elements associated with one or more products can be presented based on the inclusion of products, product categories, product trademarks, etc., in the search query. UI element 404 can be presented in product-focused feeds (e.g., shopping content feeds). UI element 404 can be presented in response to user selection of a content item—e.g., it can be displayed after a user selection to watch an associated video, presented together with associated content items, etc. UI element 404 can be presented based on detected user interest in content items. For example, a user might hover over a video thumbnail (e.g., a user might hover the cursor over the thumbnail, or a user might pause scrolling through the presented thumbnail). UI element 404 can be presented when the hover meets one or more conditions (e.g., duration condition, thumbnail position condition, etc.). UI element 404 can be presented in response to additional data such as user account history, user settings, user preferences, etc. UI element 404 can be presented along with a list of content items to be presented, or UI element 404 can be presented while content items are being presented (e.g., when a video associated with the product associated with UI element 404 is playing).

[0144] Figure 4B Device 400B is depicted presenting an example UI 420 according to some embodiments, the example UI including UI element 422 indicating associated products. UI element 422 includes information about one or more products associated with a content item. UI element 422 in... Figure 4BThe UI element 422 is presented in an expanded state. It may include multiple components. For example, a first component may include information about a first product (e.g., an image, product name, price, timestamp, etc.), a second component may include information about a second product, and so on. In some embodiments, the UI element 422 may be scrollable, for example, to access information about additional products. The UI element 422 may include multiple tabs (e.g., the UI element 422 may be associated with product information and one or more other types of information). For example, the UI element 422 may include a product tab 424 and a chapter tab 426. In some embodiments, the product tab 424 may be open by default (e.g., the content of the product tab 424 may be presented by default). In some embodiments, the chapter tab 426 may be open by default (e.g., the content of the chapter tab 426 may be presented by default). In some embodiments, another tab may be open by default. For example, chapter tab 426 can be opened by default, except for content items with associated products. The default open tab can be selected based on user history (e.g., interaction with elements such as chapter tabs, product tabs, etc.) or search queries (e.g., searches including product names, related terms or phrases such as "product review"). UI element 422 can include additional elements, such as elements that users can use to control the presentation of UI 420, UI element 422, etc. For example, UI element 422 can include a "close" element for presenting UI 420 without UI element 422, and can include elements for displaying less information (e.g., for collapsing panels to resemble...). Figure 4A The UI element 404, the "collapse" element used to modify the presentation of UI element 422 to a collapsed state, and one or more elements used to display more information (e.g., selecting one or more listed products, listing product icons, etc. to present additional information about the products to promote product purchase, etc.).

[0145] The product tab 424 of UI element 422 may include one or more images of the product, information about the product (e.g., product name, product description, etc.), prices for one or more products, timestamps of content items related to the product, etc. In some embodiments, product information may be provided by content creators, one or more users, system administrators, etc. In some embodiments, product information may be retrieved using one or more models, such as machine learning models. For example, one or more machine learning models may be used to determine the presence of a product in a content item, the association of products in a content item, the timestamp or position of a product appearing in a content item, etc. Features such as... Figure 3BThe system 300B determines one or more products associated with a content item (e.g., video). The content item-related portion (e.g., timestamp) can be determined, for example, via the timing of the narration associated with the product, the timing of the display of images or frames of the video including the product, etc. In some embodiments, selecting a product can cause the presentation of the product-related portion of the content item, or cause the presentation of the content item to begin at a time indicated by the timestamp associated with the product, etc.

[0146] In some embodiments, the product-specific associated portion (e.g., visual components) of UI element 422 may be displayed by default, displayed differently (e.g., highlighted), etc. For example, upon receiving a search query from a user that includes the name of a product, UI element 422 may be displayed, including a display associated with the searched product.

[0147] In some embodiments, one or more associations between content items and products can be stored as, for example, metadata associated with the content item (the metadata associated with the content item may further include the content item title, description, presentation history, explanatory text associated with the content item, etc.). In response to a device (e.g., device 400B) executing instructions to display a list of content items for presentation, to present content items to a user, to present UI elements (e.g., UI element 422) including information about one or more products, the device can retrieve information about the product based on the metadata associated with the content item. Information about the product (e.g., images, associated products such as color variations, availability, price, etc.) can be retrieved from a data repository. The data repository can include information about the product and can be updated; for example, when information such as the price of the product changes, the UI can retrieve and display the updated information based on the content item / product association.

[0148] UI element 422 can be presented as part of: a home feed (e.g., a list of suggested content items for a user or user account), a suggested feed (e.g., a list of suggested content items based on one or more recently presented content items), a playlist, a list of search results, a shopping content page, etc. UI element 422 can be presented after selecting content items for presentation, after presenting content items, etc. UI element 422 can be presented on pause (e.g., pausing scrolling on a content thumbnail, etc.). Figure 4AUI element 404 (such as pausing scrolling on less detailed elements) is presented. In some embodiments, UI element 422 can be removed or replaced upon user action—such as during scrolling—and UI element 422 can be collapsed to resemble UI element 404 (e.g., to facilitate selection of a content item from a list of content items, to simplify scrolling through a list of content items, etc.). UI element 422 can be displayed when presenting a list of content items, when presenting a single content item (e.g., when playing a video associated with the product associated with UI element 422), etc.

[0149] Figure 4C A device 400C is depicted presenting an example UI 430 according to some embodiments. This example UI includes a UI element 432 for presenting content items and a UI element 434 for presenting information about associated products. UI element 434 may include... Figure 4B UI element 422 provides more detailed information about the product associated with the content item. UI element 434 can be product-focused, for example, used to display product information to the user. UI element 434 can include one or more components, such as components associated with a first product, components associated with a second product, etc. In some embodiments, UI element 434 can include a list of products associated with the content item. UI element 434 can be navigational, scrollable, etc. UI element 434 can include one or more control elements, such as a back button for returning to a previous view, a close button for closing UI element 434 and viewing (e.g., a different set of UI elements unrelated to the product), etc. In some embodiments, UI element 434 can be removed from UI 430 in response to another user action—such as a user scrolling past an associated content item. In some embodiments, user selection of a product presented via UI 434 can prompt the presentation of UI elements that promote the purchase of that product. In some embodiments, UI element 434 may be displayed in response to (e.g., based on user history, based on one or more terms in a user search query, based on a user selection of a shopping content item to browse and / or be presented, based on a user selection of a product or UI elements associated with a product, etc.) determining that the user is interested in one or more products associated with the content item.

[0150] UI element 434 may include one or more images and / or additional information about one or more products associated with a content item (e.g., a content item presented via UI element 432). The images and / or information may be provided by the content creator, by one or more users, retrieved from a database (e.g., based on product / content item association metadata), etc. In some embodiments, UI element 432 may scroll automatically. For example, UI element 432 may scroll as a content item is presented, such as making products associated with a portion of the currently presented content item visible. UI element 432 may be presented in response to a user selection of the content item to be presented. UI element 432 may be presented in response to other factors such as user history. UI element 432 may be presented when the user hovers over an associated content item, an associated UI element, etc.

[0151] In some embodiments, UI element 434 can (e.g., via...) Figure 4B UI element 422) presents information about a single product—for example, the product selected by the user. UI element 434 can be product-focused, can focus on a single product, and can display product variations (e.g., such as when combined with other products). Figure 4D (as described in UI element 444), or may include other elements, components and / or information related to other UI elements described herein.

[0152] Figure 4D A device 400D is depicted presenting an example UI 440 according to some embodiments. This example UI includes UI elements 442 for presenting content items and UI elements 444 for facilitating transactions associated with the product. UI element 444 may provide one or more fields associated with a user making a transaction (e.g., purchasing a product). For example, UI element 444 may include an alternative product panel 446, which may include information, images, prices, etc., about alternative products (e.g., products related to one or more products associated with the content item, such as color variations, size variations, product bundle variations, related products such as similar products from another trademark, etc.).

[0153] UI element 444 may include transaction element 448. In some embodiments, transaction element 448 may facilitate in-app transactions (e.g., purchases) that provide UI 440. In some embodiments, transaction element 448 may facilitate transactions via another application, another website, etc. For example, interacting with transaction element 448 may direct a user to a merchant's website, or may direct device 400D to open an application associated with the purchase of a product, etc.

[0154] UI element 444 may be navigational, scrollable, etc. UI element 444 may include one or more control elements, such as a back button for returning to the previous view, a close button for closing UI element 444 and displaying a different set of UI elements via UI 440, etc. In some embodiments, UI element 444 may be displayed as part of a list of content items for the user to select, as part of UI 440 that presents the content items, etc. UI element 444 may respond to determining that the user is interested in a transaction (e.g., purchase) associated with one or more products included in the content items—for example, from, etc. Figure 4C UI element 434 is displayed through UI elements such as selecting products, selecting content items, hovering over content items, searching for products or product-related terms included in a search query, and navigating to a list of content items focused on shopping. UI element 444 may receive information about one or more products from a database, for example, based on metadata that associates content items with one or more products.

[0155] Figures 4A to 4D The UI elements depicted can be integrated into various configurations. For example, a UI element such as UI element 404 can be presented. After user interaction with UI element 404, an element such as UI element 422 can be presented. After user interaction with UI element 422, UI element 434 can be presented. After user interaction with UI element 434, UI element 444 can be presented, and so on. One or more UI elements may include navigation elements for instructing the device to display different UI elements. For example, interacting with extension element 410 may cause UI element such as UI element 422 to be presented, and interacting with different elements of UI element 404 may cause UI elements such as UI element 434, UI element 444, etc. to be presented.

[0156] Other connections between UI elements are also possible. For example, interacting with a UI element such as UI element 404 may cause UI elements such as UI element 422, UI element 434, UI element 444, etc., to be displayed. Interacting with a UI element such as UI element 422 or a portion thereof may cause UI elements such as UI element 404, UI element 434, UI element 444, etc., to be presented. User interaction with a UI element such as UI element 434 or a portion thereof may cause UI elements such as UI element 404, UI element 422, UI element 444, etc., to be presented. User interaction with a UI element such as UI element 444 or a portion thereof may cause UI elements such as UI element 404, UI element 434, UI element 444, etc., to be displayed. The default UI element presented may depend on the environment in which the UI element is presented (e.g., a list of content items presented in response to search, homepage feeds, shopping feeds, viewing feeds, etc.; the environment including the currently presented content item; etc.). For example, the choice of the form of a UI element may be based on multiple factors. In some embodiments, the product name, category, etc., in the search query may include UI elements that can change the default UI elements, such as UI elements that cause the UI to display by default information about the product, UI elements that include product purchase options, etc. The determination of the form of the UI elements to be displayed may be based on user history, user account history, user actions (e.g., opening a homepage feed, displaying a view feed, transmitting a search query, selecting a shopping feed, etc.). Transitions between the forms of UI elements associated with a product can be determined using additional data similar to the data used to determine the form of the UI elements to be displayed.

[0157] Figure 4E An example device 400E is depicted according to some embodiments, having UI elements overlaid on a content presentation element 452. The device 400E includes a UI 450. The UI 450 may be provided by an application, such as an application associated with a content delivery platform. The presentation element 452 may present content items (e.g., videos). The UI 450 may present a list of additional content items, additional information associated with the presented content items (e.g., titles, descriptions, comments, live chat, etc.), additional UI elements associated with the product (e.g., UI elements or variations such as UI elements 404, 422, 434, 444), etc.

[0158] Presentation element 452 can be overlaid with one or more UI elements. UI element 454 can indicate the product included in the content item (e.g., shown in a video). UI element 454 can perform operations related to... Figures 4A to 4DOther UI elements can perform similar functions, such as presenting information about the product, displaying further information about the product, or encouraging product purchase. The placement of overlaid elements can be determined by one or more users, content creators, models (e.g., utilizing a machine learning model configured to detect products or objects in an image to avoid displaying areas containing the object in the content item), etc. UI element 454 can include visual indicators of the location of one or more related products within the content item (e.g., their location in a video). UI element 454 can be presented during the presentation of the content item—e.g., while a video is playing. UI element 454 can be shown and / or removed in response to the presence of related products in the content item, for example, it can be displayed when the content item is in a video. UI element 454 can indicate multiple products, identify products (e.g., name one or more products), display information about the products, etc. Multiple UI elements such as UI element 454 can be displayed, for example, on a thumbnail of the video, throughout the presentation of the video, or simultaneously during the video.

[0159] UI elements such as UI element 454 can be combined with other UI elements associated with the product. For example, UI element 456 can open a panel that includes information about multiple products associated with the content item, and UI element 454 can cause UI elements that include information about the product displayed in the image to be shown.

[0160] The overlaid UI element 454 can be displayed above the visual representation of the content item (e.g., in front of the visual representation of the content item, having a higher visual priority than the visual representation of the content item, etc.). For example, UI element 454 can be overlaid on a video thumbnail. UI element 454 can be displayed above the content item. For example, UI element 454 can be overlaid on a playing video. The rendering of UI element 454 can be performed in response to user actions. For example, after determining that a user is interested in one or more products (e.g., via a search query, via interaction with product-related UI elements, via user history, etc.), UI element 454 can be displayed overlaid on another UI element. In some embodiments, the content item can be a live video. In some embodiments, the content item can be a short video.

[0161] In some embodiments, UI element 454 may perform operations similar to those described for UI element 404, such as notifying the user that one or more products are associated with a content item. UI element 454 may respond to user interactions similarly to UI element 404, such as opening or expanding a panel containing product information, modifying the presentation of UI element 404 to display more or different information, expanding the UI element to include more information, initiating the presentation of a content item, etc. UI element 454 may respond to user interactions similarly to UI element 434, such as opening or expanding a panel that facilitates transactions.

[0162] Figures 5A to 5F This is a flowchart of methods 500A-500F related to content items of an associated product, according to some embodiments. Methods 500A-500F can be executed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, processing device, etc.), software (such as instructions that run on a processing device, general-purpose computer system, or special-purpose machine), firmware, microcode, or combinations thereof. In some embodiments, methods 500A-500F may be partially executed by… Figure 1 The content platform system 102, product identification system 175, and / or client device 110 are executed in the process. Method 500A may be partially performed by product identification system 175 (e.g., Figure 1 Server machine 170 and dataset generator 172 in the middle Figure 2 The dataset generator 272 in the system performs this operation. According to embodiments of this disclosure, the product identification system 175 can use method 500A to generate a dataset for at least one of training, validating, or testing a machine learning model. Methods 500B-500D can be performed by the product identification system 175 (e.g., ...). Figure 3B Method 500A can be executed by system 300B and / or server machine 180 (e.g., training, validation, and testing operations can be performed by server machine 180). Method 500E can be executed by client device 110. Client device 110 can utilize method 500E to display one or more UI elements associated with the product, thereby facilitating user identification of the product included in the content item. Method 500F can be executed by content platform system 102, for example, by the processing logic of content providing platform 120, to facilitate client device 110 to present one or more UI elements associated with the product. In some embodiments, a non-transitory machine-readable storage medium stores instructions that, when executed by a processing device (e.g., product identification system 175, server machine 180, etc.), cause the processing device to perform one or more of methods 500A-500F.

[0163] For ease of explanation, methods 500A-500F are depicted and described as a series of operations. However, the operations according to this disclosure can occur in various orders and / or concurrently, and may include other operations not presented and described herein. Furthermore, not all operations shown can be performed to implement methods 500A-500F according to the disclosed subject matter. Additionally, those skilled in the art will understand and recognize that methods 500A-500F may alternatively be represented as a series of interrelated states via state diagrams or events.

[0164] Figure 5A This is a flowchart of a method 500A for generating datasets for a machine learning model, according to some embodiments. (Reference) Figure 5A In some embodiments, at block 401, the processing logic of method 500A initializes the training set T to an empty set.

[0165] At box 502, the processing logic generates a first data input (e.g., a first training input, a first validation input), which may include one or more of product data, image data, metadata, text data, confidence data, etc. In some embodiments, the first data input may include a first set of features specific to a data type, and the second data input may include a second set of features specific to a data type (e.g., regarding...). Figure 3A (As described). In some embodiments, input data may include historical data.

[0166] In some embodiments, at block 503, the processing logic optionally generates a first target output for one or more of the data inputs (e.g., a first data input). In some embodiments, the input includes one or more predicted products detected in a content item and associated confidence intervals, and the target output may include tags of the products included in the content item. In some embodiments, the input includes one or more sets of data associated with a content item (e.g., image data such as frames or portions of frames of a video, metadata such as title text or narration text, etc.), and the target output is a list of products included in the content item. In some embodiments, the first target output is predictive data. In some embodiments, the input data may be in the form of narration text data, and the target output may be a list of possible corrections to the narration to include product names / references for a machine learning model configured to correct the narration by including product information. In some embodiments, no target output is generated (e.g., an unsupervised machine learning model is able to group the input data or look for relevance in the input data, rather than requiring a target output).

[0167] At block 504, the processing logic optionally generates mapping data indicating the input / output mapping. The input / output mapping (or mapping data) may refer to data inputs (e.g., one or more of the data inputs described herein), a target output for the data inputs, and the association between the data inputs and the target output. In some embodiments, such as those associated with a machine learning model in which a target output is not provided, block 504 may not be executed.

[0168] In some embodiments, at block 505, the processing logic adds the mapping data generated at block 504 to the dataset T.

[0169] At box 506, the processing logic branches based on whether the dataset T is sufficient for at least one of training, validating, and / or testing a machine learning model, such as... Figure 1 One of the models in 190. If yes, execution proceeds to box 507; otherwise, execution continues back to box 502. It should be noted that in some embodiments, the sufficiency of the dataset T may be determined solely based on the number of inputs in the dataset that are mapped to outputs in some embodiments, while in other embodiments, the sufficiency of the dataset T may be determined based on one or more other criteria (e.g., a measure of the diversity of data examples, accuracy, etc.) in addition to or in lieu of the number of inputs.

[0170] At box 507, the processing logic provides a dataset T (e.g., to) for training, validating, and / or testing the machine learning model 190. Figure 1(Server machine 180). In some embodiments, dataset T is a training set and is provided to training engine 182 of server machine 180 to perform training. In some embodiments, dataset T is a validation set and is provided to validation engine 184 of server machine 180 to perform validation. In some embodiments, dataset T is a test set and is provided to test engine 186 of server machine 180 to perform testing. For example, in the case of a neural network, the input value of a given input / output mapping (e.g., the numerical value associated with data input 210) is input into the neural network, and the output value of the input / output mapping (e.g., the numerical value associated with target output 220) is stored in the output node of the neural network. The connection weights in the neural network are then adjusted according to a learning algorithm (e.g., backpropagation, etc.), and the process is repeated for other input / output mappings in dataset T. After box 507, the model (e.g., model 190) may be trained using training engine 182 of server machine 180, validated using validation engine 184 of server machine 180, or tested using test engine 186 of server machine 180. The trained model can be implemented by the product recognition system 175 to generate output data, for example, used by the product information platform 161 to provide product data to users, provided to the fusion model, or used to update the metadata of content items to include one or more product associations.

[0171] Figure 5B This is a flowchart of a method 500B for updating metadata of a content item according to some embodiments. At block 510, processing logic (e.g., a processing device, computer processor, etc.) receives first data. The first data includes a first identifier (e.g., an indicator, a pointer to further data, a code identifying the product, etc.) determined in association with the metadata of the content item. The content item may be or include visual content, audio content, text content, video content, etc. This can be achieved by providing the metadata of the content item to one or more trained machine learning models (e.g., ...). Figure 3B The text recognition module 350 in the first data identifies a first product. Metadata may include text data, such as content item titles, descriptions, explanatory text, comments, live chat, etc. The first data may further include a first confidence value associated with the first product and the content item. The first confidence value may indicate the likelihood that the first product is associated with the first content item, such as the likelihood that the first product appears or is referenced in the content item's metadata. The first data may further include an identifier for a second product and a second confidence value associated with the second product. The first data may include a list of products and associated confidence values.

[0172] At box 512, the processing logic receives second data including a second identifier of the first product. The second identifier has been determined based on image data of the content item (e.g., one or more frames of a video, portions of one or more images, etc.) associated with the content item. The second data also includes a second confidence value associated with the first product and the content item. The confidence value and the identifier can be generated by one or more machine learning models (e.g., Figure 3B The image recognition module 330 in the image data is used for generation. The machine learning model may include a system configured to reduce the dimensionality of the image data. One or more candidate product images may be dimensionality-reduced (e.g., by converting the images into value vectors via a trained machine learning model). The machine learning model may perform operations including comparing the dimensionality-reduced image data from the content item with dimensionality-reduced product images from a data repository, for example, to determine the probability that the image includes a product. The second data may include a list of products (e.g., product identifiers) and a list of confidence values, which at least include the first product. The confidence values ​​may indicate the probability that an associated product appears in the content item, is referenced by the content item, etc.

[0173] In some embodiments, the one or more images of a content item are analyzed for potential products included in one or more images. For example, the presence of a potential product can be confirmed by providing images of the content item (e.g., different frames, additional frames, etc. of a video content item) for further product image detection analysis, or by textual or metadata verification. For example, further analysis for confirming candidate products can be performed after candidate products are found; for example, a search for additional evidence of the identified product can be performed. In some embodiments, textual data and / or metadata associated with the content item can be analyzed for potential / candidate products. The presence of a potential product can be confirmed, for example, by image-based verification, textual verification, etc.

[0174] In some embodiments, one or more timestamps may be further provided to the processing logic, such as timestamps of frames in the video containing detected candidate products, timestamps of narration associated with the video or audio content in which the detected candidate products appear, etc. The processing logic may utilize the timestamps for further analysis to adjust the metadata of content items, generate UI elements associated with the presentation of content items, etc.

[0175] At box 514, the processing logic feeds the first and second data to a trained machine learning model. This trained machine learning model can be a fusion model. One or more lists of products with associated confidence values ​​can be provided to this trained machine learning model.

[0176] At box 516, the processing logic receives a third confidence value associated with the first product from a trained machine learning model. In some embodiments, the processing logic may receive a list of confidence values ​​associated with a list of products including the first product.

[0177] At box 518, the processing logic adjusts the metadata associated with the content item based on a third confidence value. In some embodiments, adjusting the metadata may include adding one or more connections between the content item and the product to the metadata. For example, adjusting the metadata may include adding indications that a particular product is associated with the content item, displayed in the content item, included in the content item, advertised by the content item, etc. Adjusting the metadata may include adjusting the description of the content item to include, for example, one or more references to the product that was incorrectly transcribed during description generation.

[0178] Figure 5C This is a flowchart of a method 500C for training a machine learning model associated with content item product pairings, according to some embodiments. In some embodiments, the machine learning model trained using method 500C may be a fusion model. Similar methods can be used to train different models associated with media item product pairings, such as image recognition models, image verification models, text recognition models, caption update models, etc.

[0179] At box 520, the processing logic receives product image data associated with multiple content items. The product image data may include data that associates products with content items, the association being derived from one or more images (e.g., frames of a video). The product image data includes indications of one or more products (e.g., potential products, candidate products) detected (e.g., determined) in the images, and one or more product image confidence values.

[0180] At box 522, the processing logic receives product text data associated with multiple content items. The product text data may include data that associates products with content items. Associations can be derived from text associated with content items (e.g., metadata associated with content items). The product text data includes indications of one or more products detected in the text (as associated with the content items) and one or more product text confidence values.

[0181] Data received by the processing logic at blocks 520 and 522 (or obtained in some embodiments) can be used as training input for training the fusion model. Training a machine learning model to perform different functions may include the processing logic receiving different data as training input.

[0182] At box 524, the processing logic receives data indicating the products included in multiple content items. For example, each of the multiple content items used to train the model (e.g., data associated with the content item can be used to train the model) may include a list of associated products, such as those tagged by one or more users, tagged by content creators, etc. The data received by the processing logic at box 524 can be used as the target output for training the fusion model. Training machine learning models to perform different functions may include the processing logic receiving different data as target outputs.

[0183] At box 526, the processing logic provides product image data and product text data as training input to a machine learning model. The processing logic can provide different types of data to train different machine learning models. In some embodiments, a machine learning model for frame selection can be trained by providing video frames as training input. In some embodiments, a machine learning model for object detection can be trained by providing images (potentially including products) as training input. A machine learning model for embedding can be trained by providing one or more images of an object (e.g., a product) as training input. In some embodiments, a text parsing model can be trained by providing text associated with content items (e.g., metadata) as training input. In some embodiments, machine-generated narration can be provided as training input to a model configured to correct narration.

[0184] At box 528, the processing logic provides data indicating the products included in multiple content items (e.g., a list of products included in each of the multiple content items) to a machine learning model as the target output. The processing logic can provide different types of data to train different machine learning models. In some embodiments, a machine learning model for frame selection can be trained by providing data indicating which frames of one or more videos include products as the target output. In some embodiments, a machine learning model for object detection can be trained by providing object labels in an image provided to the model as the target output. In some embodiments, a text parsing model can be trained by providing content items referenced by the text of the content items as the target output. In some embodiments, corrected narration (e.g., including one or more products) can be provided as the target output to a model configured to correct narration. In some embodiments, no target output is provided to train the machine learning model (e.g., an unsupervised machine learning model).

[0185] Figure 5DThis is a flowchart of a method 500D for adjusting metadata associated with a content item, according to some embodiments. At block 530, processing logic obtains first metadata associated with the content item. The metadata may include text data. The metadata may include the content item title, description, explanatory text, comments, live chat, etc. At block 531, the processing logic provides the first metadata to a first model. In some embodiments, the model is a trained machine learning model. In some embodiments, the model is a product detection model, for example, the model is configured to receive metadata and (e.g., given the metadata) generate an indication of a product associated with the content item.

[0186] At box 532, the processing logic obtains a first product identifier based on first metadata and a first confidence value associated with the first product identifier as output of the first model. The product identifier can be an ID number, indicator, product name, or any data that (uniquely) distinguishes a product. The first product identifier can identify the first product. In some embodiments, the processing logic may obtain a list of products (e.g., candidate products, potential products) and associated confidence values.

[0187] At box 533, the processing logic obtains image data for the content item. In some embodiments, the image data may include one or more frames of a video or be extracted from one or more frames of a video. In some embodiments, the image data may be obtained from an object detection model. In some embodiments, the image data may include one or more products associated with the content item.

[0188] At box 534, the processing logic provides image data to a second model. In some embodiments, the second model is a machine learning model. In some embodiments, the second model is a model configured to identify a product from an image. In some embodiments, the second model is a model configured to confirm the presence of a product identified from an image. In some embodiments, the second model may reduce the dimensionality of the provided image data. In some embodiments, the second model may compare the reduced-dimensional image data with second reduced-dimensional image data (e.g., retrieved from a data repository, output by a machine learning model, etc.).

[0189] At box 535, the processing logic obtains a second product identifier based on image data as output of the second model and a second confidence value associated with the second product identifier. In some embodiments, the second product identifier indicates a second product. In some embodiments, the second product is the same as the first product. In some embodiments, the processing logic may obtain a list of products and associated confidence values.

[0190] At box 536, the processing logic provides data including a first product identifier, a first confidence value, a second product identifier, and a second confidence value as input to the third model. The third model can be a fusion model.

[0191] At block 537, the processing logic obtains a third product identifier and a third confidence value as outputs of the third model. In some embodiments, the third confidence value may indicate the probability that the product indicated by the third product identifier is associated with a content item (e.g., exists in the content item). In some embodiments, the third model may output a list of products and associated confidence values. In some embodiments, the third product identifier identifies a third product. In some embodiments, the third product is the same as the second product. In some embodiments, the third product is the same as the first product. In some embodiments, the first, second, and third products are all the same product.

[0192] At box 538, the processing logic adjusts the second metadata associated with the content item based on a third product identifier and a third confidence value. Adjusting the metadata may include supplementing it with one or more product associations—e.g., an indication of the associated product. Adjusting the metadata may include updating the captions to include, for example, products that may have been incorrectly transcribed (e.g., incorrectly transcribed by a machine-generated captioning model). In some embodiments, the processing logic may further receive one or more timestamps associated with the content item and one or more products (e.g., the time when a product was detected in an image of a video). Updating the metadata may include adding an indication of the time when the product was found in the content item to the metadata.

[0193] Figure 5E This is a flowchart of a method 500E for presenting UI elements associated with one or more products, according to some embodiments. At block 540, processing logic (e.g., for a user device, client device, etc.) presents the UI. The UI includes one or more graphical representations of one or more content items (e.g., videos). The graphical representation of a content item (e.g., a video thumbnail) can be selected to initiate the presentation of the associated content item. The one or more graphical representations of the content item can be displayed together with UI elements associated with one or more products. Each graphical representation of a corresponding content item can be displayed together with a UI element associated with one or more products. The UI element can be presented / displayed in a collapsed state, such as a collapsed default state. The UI element includes information identifying multiple products covered by the corresponding content item. The UI element can identify one or more products associated with the content item. The UI element can identify one or more products (e.g., covered in a video) associated with the content item (e.g., via name, image, etc.).

[0194] The UI can present optional graphical representations of content items. These content items can be part of a homepage feed, provided in response to a search, part of a watchlist, part of a playlist, part of a shopping feed, etc. In some embodiments, UI elements can be overlaid on top of and / or in front of one or more other elements of the UI. For example, (e.g., in a collapsed state) UI elements can be overlaid on the graphical representation of a content item, or overlaid on the content item (e.g., while the content item is being presented), etc.

[0195] At box 542, in response to user interaction with a UI element in a collapsed state, the processing logic continues to facilitate the rendering of the corresponding video's graphical representation while simultaneously changing the UI element's presentation from a collapsed state to an expanded state. Interactions with UI elements can include selecting the UI element. Interactions with UI elements can include hovering over the UI element (e.g., placing the cursor on the UI element, scrolling to the UI element and pausing scrolling, etc.). A UI element in an expanded state can include multiple visual components. Each visual component can be associated with one of multiple products. Visual components can include images, descriptions, prices, timestamps, etc., associated with various products.

[0196] In some embodiments (e.g., in an expanded state), a UI element may include multiple tabs. For example, a UI element may include tabs for products, tabs for chapters or sections of content items, etc. For content items with an associated product, the UI element may display / open the product's tab by default. The UI element may display the product's tab by default in response to user actions and / or history.

[0197] At box 544, in response to a user selection of one of a plurality of visual components in an expanded UI element, the processing logic initiates the rendering of a corresponding content item covering the product associated with the selected visual component. The processing logic may also initiate playback of a video covering the product associated with the selected visual component. Furthermore, the processing logic may initiate the rendering of the portion of the content item associated with the product of the selected visual component (e.g., initiating playback of the portion of a video associated with the product of the selected visual component).

[0198] In some embodiments, interacting with a UI element can cause the UI element to change to a state focused on the product. The product-focused state can display additional information, details, etc., about one or more products. Interacting with a UI element in a collapsed state can cause the UI element to change to a product-focused state. Interacting with a UI element in an expanded state (e.g., interacting with a visual component of a UI element associated with a product) can cause the UI element to change to a product-focused state.

[0199] In some embodiments, interacting with a UI element can cause the UI element's presentation to change to a transactional state. Interacting with a UI element in a collapsed state can cause the UI element's presentation to change to a transactional state. Interacting with a UI element in an expanded state (e.g., selecting a component associated with a product) can cause the UI element's presentation to change to a transactional state. Interacting with a UI element in a product-focused state can cause the UI element's presentation to change to a transactional state. The determination of whether the selection or interaction with a UI element, UI element component, etc., causes a transition to a transactional state can be performed based on user history, user preferences, content items, content item feeds (e.g., search results, viewing feeds, etc.).

[0200] In some embodiments, UI elements can be overlaid on presented content items. For example, when a video is playing, or when the video is showing one or more products, UI elements identifying one or more products can be overlaid on the video. The selection of the overlaid UI elements can cause additional UI elements to be displayed, cause the overlaid UI elements to be modified to different states, or cause individual UI elements to be modified to different states, etc. The overlaid UI elements can be in a collapsed state, an expanded state, a product-focused state, a transaction state, etc. The presence and / or position of the overlaid UI elements can be determined by one or more trained machine learning models, for example, configured to detect one or more products.

[0201] Figure 5F This is a flowchart of a method 500F for instructing a device to present one or more UI elements associated with a product, according to some embodiments. At block 550, processing logic provides the device with a UI comprising one or more graphical representations of one or more content items. The graphical representations are provided for display / presentation by the device's UI. Each graphical representation of a corresponding content item is selectable to initiate the presentation of that content item. Content items may include videos. Content items may include live video. Graphical representations may be provided in response to a request from the device. Graphical representations may include home feeds, viewing feeds, playlists, search results lists, shopping feeds, etc. Instructions sent to the device (e.g., instructions associated with any step of method 500F), UI elements sent to the device, etc., may be determined / selected based on obtained user history. User history may include historical interactions and / or selections of content items that include content items with associated products. User history may include historical interactions and / or selections of UI elements or components of UI elements associated with a product. User history may include one or more searches by the user, such as searches including product names. Instructions may be provided to the device in response to the processing logic receiving the user's history.

[0202] One or more graphical representations of a content item are displayed together with a UI element in a collapsed state. In some embodiments, each graphical representation is displayed together with a UI element in a collapsed state. In some embodiments, a subgroup of graphical representations is displayed together with a UI element in a collapsed state. The UI element in a collapsed state is presented / displayed together with a first graphical representation of the first content item. The UI element includes information identifying multiple products covered by the first content item. The UI element can identify how many products are associated with the content item, can identify one or more products by name, can identify the category or classification of the products covered by the content item, etc. In some embodiments, multiple products are obtained as output from one or more trained machine learning models. The trained machine learning models can be combined with... Figure 3B The models described are similar.

[0203] At box 554, in response to receiving an instruction to interact with a UI element in a collapsed state, the processing logic causes the device to modify the presentation of the UI element. The presentation of the UI element can be changed from a collapsed state to an expanded state. A UI element in an expanded state can include multiple visual components, each associated with one of multiple products. Visual components can include images, names, descriptions, prices, timestamps, etc.

[0204] At box 556, in response to receiving an instruction for user selection of one of a plurality of visual components of a UI element in an expanded state, processing logic facilitates the presentation of a first content item. The processing logic may provide instructions to facilitate the presentation of a portion of the first content item associated with a first product, such as a product associated with one of the plurality of visual components. The processing logic may provide instructions for displaying a product-associated portion of a video (e.g., for starting video playback from a selected point in the video based on a timestamp associated with the product).

[0205] In some embodiments, the processing logic may further provide the device with instructions to modify the presentation of a UI element to focus on a product state. For example, after selecting a visual component of a UI element in an expanded state, the UI element can be modified to focus on a product state. The product-focused state may include additional details about one or more products that are covered by, included in, or associated with a content item.

[0206] In some embodiments, the processing logic may further provide the device with instructions to modify the presentation of a UI element to a transaction state. A transaction state may facilitate a user initiating a transaction associated with a product (e.g., purchasing a product). A transaction state may be presented in response to user actions, user history, user selection of one or more UI elements, etc. UI elements in a transaction state may include one or more components that facilitate a transaction associated with one or more products.

[0207] Figure 6 This is a block diagram illustrating a computer system 600 according to some embodiments. In some embodiments, the computer system 600 may be connected to other computer systems (e.g., via a network such as a local area network (LAN), intranet, extranet, or the Internet). The computer system 600 may operate as a server or client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 600 may be provided by a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, web appliance, server, network router, switch, or bridge, or any device capable of executing a set of instructions (sequential instructions or other instructions) specifying the actions to be taken by that device. Further, the term "computer" should include any collection of computers that individually or jointly execute a set (or more sets) of instructions to perform any or more of the methods described herein.

[0208] In a further aspect, the computer system 600 may include a processing device 602, a volatile memory 604 (e.g., random access memory (RAM)), a non-volatile memory 606 (e.g., read-only memory (ROM) or electrically erasable programmable ROM (EEPROM)), and a data storage device 618, which can communicate with each other via a bus 608.

[0209] The processing device 602 may be provided by one or more processors, such as general-purpose processors (e.g., complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, microprocessors that implement other types of instruction sets, or microprocessors that implement a combination of multiple types of instruction sets) or special-purpose processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or network processors).

[0210] The computer system 600 may further include (e.g., a network interface device 622 coupled to the network 674). The computer system 600 may also include a video display unit 610 (e.g., an LCD), an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse), and a signal generation device 620.

[0211] In some embodiments, the data storage device 618 may include a non-transitory computer-readable storage medium 624 (e.g., a non-transitory machine-readable medium) on which instructions 626 encoding any one or more methods or functions described herein may be stored, the instructions including instructions on... Figure 1 The components (e.g., content providing platform 120, other platforms of content platform system 102, communication application 115, model 190, etc.) are encoded with instructions for implementing the methods described herein.

[0212] During the execution of instructions by the computer system 600, instructions 626 may also reside wholly or partially in the volatile memory 604 and / or the processing device 602, thus the volatile memory 604 and the processing device 602 may also constitute a machine-readable storage medium.

[0213] Although computer-readable storage medium 624 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" should include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of executable instructions. The term "computer-readable storage medium" should also include any tangible medium capable of storing or encoding a set of instructions for execution by a computer, which can cause a computer to perform any or more of the methods described herein. The term "computer-readable storage medium" should include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0214] The methods, components, and features described herein can be implemented using discrete hardware components or integrated into the functionality of other hardware components such as ASICs, FPGAs, DSPs, or similar devices. Alternatively, the methods, components, and features can be implemented using firmware modules or functional circuitry systems within a hardware device. Furthermore, the methods, components, and features can be implemented as any combination of hardware devices and computer program components, or as a computer program.

[0215] Unless otherwise specified, terms such as “receive,” “execute,” “provide,” “obtain,” “cause,” “access,” “determine,” “add,” “use,” “train,” “reduce,” “generate,” and “correct” refer to actions and processes performed or implemented by a computer system that manipulate and transform data represented as physical (electronic) quantities in computer system registers and memories into other data similarly represented as physical quantities in computer system memory or registers or other such information storage, transmission, or display devices. Furthermore, terms such as “first,” “second,” “third,” and “fourth,” as used herein, are intended as markers to distinguish between different elements and may not have ordinal meanings assigned according to their numerical designations.

[0216] The examples described herein also relate to an apparatus for performing the methods described herein. This apparatus may be specifically constructed for performing the methods described herein, or it may comprise a general-purpose computer system selectively programmed by a computer program stored in a computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0217] The methods and illustrative examples described herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used based on the teachings described herein, or more specialized devices can be readily constructed to perform the methods described herein and / or each of their individual functions, routines, subroutines, or operations. Examples of various structures of such systems are illustrated in the foregoing description.

[0218] The above description is intended to be illustrative and not restrictive. Although this disclosure has been described with reference to specific illustrative examples and embodiments, it will be appreciated that this disclosure is not limited to the described examples and embodiments. The scope of this disclosure should be determined by reference to the appended claims and the full scope of the authorized equivalents of the claims.

[0219] Throughout the specification, references to "an implementation" or "implementation" mean that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Therefore, depending on the context, the appearance of the phrase "in an implementation" or "in an implementation" in various places throughout the specification may, but not necessarily, refer to the same implementation. Furthermore, a particular feature, structure, or characteristic may be combined in one or more implementations in any suitable manner.

[0220] With regard to the use of the terms “includes,” “including,” “has,” “contains,” variations thereof, and other similar terms in the Detailed Description or claims, these terms are intended to be inclusive in a manner similar to the term “comprising” as an open-ended transitional term, without excluding any additional or other elements.

[0221] As used herein, the terms “component,” “module,” “system,” etc., are generally intended to refer to computer-related entities, i.e., hardware (e.g., circuitry), software, combinations of hardware and software, or entities associated with an operating machine having one or more specific functionalities. For example, a component can be, but is not limited to, a process, processor, object, executable program, execution thread, program, and / or computer running on a processor (e.g., a digital signal processor). For instance, both an application running on a controller and the controller itself can be components. One or more components may reside within a process and / or execution thread, and components may be confined to a single computer and / or distributed across two or more computers. Furthermore, “apparatus” can appear in the form of: specially designed hardware; general-purpose hardware specialized by software executed thereon that enables the hardware to perform specific functions (e.g., generating points of interest and / or descriptors); software on a computer-readable medium; or combinations thereof.

[0222] The aforementioned systems, circuits, modules, etc., have been described with respect to the interactions between several components and / or blocks. It will be understood that such systems, circuits, components, blocks, etc., may include those components or designated sub-components, some and / or additional components among designated components or sub-components, and according to various permutations and combinations thereof. Sub-components may also be implemented as components communicatively coupled to other components rather than being included within a parent component (hierarchically). Furthermore, it should be noted that one or more components may be combined into a single component providing aggregate functionality or divided into several separate sub-components, and any one or more intermediate layers, such as a management layer, may be provided to communicatively couple to such sub-components to provide integrated functionality. Any component described herein may also interact with one or more other components not specifically described herein but known to those skilled in the art.

[0223] Furthermore, the terms “example” or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the terms “example” or “exemplary” is intended to present concepts in a specific manner. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X adopts A or B” is intended to mean any of the natural inclusive arrangements. That is, “X adopts A or B” is satisfied under any of the foregoing examples if X adopts A; X adopts B; or X adopts both A and B. Additionally, unless otherwise specified or clear from the context as to be directed to the singular form, the articles “a” and “an” used in this application and the appended claims should generally be interpreted as meaning “one or more”.

Claims

1. A method comprising: The processing device obtains first data using a first machine learning model, the first data including (i) a first identifier of a first product determined based on first metadata of a content item and associated with the content item, and (ii) a first confidence value indicating a first probability that the first product is associated with the content item; The processing device obtains second data using a second machine learning model, the second data including (i) a second identifier of the first product determined based on the first image data of the content item and associated with the content item, and (ii) a second confidence value indicating a second probability that the first product is associated with the content item; The processing device provides the first data and the second data as input to a third machine learning model. The first data includes (i) the first identifier of the first product determined based on the first metadata of the content item and (ii) the first confidence value, and the second data includes (i) the second identifier of the first product determined based on the first image data of the content item and (ii) the second confidence value. Obtain the output of the third machine learning model, the output of which includes a third confidence value associated with the first product; and The second metadata associated with the content item is adjusted based on the third confidence value.

2. The method of claim 1, further comprising: The first metadata of the content item is provided as input to the first machine learning model; as well as Obtain the first data as the output of the first machine learning model.

3. The method as described in claim 2, wherein, The first metadata includes at least one of the following: The title of the content item; The description of the content item; or Explanatory terms associated with the content item.

4. The method of claim 1, further comprising: The first image data of the content item is provided as input to the second machine learning model; Obtain the first dimension-reduced data as the output of the second machine learning model; as well as Obtain data from the data repository that is associated with the first product in a second, reduced dimension.

5. The method of claim 4, wherein, The second dimension-reduced data is obtained from the data repository in response to obtaining the first data, and wherein the second data is generated based at least on the first dimension-reduced data and the second dimension-reduced data.

6. The method of claim 4, further comprising: The second image data is fed into the fourth machine learning model, and The third identifier of the first product is obtained from the fourth machine learning model, wherein the second dimension-reduced data is obtained from the data repository in response to obtaining the third identifier of the first product, and wherein the second data is generated based at least on the first dimension-reduced data and the second dimension-reduced data.

7. The method of claim 1, wherein, The content item is a video, and the first data further includes an indication of timestamps of one or more frames of the video associated with the product, and adjusting the second metadata includes including the indication of the first product and the indication of the timestamps in the second metadata.

8. The method of claim 1, wherein, Adjusting the second metadata includes adjusting the descriptive text associated with the product.

9. The method of claim 1, further comprising training the third machine learning model, wherein, Training the third machine learning model includes: Receive image-based product data associated with multiple content items, wherein the image-based product data includes indications of one or more products detected in an image and one or more product image confidence values; Receive metadata-based product data associated with the plurality of content items, wherein the metadata-based product data includes indications of one or more products detected in the text and one or more confidence values; Receive data indicating the products included in the plurality of content items; The image-based product data and the metadata-based product data are provided as training inputs to the third machine learning model; and The data indicating the products included in the plurality of content items will be provided as the target output to the third machine learning model.

10. The method of claim 1, further comprising: Receive third data including a third identifier of a first product category associated with the content item; as well as The third data is provided to the third machine learning model, wherein the third confidence value is generated based on the first data, the second data, and the third data.

11. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a processing device to perform operations including: First data is obtained using a first machine learning model, the first data including (i) a first identifier of a first product determined in association with the content item based on the first metadata of the content item, and (ii) a first confidence value indicating a first probability that the first product is associated with the content item; The second data is obtained using a second machine learning model, the second data including (i) a second identifier of the first product determined based on the first image data of the content item and associated with the content item, and (ii) a second confidence value indicating a second probability that the first product is associated with the content item; The first data and the second data are provided as input to a third machine learning model. The first data includes (i) the first identifier of the first product determined based on the first metadata of the content item and (ii) the first confidence value, and the second data includes (i) the second identifier of the first product determined based on the first image data of the content item and (ii) the second confidence value. Obtain the output of the third machine learning model, the output of which includes a third confidence value associated with the first product; and The second metadata associated with the content item is adjusted based on the third confidence value.

12. The non-transitory machine-readable storage medium of claim 11, wherein, The operation further includes: The first image data of the content item is provided as input to the second machine learning model; Obtain the first dimension-reduced data as the output of the second machine learning model; and Obtain second-dimensional reduced data associated with the first product from the data repository, wherein the second data is based at least on the first-dimensional reduced data and the second-dimensional reduced data.

13. The non-transitory machine-readable storage medium of claim 11, wherein, The content item is a video, and the first data further includes an indication of timestamps of one or more frames of the video associated with the product, and adjusting the second metadata includes including the indication of the first product and the indication of the timestamps in the second metadata.

14. The non-transitory machine-readable storage medium of claim 11, wherein, The operation further includes: Receive third data including a third identifier of a first product category associated with the content item; and The third data is provided to the third machine learning model, wherein the third confidence value is generated based on the first data, the second data, and the third data.

15. A system comprising: Memory; as well as Processing device, coupled to the memory, for performing operations including: First data is obtained using a first machine learning model, the first data including (i) a first identifier of a first product determined in association with the content item based on the first metadata of the content item, and (ii) a first confidence value indicating a first probability that the first product is associated with the content item; The second data is obtained using a second machine learning model, the second data including (i) a second identifier of the first product determined based on the first image data of the content item and associated with the content item, and (ii) a second confidence value indicating a second probability that the first product is associated with the content item; The first data and the second data are provided as input to a third machine learning model. The first data includes (i) the first identifier of the first product determined based on the first metadata of the content item and (ii) the first confidence value, and the second data includes (i) the second identifier of the first product determined based on the first image data of the content item and (ii) the second confidence value. Obtain the output of the third machine learning model, the output of which includes a third confidence value associated with the first product; and The second metadata associated with the content item is adjusted based on the third confidence value.

16. The system of claim 15, wherein the operation further comprises: The first metadata of the content item is provided as input to the first machine learning model; as well as Obtain the first data as the output of the first machine learning model.

17. The system according to claim 16, wherein, The first metadata includes at least one of the following: The title of the content item; The description of the content item; or Explanatory terms associated with the content item.

18. The system of claim 15, wherein the operation further comprises: The first image data of the content item is provided as input to the second machine learning model; Obtain the first dimension-reduced data as the output of the second machine learning model; as well as Obtain data from the data repository that is associated with the first product in a second, reduced dimension.

19. The system according to claim 18, wherein, The second dimension-reduced data is obtained from the data repository in response to obtaining the first data, and wherein the second data is generated based at least on the first dimension-reduced data and the second dimension-reduced data.

20. The system of claim 18, wherein the operation further comprises: The second image data is fed into the fourth machine learning model, and The third identifier of the first product is obtained from the fourth machine learning model, wherein the second dimension-reduced data is obtained from the data repository in response to obtaining the third identifier of the first product, and wherein the second data is generated based at least on the first dimension-reduced data and the second dimension-reduced data.

Images