Media item and product pairing

Abstract

A method includes presenting a user interface (UI) including one or more graphical representations of one or more videos. Each graphical representation of the respective video may be selected to initiate playback of the respective video. The graphical representation is displayed in a UI element in a folded state. The UI element includes information identifying a plurality of products encompassed by the respective video. The method further includes modifying the presentation of the UI element from the folded state to the unfolded state. The UI element in the expanded state includes a plurality of visual components, each visual component associated with one of the plurality of products. The method further includes initiating presentation of the video after the user selects one of the plurality of visual components.

Description

Technical Field

[0001] Aspects and implementations of the present disclosure relate to methods and systems for facilitating pairing of media items and associated objects, and more particularly, to systems for facilitating pairing of media items and products. Background Art

[0002] A platform (e.g., a content sharing platform) can transmit (e.g., stream) media items to client devices connected to the platform via a network. Different types of client devices can be optimized for different tasks, preferred by users for different tasks, etc. A media item can contain references to one or more products. Summary of the invention

[0003] The following summary of the invention is a simplified overview of the present disclosure in order to provide a basic understanding of some aspects of the present disclosure. This summary of the invention is not a comprehensive review of the present disclosure. It is neither intended to identify the key or important elements of the present disclosure, nor to indicate any scope of a specific implementation of the present disclosure or any scope of the claims. Its only purpose is to present some concepts of the present disclosure in a simplified form as a preface to a more detailed description presented later.

[0004] Disclosed are systems and methods for promoting user participation in one or more interactive activities associated with one or more products of a media item. In some implementations, the method includes presenting a user interface (UI) including one or more graphical representations of one or more videos. Each graphical representation of a corresponding video can be selected to start the playback of the corresponding video and displayed with a UI element in a collapsed state. The UI element in the collapsed state includes information identifying multiple products covered by the corresponding video. The method further includes continuing to present the graphical representation of the corresponding video while modifying the presentation of the UI element from a collapsed state to an expanded state. Modification of the presentation of the UI element is performed in response to a user's interaction with the UI element in the collapsed state. The UI element in the expanded state includes multiple visual components, each of which is associated with one of the multiple products. The method further includes, in response to a user's selection of a corresponding one of the multiple visual components of the UI element in the expanded state, starting the playback of a video covering a product associated with the selected visual component.

[0005] In some embodiments, the presentation of the UI element can be modified to a product focus state. The product focus state can include detailed information about one or more products associated with the content item. In response to a user's selection of a UI element or UI element component, the presentation of the UI element can be modified to a product focus state. In response to a user selecting one of the multiple visual components of the UI element in the expanded state, the presentation of the UI element can be modified to a product focus state.

[0006] In some embodiments, the presentation of UI elements and graphical representations may be performed in response to a user device requesting a content item home feed for display. The presentation of UI elements and graphical representations may be performed in response to a search query of a user device. The presentation of UI elements and graphical representations may be performed in response to a user request to view a content item.

[0007] The UI element in the expanded state may include one or more tabs. A first tab may be associated with multiple products. A second tab may be associated with multiple parts (e.g., chapters) of a corresponding content item. The first tab may be presented by default.

[0008] In some embodiments, the presentation of the UI element can be modified to a transaction state. The modification to the transaction state can be performed in response to a user's selection of the UI element or a component of the UI element. The UI element in the transaction state can facilitate a transaction associated with one of the multiple products.

[0009] One or more UI elements may be superimposed. A UI element may be superimposed on a content item being presented. A UI element may be superimposed on a graphical representation of a content item. User interaction with a UI element may include staying on a UI element.

[0010] In another aspect, a method includes providing one or more graphical representations of one or more content items to a device for display by a UI of the device. Each graphical representation of a corresponding content item may be selected to start the presentation of the corresponding content item. The method further includes providing instructions to the device to display a UI element in a folded state and a first graphical representation of a first content item. The UI element may include information identifying multiple products covered by the first content item. The method further includes providing instructions to the device to modify the presentation of the UI element. Instructions may be provided in response to receiving an indication that a user interacts with the UI element. The instruction may facilitate the conversion of the UI element from a folded state to an expanded state. The UI element in the expanded state includes multiple visual components. Each visual component is associated with one of the multiple products. The method further includes providing instructions to the device to facilitate the presentation of the content item. In response to receiving an indication that a user selects one of the multiple visual components of the UI element in the expanded state, instructions to facilitate the presentation of the content item may be provided.

[0011] In some embodiments, the method includes providing instructions to the device to facilitate modifying the presentation of the UI element from an expanded state to a product focused state. The modification of the UI element may be performed in response to receiving an indication that a user selects one of a plurality of visual components of the UI element in the expanded state. The UI element in the product focused state may include detailed information about a product associated with the selected visual component.

[0012] In some embodiments, the method includes obtaining identifiers of a plurality of products covered by the first content item as output from a trained machine learning model. The trained machine learning model receives as input data associated with the first content item.

[0013] In some embodiments, the method includes providing instructions to the device to facilitate modifying the presentation of the UI element to a transaction state. The transaction state includes one or more components that facilitate a transaction associated with one of a plurality of products associated with the content item. The instructions to facilitate modifying the presentation of the UI element to the transaction state may be provided in response to a user selecting a component of the UI element associated with the product.

[0014] In some embodiments, the instruction to promote the presentation of the content item may include an indication of a portion of the content item to be presented. The portion of the content item may cover a product associated with a visual component in a plurality of visual components in a plurality of products. In some embodiments, in response to obtaining the user's history, one or more instructions may be provided to the device. The user's history may include one or more interactions with the content item associated with the product. In some embodiments, the content item is a live video. In some embodiments, the content item is a short video.

[0015] In another aspect, a non-transitory machine-readable storage medium stores instructions that, when executed, cause a processing device to perform operations including presenting a UI. The UI includes one or more graphical representations of one or more videos. Each graphical representation of a corresponding video can be selected to start playback of the corresponding video and displayed with a UI element in a folded state. The UI element includes information identifying a plurality of products covered by the corresponding video. The operation further includes continuing to present the graphical representation of the corresponding video while modifying the presentation of the UI element from a folded state to an expanded state. The modification can be performed in response to a user's interaction with the UI element in the folded state. The UI element in the expanded state can include multiple visual components. Each visual component can be associated with one of the multiple products. The operation further includes starting playback of a corresponding video covering a product associated with the selected visual component. Playback can be initiated in response to a user selecting a selected visual component from the multiple visual components.

[0016] In some embodiments, the operation includes modifying the presentation of the UI element from an expanded state to a product focused state. The modification may be performed in response to a user selecting one of a plurality of visual components of the UI element in the expanded state. The UI element in the product focused state may include detailed information about a product associated with the selected visual component.

[0017] In some embodiments, the operation further includes modifying the presentation of the UI element to a transaction state. Modifying the presentation of the UI element may be performed in response to a user selecting a component of the UI element associated with the product. The UI element in the transaction state facilitates a transaction associated with the product.

[0018] In some embodiments, the UI element may include a tab. The UI element may include a first tab associated with a plurality of products. The UI element may include a second tab associated with a plurality of portions of a corresponding video. The content of the first tab may be presented by default. In some embodiments, starting playback of the corresponding video includes presenting a portion of the video depicting the product. The product may be associated with a selected visual component of the UI element.

[0019] Where appropriate, optional features of one aspect may be combined with other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Aspects and implementations of the present disclosure will be more fully understood from the detailed description given below and from the drawings of various aspects and implementations of the present disclosure, however, they should not be construed as limiting the present disclosure to specific aspects or implementations, but are only for explanation and understanding.

[0021] Figure 1 An example system architecture for providing related and associated product information in accordance with some embodiments is shown.

[0022] Figure 2 is a block diagram of a system 200 including a dataset generator for creating a dataset for one or more models, according to some embodiments.

[0023] Figure 3A is a block diagram illustrating a system for generating output data, such as associations between products and content items, according to some embodiments.

[0024] Figure 3B is a block diagram of an example system for generating data describing an association between a content item and one or more products, according to some embodiments.

[0025] Figure 4A Depicted is an apparatus presenting an example user interface (UI) including a UI element indicating a product associated with a content item in accordance with some embodiments.

[0026] Figure 4B Depicted is an apparatus presenting an example UI including a UI element indicating a product associated with a content item in accordance with some embodiments.

[0027] Figure 4CDepicted is an apparatus presenting an example UI including a UI element presenting a content item and a UI element presenting information about a product associated with the content item in accordance with some embodiments.

[0028] Figure 4D Depicted is an apparatus presenting an example UI including UI elements that present content items and UI elements that facilitate transactions associated with products in accordance with some embodiments.

[0029] Figure 4E Depicted is an example device having a UI element overlaid on a content presentation element in accordance with some embodiments.

[0030] Figure 5A is a flowchart of a method for generating a data set for a machine learning model according to some embodiments.

[0031] Figure 5B is a flow diagram of a method for updating metadata for a content item according to some embodiments.

[0032] Figure 5C is a flow diagram of a method for training a machine learning model associated with content item and product pairings according to some embodiments.

[0033] Figure 5D is a flow diagram of a method for adjusting metadata associated with a content item according to some embodiments.

[0034] Figure 5E is a flow diagram of a method for presenting UI elements associated with one or more products according to some embodiments.

[0035] Fig. 5F is a flow diagram of a method for directing a device to present one or more UI elements associated with a product according to some embodiments.

[0036] Figure 6 is a block diagram illustrating an exemplary computer system according to an implementation of the present disclosure. DETAILED DESCRIPTION

[0037] Aspects of the present disclosure relate to methods and systems for facilitating the pairing of media items (e.g., content items, content, etc.) with associated products. A platform (e.g., a content sharing platform, etc.) can enable a user to access media items (e.g., video items, audio items, etc.) hosted by the platform (e.g., via a client device connected to the platform). The platform can provide access to media items to a user's client device via a network (e.g., the Internet) (e.g., by transmitting the media items to the user's client device, etc.). The media / content items can have one or more additional associated activities that can deepen the user's interaction with the content items. For example, an interactive activity can include a comment area for a content item, a real-time chat associated with a content item, etc. Some content items can be related to one or more products. For example, a product can be displayed as part of a content item, a product can be evaluated in a content item, a content item can be associated with one or more products (e.g., sponsored by a company related thereto), and the like.

[0038] In conventional systems, identifying products associated with a content item can be difficult, inconvenient, time consuming, etc. It can be difficult for a content platform (e.g., a platform that provides content for presentation to a user) to identify products associated with a content item. In some systems, a content creator can identify one or more products associated with a content item, a content channel, a list of content items, etc. In some systems, a content creator can include information about one or more products in a content item, such as a photo of the item, an item that appears in a video, etc. In some systems, a content creator can include information about one or more products in one or more fields associated with a content item, such as a content item title, a content item description, a content item comment (e.g., a pinned comment), etc.

[0039] In conventional systems, a user may not be able to easily determine whether a content item has one or more associated products. A content item may not have an indicator of associated products. A user may need to be presented with a content item (e.g., watch a video) to determine whether a content item has one or more associated products.

[0040] In conventional systems, it may be difficult for a user to identify products associated with a content item. There may be no direct indication that a content item has associated products (e.g., when selecting a content item to consume, selecting a content item to view, etc.). Product information may be difficult to find, e.g., scattered among different fields such as a content item title, a content item description, a comment area, etc. Product information may be included in a content item, e.g., a video may include audio describing one or more products, an image item may include images of one or more products, etc. Extracting this information from a content item may be difficult, time consuming, error prone, etc.

[0041] In conventional systems, receiving additional information about products featured in content items can be difficult, time-consuming, and the like. In some systems, products may appear in content items (e.g., may appear in videos). Users may not be given additional information (e.g., product names, merchant names, etc.), but may perform searches independently of the content platform to learn more about the product. Names and / or merchants associated with the product may be provided (e.g., in the description or title of the content item). Users may perform searches (e.g., independently of the content platform) to learn more about the product, such as product variations, related products, availability, and prices, etc. Guidance to promote a better understanding of the product (e.g., guidance to users, guidance to processing devices in the form of links to merchant websites, etc.) may be provided. Users may receive additional information from a separate source (e.g., a website) independent of the content platform.

[0042] In conventional systems, information about products included in content items may become outdated. For example, within a content item or associated fields (e.g., content item title, content item description, content item comments, etc.), a content creator may include additional information about one or more products, such as price information, merchant information, availability information, alternate versions or variations information, etc. Some information included in a content item or associated fields may not be updated as the information changes, may depend on updates by the content creator, etc.

[0043] In conventional systems, there may be obstacles that make it difficult, time-consuming, cumbersome, etc. for a user to purchase one or more items associated with a content item. A user may search for a product, search for a merchant that stocks or sells the product, etc. In some embodiments, a content item or an associated field may include instructions for purchasing an item (e.g., a description of a content item may include one or more links to products associated with the content item). A user may be directed to another platform independent of the content platform to complete the purchase of one or more products.

[0044] The user may spend a lot of time and computing resources to find information about the products covered by the content items. For example, a video can be very long and may focus on the products of interest near the end. The user may spend a lot of time consuming the video to obtain accurate information about the products of interest, resulting in increased use of the computing resources of the client device. In addition, the computing resources of the client device that enables the user to consume the media items may not be available for other processes, which may reduce overall efficiency and increase the overall latency of the client device.

[0045] Aspects of the present disclosure can solve one or more of these defects of conventional systems. In some embodiments, aspects of the present disclosure can enable automatic identification of products highlighted and / or included in content items. In some embodiments, aspects of the present disclosure can enable the use of models to identify products from text associated with content items. Text can include the title of the content item, the description of the content item, subtitles associated with the content item, etc. Subtitles can be machine-generated subtitles, such as subtitles generated by a speech-to-text model for a video or audio content item. The model can be a machine learning model. The model can output a confidence value indicating that a content item includes the possibility of a product.

[0046] In some embodiments, aspects of the present disclosure may enable the use of a model to identify a product from an image of a content item. The content item may be or include a picture. The content item may be or include a video. One or more images of the content item may be provided to a model configured to identify a product from an image. The model may reduce the dimensionality of the image. The model may search for similar images associated with the product. The model may search for similar reduced dimensionality images in the reduced dimensionality space. The model may be a machine learning model. The model may output a confidence value indicating the likelihood that the content item includes the product.

[0047] In some embodiments, aspects of the present disclosure enable the use of a model (e.g., a fusion model) to determine whether a product is included in a content item. The fusion model may receive an indication of one or more products detected by a model that receives text associated with the content item as input. The fusion model may receive an indication of a confidence value that the one or more products are included in the text. The fusion model may receive an indication of one or more products detected by a model that receives an image of the content item as input. The fusion model may receive an indication of a confidence value that the one or more products are included in the image. The fusion model may determine the confidence that one or more products appear in the content item and associated data (e.g., title, description, etc.). The fusion model may be a machine learning model.

[0048] In some embodiments, product detection can be used to improve content or associated information (e.g., descriptions, subtitles). The model can be provided with data associated with the content item. For example, the model can be provided with machine-generated subtitles for the content item. The model can identify one or more subtitles that can incorrectly express the product name (e.g., the machine-generated subtitles can include the closest English equivalent to the spoken product name). The model can be provided with one or more images of the content item. The model can determine the likelihood that one or more products are included in the image. Information associated with the content item (e.g., metadata, descriptions, subtitles, etc.) can be updated in view of one or more detected products.

[0049] In some embodiments, aspects of the present disclosure implement indicators of content items with one or more associated products. A list of content items may include one or more indicators indicating that one or more content items in the list include associated products. Indicators may include visual indicators (e.g., a "shopping" symbol or text indicating one or more products displayed in association with a content item), additional fields (e.g., a panel including product information), etc. In some embodiments, a list of content items may be presented to a user via a user interface (UI). UI elements may be associated with a content platform (e.g., may be presented via an application associated with a content providing platform). UI may include elements indicating that a content item is associated with one or more products. In some embodiments, user interaction with UI elements may cause presentation of additional UI elements with additional information, such as a list of products associated with a content item.

[0050] In some embodiments, aspects of the present disclosure enable users to identify products associated with content items. In some embodiments, a list of content items for presentation to a user may include a list of products associated with one or more content items. For example, a list of content items may be displayed to a user via a UI. The UI may include one or more UI elements that present one or more products associated with a content item to the user. The UI element may be provided via an application associated with a content providing platform for the content item. In some embodiments, a UI element that lists products associated with a content item may be presented in response to detecting a user interaction with a UI element indicating that a content item has associated products. In some embodiments, user interaction with a product in the product list may cause the UI to present additional information to the user.

[0051] In some embodiments, aspects of the present disclosure enable additional information about one or more products associated with a content item to be provided to a user. A UI element may be provided to a user, which provides additional information about a product. The UI element may provide information such as changes in a product (e.g., color changes, size changes, etc.), related products, availability, and / or price (e.g., associated with one or more merchants). The UI element may be provided via an application associated with a content item, a content providing platform, etc. The UI element displaying additional product information may be presented in response to a user interaction with another UI element (e.g., selecting a content item to be viewed, selecting a UI element indicating one or more associated products, etc.). In some embodiments, after user interaction with a UI element, another UI element may be provided, such as to facilitate the purchase of a product.

[0052] In some embodiments, aspects of the present disclosure enable automatic updating of information related to one or more products associated with a content item. In some embodiments, a content providing platform associated with a content item may include one or more memory devices containing product data, communicate with the one or more memory devices, be connected to the one or more memory devices, etc. For example, the content platform may maintain and update a database of information associated with the product, and changes made to the database may be reflected in a UI element presented to the user.

[0053] In some embodiments, aspects of the present disclosure implement a simplified purchase process for a user. Upon receiving an indication of a user's intent to purchase a product (e.g., upon user interaction with a UI element associated with a product), a UI element may be presented to the user to facilitate the purchase of the product. In some embodiments, the user may purchase the product via an application associated with a content item, a content providing platform, etc. In some embodiments, the user may be directed to one or more external merchants (e.g., merchant applications, merchant web pages, etc.) to purchase the product.

[0054] In some embodiments, UI elements associated with products may be provided in response to various operations of an application (e.g., an application associated with a content platform). UI elements associated with products may be provided as part of a list of content items customized for a user (e.g., customized for a user account associated with a user). UI elements associated with products may be provided as part of a list of content times associated with previously presented content items, such as a list of next viewings, a list of recommended videos, etc. UI elements associated with products may be provided as part of a list of content items generated in response to a user search. UI elements associated with products may be provided as part of a product-specific list, such as a shopping area of ​​an application associated with a content platform. The inclusion of UI elements associated with products, content items associated with products, the number of UI elements or content items presented associated with products, etc. may be based on multiple indicators. Indicators may include user viewing history, user search history, etc.

[0055] In some embodiments, aspects of the present disclosure can enable quick access to one or more parts of content items associated with a product. For example, a UI element can include a list of products associated with a content item. One or more of the product list can be associated with a part of the content item (e.g., the timestamp of a video content item). After partially interacting with a UI element associated with a product, the content item can present the relevant part of the content item (e.g., a video can start playing the part associated with the content item in the video). In some embodiments, the list of products associated with a content item can be updated when the content item is presented. For example, a list of products associated with a video can be rearranged when the video is played. For example, a product currently highlighted by the video can be at the top of the product list, and the product currently on the screen can be grouped together in the product presentation UI element, and the like.

[0056] Aspects of the present disclosure may provide technical advantages over previous solutions. Aspects of the present disclosure may enable automatic product detection in content items. This may improve the experience of content creators by automatically associating one or more products with content items (e.g., eliminating the burden on creators to associate products with content items). Automatically generated product associations may have improved accuracy due to the use of multiple sources (e.g., text and image searches for products), fusion models, etc. Model-based product detection may be used to improve content items, associated data, etc., for example, machine-generated subtitles or descriptions of content items may be improved using object detection. More accurate subtitles may improve the user's experience when consuming content items. These improvements may reduce the time required for content creators to generate accurate content, may reduce the time it takes for users to find content items and / or products of interest, may improve the accuracy of machine-generated information associated with content items, etc. Therefore, computing resources at client devices associated with content creators, content viewers, and / or platforms are reduced and may be used for other processes, which improves the overall efficiency of the system and reduces the overall latency of the system.

[0057] Aspects of the present disclosure can improve the user's experience when searching for content items, viewing content items, scrolling through a list of content items, etc. A user can be able to identify a content item with an associated product, for example, from a list of content items, without presenting the content item. A seamless method can be provided to the user to increase interaction with products associated with a content item, for example, interaction with a UI element indicating the presence of a product associated with a content item can cause the presentation of a content item with additional information about one or more products, and further interaction can facilitate the purchase of one or more products, etc. The presentation of one or more UI elements can streamline the user's experience. A user can easily retrieve additional information, for example, in an application associated with a content platform / content item. A user can more easily purchase products associated with a content item. A user can be directed to the relevant part of a content item based on an interest expressed in a product associated with a content item. A user can easily view price information, availability information, product changes, related products, etc. within the context of a single application. Such an implementation can save user time and energy, simplify the shopping and / or purchasing process, simplify product research, evaluation and / or selection processes, and the like.

[0058] Figure 1 An example system architecture 100 for providing content and associated product information according to some embodiments is shown. The system architecture 100 includes a client device 110, one or more networks 105, a content platform system 102, and a product identification system 175. The content platform system 102 includes one or more server machines 106, one or more data repositories 140, and may include various platforms designed to perform tasks (e.g., tasks associated with content delivery). The platforms of the content platform system 102 may be hosted by one or more server machines 106. The platforms of the content platform system 102 may include and / or be hosted on one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.) and one or more data repositories (e.g., hard disks, memories, and databases), and may be coupled to one or more networks 105. In some embodiments, components of the content platform system 102 (e.g., server machines 106, data repositories 140, hardware associated with one or more platforms, etc.) may be directly connected to one or more networks 105. In some embodiments, one or more components of the content platform system 102 may access the network 105 via another device (e.g., a hub, a switch, etc.). In some embodiments, one or more components of the content platform system 102 may directly communicate with the network 105. Figure 1170 and / or 180). The data repository 140 may be included in one or more server machines 106, including external data storage devices, etc. The platforms of the content platform system 102 may include an advertising platform 165, a social network platform 160, a recommendation platform 157, a search platform 145, and a content provision platform 120. The product identification system 175 includes a server machine 170, a server machine 180, and a set of models 190. The product identification system 175 may include additional devices, such as data repositories, additional servers, etc. Multiple operations of the product identification system 175 may be performed by a single physical or virtual device.

[0059] The one or more networks 105 may include one or more public networks (e.g., the Internet), one or more private networks (e.g., a local area network (LAN), a wide area network (WAN), one or more wired networks (e.g., Ethernet), one or more wireless networks (e.g., 802.11 networks), one or more cellular networks (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or combinations thereof. In one embodiment, some components of the architecture 100 are not directly connected to each other. In one implementation, the system architecture 100 includes separate networks 105.

[0060] One or more data repositories 140 may reside in a memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, etc., and may be part of one or more database systems, one or more file systems, or other types of components or devices capable of storing data. One or more data repositories 140 may include multiple storage components (e.g., multiple drives or multiple databases) that may also span multiple computing devices (e.g., multiple server computers). A data repository may be a persistent storage device capable of storing data. A persistent storage device may be a local storage unit or a remote storage unit, an electronic storage unit (e.g., a main memory), or a similar storage unit. A persistent storage device may be a monolithic device, or a group of distributed devices.

[0061] Content items 121A-C (e.g., media content items) can be stored in one or more data repositories. The data repository can be part of one or more platforms. Examples of content items 121 can include, but are not limited to, digital videos, digital movies, animated images, digital photos, digital music, digital audio, digital video games, collaborative media content presentations, website content, social media updates, e-books, e-journals, digital audiobooks, web blogs, software applications, etc. Content items 121A-C can also be referred to as media items. Content items 121A-C can be pre-recorded or live. For the sake of brevity, videos can be used as examples of content items 121 (e.g., content item 121A) throughout the document. Videos can include pre-recorded videos, live videos, short videos, etc.

[0062] Content items 121A-C may be provided by a content provider. The content provider may be a user, a company, an organization, etc. The content provider may provide content item 121 (e.g., content item 121A) as a video. The content provider may provide content item 121 including live content, for example, content item 121 may include live video, real-time chat associated with the video, etc.

[0063] Client devices 110 may include devices such as televisions, smart phones, personal digital assistants, portable media players, laptop computers, e-book readers, tablet computers, desktop computers, game consoles, set-top boxes, and the like.

[0064] The client device 110A may include a communication application 115A. A user may consume content items 121 (e.g., content items 121A) via the communication application 115. For example, the communication application 115 may access one or more networks 105 (e.g., the Internet) via the hardware of the client device 110 to provide the content items 121 to the user. As used herein, "media," "media items," "online media items," "digital media," "digital media items," "content," "media content items," and "content items" may include electronic files that may be executed or loaded using software, firmware, and / or hardware configured to present content items. In one implementation, the communication application 115 may be an application that allows a user to compose, send, and receive content items 121 (e.g., videos) through a platform (e.g., a content providing platform 120, a recommendation platform 157, a social networking platform 160, and / or a search platform 145) and / or a combination of platforms and / or networks.

[0065] In some embodiments, the communication application 115 may be (or include aspects of) a social networking application, a video sharing application, a video streaming application, a video game streaming application, a photo sharing application, a chat application, or a combination of these applications. The communication application 115 associated with the client device 110 may render, display, present, and / or play one or more content items 121 to one or more users. For example, the communication application 115 may provide a user interface 116 (e.g., a graphical user interface) to be displayed on the endpoint device 110 for receiving and / or playing video content. In some embodiments, the communication application 115 is associated with (and managed by) the content platform 102 or the content providing platform 120.

[0066] In some embodiments, the communication application 115 may include a content viewer 113 and a related product component 114. A user interface 116 (UI) may display the content viewer 113 and the related product component 114. The related product component 114 may be used to display a UI element that displays information about one or more products (e.g., one or more products associated with a content item). The related product component 114 may display a UI element to notify the user that the content item has one or more associated products (e.g., the related product component 114 may cause a "shopping" symbol to be displayed near the content viewer 113, the related product component 114 may cause an element to be displayed near an element for selecting a content item to be presented from a list of content items and indicating that the content item has one or more associated products, etc.). The related product component 114 may cause the UI to display information about one or more products associated with the content item (e.g., the related product component 114 may cause the following UI to be displayed, which may include a list of products associated with the content item, may include images of products associated with the content item, may include price information or information linking the product to the content item, such as a timestamp of a video portion associated with the product, etc.). Related products component 114 may cause a UI element to display additional information about one or more products, such as variations of a product (e.g., color variations, size variations, etc.), related products, recommended products, etc. Related products component 114 may cause a UI element to display one or more options for purchasing one or more products. In some embodiments, a user may be able to navigate between different views provided by related products component 114. Figure 4A-4EFind further description of example UI elements associated with products related to content items. In some embodiments, for example, if more than one content item with associated products is displayed, then the related product component 114 can cause more than one UI element to be displayed. In some embodiments, the related product component 114 can, for example, cause the UI element indicating the related product to not be displayed based on user settings, user preferences, user history, etc. In some embodiments, multiple content viewers 113 and / or related product components 114 can be associated with a user interface, communication application, client device, etc. For example, multiple content items can be displayed for a user at one time. In some implementations, the communication application 115 is a web browser that can access, retrieve, present and / or navigate content (e.g., web pages, such as hypertext markup language (HTML) pages, digital media items, etc.), and can include a related product component 114 and a content viewer 113, which can be an embedded media player embedded in a user interface 116 (e.g., a web page associated with viewing content) provided by a content providing platform 120. Alternatively, the application 115 is not a web browser, but is a standalone application (e.g., mobile application, desktop application, game console application, TV application, etc.) downloaded from a platform (e.g., content providing platform 120, recommendation platform 157, social networking platform 160, or search platform 145) or pre-installed on the client device 110. The standalone application 115 can provide a user interface 116, including a content viewer 113 (e.g., an embedded media player) and related product components 114.

[0067] In some embodiments, the content platform system 102 may include a product information platform 161 (e.g., hosted by the server machine 106). The product information platform 161 may store, retrieve, provide, receive, etc. data related to one or more products associated with one or more content items. The content platform system 102 may provide data to the related product component 114 of the client device 110. The product information platform 161 may include information provided by a content creator, for example, a content creator may provide a list of products associated with a content item provided by the content creator to the content platform system 102. The product information platform 161 may include information provided by one or more users, for example, one or more users may identify products associated with a content item, for example, in response to having the content item presented to them. The product information platform 161 may include information provided by a product identification system 175, for example, one or more machine learning models may be used to identify products highlighted in a content item and provide an indication of associated products and content items to the product information platform 161.

[0068] In some embodiments, the communication application 115 installed on the client device 110 can be associated with a user account, for example, the user can log into the account on the client device 110. In some embodiments, multiple client devices 110 can be associated with the same client account. In some embodiments, providing information about the association of a product with one or more content items can be performed based on the user account (e.g., account settings, account history (e.g., history of interaction with UI elements including associated product information), etc.).

[0069] In some embodiments, the client device 110 may include one or more data repositories. The data repository may include commands (e.g., instructions that cause operations when executed by a processing device) to render a UI (e.g., user interface 116). The instructions may include instructions to render interactive components (e.g., UI elements with which a user can interact to present additional information about one or more products associated with a content item). In some embodiments, the instructions may cause the processing device to render a UI element that presents information about one or more products associated with one or more content items (e.g., multiple videos reviewing products of interest to the user may be presented together with UI elements presenting more information about the products).

[0070] In some embodiments, one or more server machines 106 may include computing devices such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc., and may be coupled to one or more networks 105. Server machines 106 may be stand-alone devices or part of any platform (e.g., content provision platform 120, social networking platform 160, etc.).

[0071] The social networking platform 160 may provide an online social networking service. The social networking platform 160 may provide a communication application 115 for users to create profiles and perform activities using their profiles. Activities may include updating profiles, exchanging messages with other users, evaluating (e.g., liking, commenting, sharing, recommending) status updates, photos, videos, etc., and receiving notifications associated with other user activities. In some embodiments, a user may share additional product information (e.g., provided by the product information platform 161) with one or more additional users via the social networking platform 160.

[0072] The recommendation platform 157 may be used to generate and provide content recommendations (e.g., articles, videos, posts, news, games, etc.). Recommendations may be based on search history, content consumption history, followed / subscribed channel content, linked profiles (e.g., friend lists), popular content, etc. The recommendation platform 157 may be used to generate, for example, a user home feed, a user watch list, a user play list, etc. One or more UI elements indicating associated products, displaying an associated product list, presenting product information, presenting one or more options for purchasing a product, etc. may be presented in conjunction with, as part of, associated with, accessible from, etc., a home feed, a watch list, a play list, a next watch list, etc. The presentation of the one or more UI elements may be performed based on user history, user settings, data indicating user preferences (e.g., demographic data), etc.

[0073] The search platform 145 can be used to allow users to query one or more data repositories 140 and / or one or more platforms and receive query results. Users can use the search platform 145 to search for content items, search topics, and the like. For example, users can use the search platform 145 to search for content items with one or more associated products. The search platform 145 can be used to search for content items related to a certain type of product (e.g., headphone review videos). The search platform 145 can be used to search for content items related to a certain product (e.g., headphones of a specific manufacturer and / or model). In response to receiving a search query, one or more UI elements can be displayed to the user. The type, style, etc. of the displayed UI elements can be based on the content searched by the user. For example, in response to a search for content related to a certain type of product (e.g., headphones), a UI element can be displayed to indicate that the content item suggested in view of the search has one or more associated products. By further example, in response to a search for content items related to a more specific product (e.g., "best headphones for podcasts"), different UI elements can be displayed to provide information about products associated with the content items. By further example, in response to a search for content items related to a particular product (e.g., a particular manufacturer and / or model), different UI elements may be displayed to provide specific information about the searched product, indicating that the product is associated with content items recommended based on the search.

[0074] The content providing platform 120 may be used to provide access to content items 121 to one or more users and / or provide content items 121 to one or more users. For example, the content providing platform 120 may allow users to consume, upload, download, and / or search for content items 121. In another example, the content providing platform 120 may allow users to evaluate content items 121, such as to approve (“like”), disapprove, recommend, share, rate, and / or comment on content items 121. In another example, the content providing platform 120 may allow users to edit content items 121. The content providing platform 120 may also include a website (e.g., one or more web pages) and / or one or more applications (e.g., communication applications 115), which may be used to provide access to content items 121 to one or more users. For example, the client device 110 may use the communication application 115 to access the content items 121. The content providing platform 120 may include any type of content delivery network that provides access to the content items 121.

[0075] The content providing platform 120 may include a plurality of channels (e.g., channel A 125, channel B 126, etc.). A channel may be a collection of content available from a common source, a collection of content having a common topic or theme, etc. The data content may be digital content selected by a user, digital content provided by a user, digital content uploaded by a user, digital content selected by a content provider, digital content selected by a broadcaster, etc. For example, channel A 125 may include two videos (e.g., content items 121A-B). A channel may be associated with an owner, who may be a user who can perform actions on the channel. The content may be one or more content items 121. The data content of a channel may be pre-recorded content, live content, etc. Although a channel is described as one implementation of a content providing platform, implementations of the present disclosure are not limited to content sharing platforms that provide content items 121 via a channel model.

[0076] Product identification system 175, server machine 170, and server machine 180 may each include one or more computing devices, such as a rack server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, a graphics processing unit (GPU), an accelerator application specific integrated circuit (ASIC) (e.g., a tensor processing unit (TPU)), etc. The operations of prediction server 112, server machine 170, server machine 180, data repository 140, etc. may be performed by a cloud computing service, a cloud data storage service, etc.

[0077] Product identification system 175 may include one or more models 190. Models 190 included in product identification system 175 may perform tasks related to identifying one or more products from content items. One or more of models 190 may be trained machine learning models. Figure 3A and Figure 5C Describes the operations for generating a trained machine learning model, including training, validating, and testing the model.

[0078] Model 190 may include one or more text parsing models 191. Text parsing model 191 may be configured to receive text as input and generate one or more indications of products associated with the text as output. For example, a first model of text parsing model 191 may be configured to predict associated products based on the title of a content item, a second model of text parsing model 191 may be configured to predict associated products based on a (e.g., written) description of a content item, a third model may be configured to predict associated products based on subtitles of a content item (e.g., automatically generated subtitles, machine generated subtitles, user provided subtitles, etc.), and so on. In some embodiments, all operations of text parsing model 191 may be performed by a single model. In some embodiments, a model in text parsing model 191 may be configured to generate product context information (e.g., information indicating that a content item is associated with one or more products) as output. For example, product context information may indicate that a content item includes multiple categories of products, such as a group of different products (e.g., product type, product brand, product category such as "electronic equipment", etc.).

[0079] Model 190 may include one or more image parsing models 192. Image parsing models 192 may be configured to identify products from one or more images. Image parsing models 192 may include one or more models designed to identify that an image includes a product, a model configured to isolate a product image from a content item image (e.g., remove background elements, etc.), a model configured to determine the identity of a product in a content item image, and the like. The operations of image parsing models 192 may be performed by a single model. Image parsing models 192 may include one or more models configured to provide an image to a product recognition model. For example, image parsing models 192 may include a model configured to extract a portion of a still image of a content item, a model configured to extract one or more frames of a video content item, and the like. Models in image parsing models 192 may be provided with one or more frames of a video, one or more portions of one or more frames of a video, and the like, and generate one or more products and one or more confidence values ​​associated with the one or more products as output. For example, a model in image parsing model 192 may receive one or more frames of a video content item as input and generate as output a list of products with a confidence value indicating the likelihood that the product is included in the image of the content item. Image parsing model 192 may include one or more models that determine which images to utilize from a content item. For example, image parsing model 192 may include one or more models that select frames from a video content item for image recognition.

[0080] The image parsing model 192 may include one or more models configured to reduce the dimensionality of an image. For example, an image may be reduced to a vector of values. In some embodiments, one or more models in the image parsing model 192 may be configured to reduce the dimensionality of an image in such a way that similar images (e.g., images of the same product) may be represented similarly (e.g., by similar vectors) after dimensionality reduction. One or more models in the image parsing model 192 may be configured to compare a reduced dimensionality image from a content item (e.g., a vector of values ​​generated from one or more frames of a content item video) with a reduced dimensionality image of a known product (e.g., via the product information platform 161).

[0081] Model 190 may include text correction model 193. Text correction model 193 may be configured to provide corrections to text associated with content items. Text correction model 193 may be configured to adjust text associated with content items to include one or more products referenced in the content items. One or more models in text correction model 193 may be configured to adjust computer-generated, machine-generated, automatically generated, etc. text associated with content items. One or more models in text correction model 193 may be configured to update the subtitles of a video (e.g., incorrect subtitles) to include one or more products. In some embodiments, the machine-generated text (e.g., subtitles) associated with a content item may be incorrect. For example, when generating subtitles, the name of a product may be replaced by an approximation (e.g., the name of the product may not be a word in the subtitle language, the name of the product may be a word in a language different from the subtitle language, etc.). The model in text correction model 193 may be configured to identify parts of text that may be incorrect and recommend corrections, perform corrections, warn a user or another system, etc. For example, a model in text correction model 193 may receive machine-generated subtitles for a video, identify portions of the subtitles that may incorrectly replace a product name with words in the subtitle language, and provide data indicating the potentially incorrect text to another model, system, user, etc.

[0082] Model 190 may include fusion model 194. Fusion model 194 may receive one or more indications of products associated with a content item as input. In some embodiments, fusion model 194 receives outputs from one or more other models (e.g., text parsing model 191, image parsing model 192, etc.) as input. Fusion model 194 may receive one or more indications of products associated with a content item and one or more confidence value indications. For example, fusion model 194 may receive indications of one or more products detected in the title of a content item and confidence values ​​associated with the one or more products. Fusion model 194 may further receive indications of one or more products detected in the description of the content item and confidence values ​​associated with the one or more products. Fusion model 194 may further receive indications of one or more products detected in the subtitles of the content item and confidence values ​​associated with the one or more products. Fusion model 195 may further receive indications of one or more products detected in the image of the content item and confidence values ​​associated with the one or more products. Fusion model 195 may generate one or more products detected in association with a content item as output. Fusion model 195 may further generate a confidence value associated with the confidence that the product appears in the content item, that the product is associated with the content item, etc. Further operations may be performed based on the output of fusion model 195 (e.g., product identity information and confidence value) (e.g., a UI element describing the product associated with the content item may be presented).

[0083] One machine learning model that can be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network. An artificial neural network typically includes a feature representation component with a classifier or regression layer that maps features to a desired output space. For example, a convolutional neural network (CNN) has multiple layers of convolutional filters. Pooling is performed at the lower layers, and nonlinearities can be resolved, with a multi-layer perceptron typically attached on top to map the top features extracted by the convolutional layers to decisions (e.g., classification outputs).

[0084] Recurrent Neural Network (RNN) is another machine learning model. Recurrent Neural Network models are designed to interpret a series of inputs where the inputs are intrinsically related to each other, such as time trajectory data, sequential data, etc. The output of the perceptron of the RNN is fed back into the perceptron as input to generate the next output.

[0085] Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure in which different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly abstract and composite representation. For example, in an image recognition application, the raw input may be a matrix of pixels; the first representation layer may abstract the pixels and encode edges; the second layer may compose and encode the arrangement of edges; the third layer may encode higher-level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer may recognize the scanned character. Notably, the deep learning process can learn by itself which features are best placed at which level. The "depth" in "deep learning" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a considerable credit assignment path (CAP) depth. CAP is a chain of transformations from input to output. CAP describes the potential causal relationship between input and output. For feedforward neural networks, the depth of CAP can be either the depth of the network or the number of hidden layers plus 1. For recurrent neural networks, where signals can propagate through a layer more than once, the CAP depth is potentially unlimited.

[0086] In some embodiments, product identification system 175 further includes server machine 170 and server machine 180. Server machine 170 includes a data set generator 172 that can generate data sets (e.g., data input sets and target output sets) to train, validate, and / or test models 190 including one or more machine learning models. Figure 2 and Figure 5A Some operations of the dataset generator 172 are described in detail. In some embodiments, the dataset generator 172 may divide the historical data (e.g., pre-existing content item data, content items with one or more specified associated products, content items with product assignments provided by one or more users, etc.) into a training set (e.g., sixty percent of the historical data), a validation set (e.g., twenty percent of the historical data), and a test set (e.g., twenty percent of the historical data).

[0087] In some embodiments, components of the product identification system 175 can generate multiple feature sets. For example, a feature can be a rearrangement of input data, a combination of input data, a dimensionality reduction of input data, a subset of input data, etc. One or more data sets can be generated based on one or more features of the input data.

[0088] The server machine 180 includes a training engine 182, a validation engine 184, a selection engine 185, and / or a test engine 186. The engines (e.g., the training engine 182, the validation engine 184, the selection engine 185, and the test engine 186) may refer to hardware (e.g., circuits, dedicated logic, programmable logic, microcode, processing devices, etc.), software (such as instructions running on a processing device, a general-purpose computer system, or a dedicated machine), firmware, microcode, or a combination thereof. The training engine 182 may be capable of training one or more models 190 using one or more feature sets associated with a training set from the dataset generator 172. The training engine 182 may generate a plurality of trained models 190, wherein each trained model 190 corresponds to a different feature set of the training set. The dataset generator 172 may receive the output of the trained model (e.g., the fusion model 194 may be trained based on the output of the text parsing model 191 and / or the image parsing model 192), collect the data into a training dataset, a validation dataset, and a test dataset, and use these datasets to train a second model (e.g., the fusion model 194).

[0089] The validation engine 184 may be able to validate the trained models 190 using the corresponding feature set of the validation set from the dataset generator 172. For example, a first trained machine learning model 190 trained using the first feature set of the training set may be validated using the first feature set of the validation set. The validation engine 184 may determine the accuracy of each trained model 190 based on the corresponding feature set of the validation set. The validation engine 184 may discard trained models 190 whose accuracy does not meet a threshold accuracy. In some embodiments, the selection engine 185 may be able to select one or more trained models 190 having an accuracy that meets the threshold accuracy. In some embodiments, the selection engine 185 may be able to select a trained model 190 with the highest accuracy among the trained models 190.

[0090] The testing engine 186 may be able to test the trained model 190 using a corresponding feature set of the test set from the dataset generator 172. For example, a first trained machine learning model 190 trained using a first feature set of the training set may be tested using a first feature set of the test set. The testing engine 186 may determine the trained model 190 with the highest accuracy among all trained models based on the test set.

[0091] In the case of a machine learning model, the model 190 may refer to a model artifact created by the training engine 182 using a training set that includes data inputs and corresponding target outputs (correct answers for the corresponding training inputs). Patterns in the data set that map the data inputs to the target outputs (correct answers) may be found, and the machine learning model 190 is provided with a mapping that captures these patterns. The machine learning model 190 may use one or more of a support vector machine (SVM), a radial basis function (RBF), clustering, supervised machine learning, semi-supervised machine learning, unsupervised machine learning, a k-nearest neighbor algorithm (k-NN), linear regression, a random forest, a decision forest, a neural network (e.g., an artificial neural network, a recurrent neural network), a linear model, a function-based model (e.g., an NG3 model), and the like. The synthetic data generator 174 may include one or more machine learning models, which may include one or more models of the same type (e.g., an artificial neural network).

[0092] Automatically (e.g., based on a model) detecting products from content items and associated data provides significant technical advantages over other methods. In some embodiments, a content item that focuses on displaying a product (e.g., a product review video that evaluates the product) can become linked or associated with the product (e.g., data linking the product to the content item can be generated), without the need for the attention, action, time, etc. of the content creator. In some embodiments, a content item for an advertising product (e.g., a content item can be sponsored or can promote one or more products) can be linked or associated with the product. Model-based detection of products in content items can generate product associations for products that are not particularly highlighted in content items but are present in content items (e.g., products that users may be interested in purchasing can be on the screen of video content items). Model-based detection of products in content items can generate product associations for products advertised in content items. Based on this model-based detection, by providing a UI element to the user (e.g., indicating that the product is associated with the content item), the user can be guided to the product present in the content item.

[0093] One or more models 190 may be run on an input to generate one or more outputs. The model may determine (e.g., extract) confidence data from the output, the confidence data indicating a level of confidence that the output of the model is an accurate description of the content item. For example, the model may determine that a first product is associated with a content item, and determine the confidence that the model has correctly discovered the first product in the content item. One or more components of the product identification system 175 may use the confidence data to decide whether to update data associated with the content item, e.g., whether to associate one or more products with the content item, whether to update one or more subtitles of the content item, etc.

[0094] The confidence data may include or indicate a confidence level that the output of the model (e.g., one or more products) is an accurate indication of the product associated with the content item. For example, the confidence level of the model output (e.g., associated with a product identified in the content item) may be a real number between 0 and 1 (including 0 and 1). 0 may indicate zero confidence that the predicted product is associated with the content item, while 1 may indicate an absolute confidence that the predicted product is associated with the content item. In response to the confidence data indicating that the confidence level of a predetermined number of instances (e.g., a percentage of instances, a frequency of instances, a total number of instances, etc.) is below a threshold level, the product identification system 175 may cause one or more trained models 190 (e.g., based on updated and / or new data for training, validation, testing, etc.) to be retrained. Retraining may include generating one or more data sets (e.g., via a data set generator 172).

[0095] For purposes of illustration and not limitation, aspects of the present disclosure describe training one or more machine learning models 190 using historical data, and inputting current data (e.g., newly updated content items, content items not previously associated with products, etc.) into one or more trained machine learning models to determine outputs indicating content item-product associations. In other embodiments, heuristic models, physics-based models, or rule-based models are used to determine that one or more products are associated with content items (e.g., without using trained machine learning models). In some embodiments, such models can be trained using historical data. In some embodiments, these models can be retrained using historical data. Figure 2 The data input 210 may be used to input any information described herein, or otherwise use the information in a heuristic model, a physics-based model, or a rule-based model.

[0096] In some embodiments, the functionality of client device 110, product identification system 175, content platform system 102, server machine 170, and server machine 180, server machine 106 may be provided by a smaller number of machines. For example, in some embodiments, server machines 170 and 180 may be integrated into a single machine, while in some other embodiments, server machines 170, server machines 180, and server machine 106 may be integrated into a single machine. In some embodiments, client device 110 and server machine 106 may be integrated into a single machine. In some embodiments, the functionality of client device 110, server machine 106, server machine 170, server machine 180, and data repository 140 may be performed by a cloud-based service.

[0097] In general, functions described in one embodiment as being performed by client device 110, server machine 106, server machine 170, and server machine 180 may also be performed on server machine 106 in other embodiments, if appropriate. In addition, functionality attributed to a particular component may be performed by different or multiple components operating together. For example, in some embodiments, product identification system 175 may determine an association between a product and a content item. In another example, content platform system 102 may determine an association between a content item and one or more products.

[0098] Additionally, the functionality of a particular component may be performed by different or multiple components operating together.One or more of server machine 106, server machine 170, or server machine 180 may be accessed as a service provided to other systems or devices through an appropriate application programming interface (API).

[0099] In the implementation of the present disclosure, a "user" can be represented as a single individual. However, other implementations of the present disclosure cover "users" as entities controlled by a group of users and / or automatic sources. For example, a group of individual users united into a community in a social network can be regarded as "users". In another example, an automated consumer can be an automated ingestion channel, such as a theme channel, for one or more platforms, one or more content items, etc. In addition to the above description, a user can be provided with a control that allows the user to make a choice of whether and when the system, program or feature described herein can be able to collect user information (e.g., information about the user's social network, social behavior, or activity, occupation, user preferences, or the user's current location) and whether to send content or communication from the server to the user. In addition, before storing or using certain data, it can be processed in one or more ways to delete personally identifiable information. For example, the identity of the user can be processed so that the user's personally identifiable information cannot be determined, or in the case of obtaining location information, the user's geographic location can be generalized (such as generalized to a city, zip code, or state province) so that the user's specific location cannot be determined. Therefore, the user can control what information is collected about the user, how to use the information, and what information is provided to the user.

[0100] Figure 22 is a block diagram of a system 200 including a dataset generator 272 for creating a dataset for one or more models according to some embodiments. The dataset generator 272 can use historical data to create a dataset (e.g., data input 210, target output 220). An unsupervised machine learning model can be trained using a dataset generator similar to the dataset generator 272, e.g., the target output 220 may not be generated by the dataset generator 272. A semi-supervised machine learning model can be trained using a dataset generator similar to the dataset generator 272, e.g., the target output 220 corresponding to a subset of the data input 210 may be generated by the dataset generator 272.

[0101] Dataset generator 272 can generate data sets to train, test and validate models. Dataset generator 272 can generate data sets for machine learning models. System 200 can generate data sets for training, testing and / or validating fusion models, for example, for determining the likelihood that one or more products appear in a content item. Systems similar to system 200 can generate data sets for training, testing and / or validating models with different functions, and make corresponding changes to the input data and / or output data included in the data sets. Models for parsing text (e.g., extracting one or more references to products from text associated with a content item), parsing images (e.g., extracting one or more references to products from images associated with a content item), correcting text (e.g., including one or more references to products in machine-generated text associated with a content item), etc. can have data sets for training, testing and / or validating models generated by data set generators similar to dataset generator 272.

[0102] In some embodiments, a dataset generator (such as dataset generator 272) can be associated with two or more separate models (e.g., a dataset can be used to train an integrated model). For example, an input dataset can be provided to a first model, an output of the first model can be provided to a second model, and a target output can be provided to the second model to train, test, and / or validate the first model and the second model (e.g., an integrated model).

[0103] The data set generator 272 can generate one or more data sets to provide to the model, for example, during training, validation and / or testing operations. The machine learning model can be provided with a historical data set. A historical text parsing data set 264A-264Z can be provided as data input to a machine learning model (e.g., a fusion model). The text parsing data can be provided by the machine learning model, for example, and can include one or more products and confidence values ​​identified by the machine learning model in the text associated with the content item. A historical image parsing data set 265A-265Z can be provided as data input to the machine learning model. Image parsing data can be provided by a trained machine learning model, for example, and can include one or more products and confidence values ​​identified by the machine learning model in one or more images associated with the content item.

[0104] In some embodiments, the dataset generator 272 may be configured to generate a dataset for training, testing, validating, etc., the fusion model. A dataset generator similar to the dataset generator 272 may generate a text dataset (e.g., content item title text, content item description text, content item subtitle text, etc.) as data input to train a machine learning model to determine one or more products associated with a content item. A dataset generator similar to the dataset generator 272 may generate an image dataset (e.g., one or more frames from a video content item, portions of one or more frames from a video content item, etc.) as data input to train a machine learning model to determine one or more products associated with a content item.

[0105] In some embodiments, the data set generator 272 generates a data set (e.g., a training set, a validation set, a test set) including one or more data inputs 210 (e.g., a training input, a validation input, a test input). The data input 210 may be provided to Figure 1 The training engine 182, validation engine 184, or test engine 186 of the embodiment of the present invention can be used to train, validate, or test a model (e.g., a fusion model, a text parsing model, an image parsing model, etc.). The dataset generator 272 can generate a dataset (e.g., a training set, a validation set, a test set) including one or more data inputs 210. The data inputs 210 can be referred to as "features," "attributes," "vectors," or "information."

[0106] In some embodiments, the dataset generator 272 may generate a first data input corresponding to the first historical text parsing dataset 264A and / or the first historical image parsing dataset 265A to train, validate, or test the first machine learning model. The dataset generator 272 may generate a second data input corresponding to the second historical metrology dataset 264B and / or the second design rule dataset 265B to train, validate, or test the second machine learning model. Figure 5ASome embodiments of generating training sets, test sets, validation sets, etc. are further described.

[0107] In some embodiments, the data set generator 272 may generate a target output 220 to provide for training, testing, validating, etc., one or more machine learning models. The data set generator 272 may generate product association data 268 as the target output 220. The product association data 268 may include identifiers of one or more products associated with the content item (e.g., manually labeled product associations). The product association data 268 may include an input-output mapping, for example, the historical text parsing data set 264A may be associated with the first product association data set 268, and so on. The machine learning model may be updated (e.g., trained) by providing input data, generating outputs, and comparing them with the provided target outputs (e.g., "correct answers"). The various weights, biases, etc. of the model are then updated to better align the model with the training data. This process may be repeated multiple times to generate a model that provides accurate outputs for a threshold portion of the provided input. The target output 220 may share one or more features of the data input 210, for example, the target output 220 may be organized into attributes or vectors, the target output 220 may be organized into sets AZ, and the like.

[0108] In some embodiments, a data set generator similar to data set generator 272 can be used in combination with a text parsing model configured to determine one or more product associations with a content item. Product associations can include contextual associations, such as product brands, product types, product classifications, etc. The data set generator can generate a product list, product type, product brand, etc. associated with the content item, associated with the text of the content item as a target output. A data set generator similar to data set generator 272 can be used in combination with an image parsing model configured to determine one or more product associations with a content item. The data set generator can generate a product list, product classification, product category, etc. associated with an image input as a target output. A data set generator similar to data set generator 272 can be used in combination with a text correction model. The text correction model can be configured to identify machine-generated text that incorrectly provides one or more words in a target language to replace a product name. The text correction model can be provided with one or more sets of machine-generated text as input data, and provided with products associated with the text (e.g., products that are not correctly captured by the machine generation of the text) as a target output.

[0109] In some embodiments, after a data set is generated and used to train, validate, or test a machine learning model, the model may be further trained, validated, or tested or adjusted (e.g., adjusting weights or parameters associated with the input data of the model, such as connection weights in a neural network). The model may be adjusted and / or retrained based on data different from the original training operation (e.g., data generated after training, validation, and / or testing of the model).

[0110] Figure 3A is a block diagram illustrating a system 300A for generating output data (e.g., product / content item association data) according to some embodiments. System 300A can be used in conjunction with a fusion model to generate product / content item association data and confidence data based on potential product / content item associations generated by other models (e.g., a text-based model that detects products in text associated with a content item, an image-based model that detects products in images associated with a content item, etc.). Systems similar to system 300A can be used to generate output data from other types of models (e.g., text parsing models, image parsing models, text correction models, etc.).

[0111] At block 310, system 300A (eg, Figure 1 ) performs data partitioning (e.g., via Figure 2 The training data 364 may be generated by one or more trained machine learning models (e.g., models configured to detect products in text or images associated with content items). The training data 364 may be subjected to data partitioning at block 310 to generate a training set 302, a validation set 304, and a test set 306. For example, the training set may be 60% of the training data, the validation set may be 20% of the training data, and the test set may be 20% of the training data.

[0112] The generation of training set 302, validation set 304 and test set 306 can be customized for a specific application. For example, the training set can be 60% of the training data, the validation set can be 20% of the training data, and the test set can be 20% of the training data. System 300A can generate multiple feature sets for each of the training set, validation set and test set. For example, if the training data 364 includes product associations (e.g., titles associated with content items and descriptions associated with content items) extracted from text data from more than one text source, the input training data can be divided into a first feature set and a second feature set, the first feature set including products identified from the text from the first source, and the second feature set including products identified from the text from the second source. Either or both of the target input and target output can be divided into sets or neither can be divided into sets. Multiple models can be trained on different data sets.

[0113] At block 312, system 300A performs model training using training set 302 (e.g., via Figure 1 Training engine 182). The training of the machine learning model can be implemented in a supervised learning manner, that is, the model is provided with a training data set including labeled inputs, its output is observed, an error is defined (by measuring the difference between the output and the label value), and the weights of the model are adjusted using techniques such as deep gradient descent and back propagation to minimize the error. In many applications, repeating this process for many labeled inputs in the training data set will produce a model that is able to produce correct outputs when given inputs that are different from the inputs present in the training data set. In some embodiments, the training of the machine learning model can be implemented in an unsupervised manner, for example, no labels or classifications may be provided during training. Unsupervised models can be configured to perform anomaly detection, result clustering, etc.

[0114] For each training data item in the training data set, the training data item can be input into a model (e.g., into a machine learning model). The model can then process the input training data item (e.g., an indication of one or more products detected in conjunction with a content item, and an associated confidence value, etc.) to generate an output. The output can include, for example, a list of products that can be associated with the content item and a corresponding confidence value. The output can be compared to the labels of the training data item (e.g., a set of manually labeled products associated with the content item).

[0115] The processing logic may then compare the generated output (e.g., predicted product / content item associations) to the labels included in the training data items (e.g., a manually generated list of product / content item associations). The processing logic determines an error (i.e., a classification error) based on the difference between the output and the labels. The processing logic adjusts one or more weights and / or values ​​of the model based on the error.

[0116] In the case of training a neural network, an error term or increment may be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters (weights of one or more inputs to the node) for one or more of its nodes. The parameters may be updated in a back-propagation manner so that the nodes of the highest layer are updated first, followed by the nodes of the next layer, and so on. The artificial neural network comprises multiple layers of "neurons", each of which receives as input the values ​​of the neurons from the previous layer. The parameters of each neuron include weights associated with the values ​​received from each neuron of the previous layer. Therefore, adjusting the parameters may include adjusting the weights of each input assigned to one or more neurons of one or more layers in the artificial neural network.

[0117] System 300A can train multiple models using multiple feature sets of training set 302 (e.g., a first feature set of training set 302, a second feature set of training set 302, etc.). For example, system 300A can train models to generate a first trained model using a first feature set in the training set (e.g., a subset of training data 364, such as data associated only with a subset of models configured to generate product / content item associations, etc.), and to generate a second trained model using a second feature set in the training set. In some embodiments, the first trained model and the second trained model can be combined to generate a third trained model (e.g., which can be better than the first or second trained model alone). In some embodiments, the feature sets used to compare models can overlap (e.g., a first feature set is a product based on a content item title, description, and some images, while a second feature set is a product detected based on a content item description, a different set of images of the content item, and a detected context of the content item (e.g., a product type associated with the content item)). In some embodiments, hundreds of models can be generated, including models with various feature arrangements and model combinations.

[0118] At block 314, system 300A performs model validation using validation set 304 (e.g., via Figure 1314 ). The system 300A may validate each trained model using a corresponding feature set of the validation set 304. For example, the system 300A may validate a first trained model using a first feature set in the validation set, and validate a second trained model using a second feature set in the validation set. In some embodiments, the system 300A may validate hundreds of models generated at block 312 (e.g., models with various feature arrangements, model combinations, etc.). At block 314, the system 300A may determine the accuracy of each of the one or more trained models (e.g., via model validation), and may determine whether one or more of the trained models have an accuracy that satisfies a threshold accuracy. In response to determining that none of the trained models have an accuracy that satisfies the threshold accuracy, the process returns to block 312, where the system 300A performs model training using a different feature set of the training set, an updated or expanded training set provided by the data set generator, etc. In response to determining that one or more of the trained models have an accuracy that satisfies the threshold accuracy, the process continues to block 316. The system 300A may discard trained models whose accuracy is below a threshold accuracy (eg, based on a validation set).

[0119] At block 316, system 300A performs model selection (eg, via Figure 1 The selection engine 185 of block 314 may be used to determine which of the one or more trained models that meet the threshold accuracy has the highest accuracy (e.g., the selected model 308 based on the verification of block 314). In response to determining that two or more trained models that meet the threshold accuracy have the same accuracy, the process may return to block 312, where the system 300A performs model training using a further refined training set corresponding to the further refined feature set to determine the trained model with the highest accuracy.

[0120] At block 318, system 300A performs model testing using test set 306 (eg, via Figure 1The system 300A may test the selected model 308 using the first feature set in the test set to determine that the first trained model meets the threshold accuracy (e.g., based on the first feature set of the test set 306). In response to the accuracy of the selected model 308 not meeting the threshold accuracy (e.g., the selected model 308 is overfit to the training set 302 and / or the validation set 304 and is not applicable to other data sets such as the test set 306), the process continues to block 312, where the system 300A performs model training (e.g., retraining) using a different training set corresponding to a different feature set, a different content item, etc. In response to determining that the selected model 308 has an accuracy that meets the threshold accuracy based on the test set 306, the process continues to block 320. At least in block 312, the model may learn patterns in the training data to make predictions, and in block 318, the system 300A may apply the model to the remaining data (e.g., the test set 306) to test the predictions.

[0121] At block 320, the system 300A uses a trained model (e.g., the selected model 308) to receive current data 322 (e.g., newly uploaded content items, newly created content items, content items not included in the training set, test set, or validation set of the selected model 308, etc.) and determines (e.g., extracts) output data 324 (e.g., product / content item associations and corresponding confidence values) from the output of the trained model. Corrective actions associated with content items and / or associated data may be performed in view of the output data 324. For example, the instructions may be updated to include presenting a UI element with a content item that specifies that the content item includes one or more associated products, the instructions may be updated to include presenting a UI element with a content item that includes additional information about the associated products, and so on. The instructions may depend on additional factors, such as user preferences for content items (e.g., search pages, home pages, etc.), presentation context, and the like. In some embodiments, the current data 322 may correspond to the same type of features in the historical data used to train the machine learning model. In some embodiments, current data 322 corresponds to a subset of those types of features in the historical data used to train the selected model 308 (e.g., a machine learning model may be trained using product associations and / or contextual information and confidence values ​​from several sources, such as text-based and image-based sources, and a subset of that data is provided as current data 322).

[0122] In some embodiments, the performance of a machine learning model (e.g., selected model 308) can be adjusted, improved, and / or updated over time. For example, additional training data can be provided to the model to improve the model's ability to correctly classify product associations with content items. In some embodiments, portions of current data 322 can be provided to retrain the model (e.g., via Figure 1 The training engine 182 of the current data 322 can be used to update and / or improve the selected model 308 in a periodic, continuous, etc. manner.

[0123] In some embodiments, one or more of actions 310-320 may occur in various orders and / or with other actions not presented and described herein. In some embodiments, one or more of actions 310-320 may not be performed. For example, in some embodiments, one or more of the data partitioning of block 310, the model validation of block 314, the model selection of block 316, or the model testing of block 318 may not be performed.

[0124] System 300A has been described with respect to a fusion model. The fusion model accepts one or more indications of a product detected in association with a content item (e.g., from text associated with the content item, from an image associated with the content item, etc.) and a confidence value (e.g., confidence that the product is indeed referenced in the content item), and generates as an output an overall likelihood that the product is referenced in the content item based on multiple inputs. Systems similar to system 300A can be used to perform other machine learning-based tasks, for example, a text parsing model or an image parsing model (whose output is provided as input to the fusion model) can operate in a manner similar to that described in conjunction with system 300A, with appropriate substitution of input data, output data, etc.

[0125] Figure 3B 300B is a block diagram of an example system 300B for generating associations between content items and one or more products according to some embodiments. System 300B may include multiple modules, such as image recognition 330, image verification 340, text recognition 350, fusion 360, etc. In some embodiments, multiple modules of system 300B may operate together to identify products from content items. For example, products detected in images of content items and metadata of content items (e.g., titles, descriptions, subtitles, etc.) may be provided to a fusion model to determine the likelihood that one or more products are associated with a content item based on multiple input channels. Actions may be taken in response to the likelihood that a product appears in a content item as determined by the fusion model (e.g., updating the metadata of the content item to include an indication of the product).

[0126] The image recognition module 330 can be used to identify one or more products from an image associated with a model, for example, an image from a content item can be compared with images in a product database (e.g., thousands of products) to identify products present in a video. The presented products can include a specific subject of the content item (e.g., a product reviewed in the content item), products included in the content item (e.g., products that appear by chance, products that appear without being particularly highlighted, etc.), and the like. The text recognition module 350 can identify one or more products associated with the content item from metadata / text data associated with the content item (e.g., from text including the title of the content item, a description of the content item, subtitles associated with the content item, etc.). The image verification module 340 can use one or more images to verify the products identified in the content item. For example, the image verification module 340 can work similarly to the image recognition module 330, but can be used to confirm the existence of one or more products identified by a separate module (e.g., by comparing an image of a potential product with a more limited range of product images provided by another module). The fusion module 360 ​​may receive candidate products and associated confidence values ​​included in a content item and determine, based on various inputs, a likelihood that one or more products appear in the content item.

[0127] Image recognition 330 can be used to determine products associated with content items (e.g., videos) having a visual component. Image recognition 330 can include frame selection 332. Frame selection 332 can be used to select one or more frames of a video to search for images of products. Frame selection 332 can occur via random sampling, periodic sampling, smart sampling methods, etc. For example, a content item (e.g., a video) can be provided to a machine learning model, and the machine learning model can be trained to predict video frames that are likely to include one or more products.

[0128] One or more frames may be provided to an object detection model 334. Object detection 334 may extract predicted objects from the one or more frames. For example, object detection 334 may isolate potential products from people, animals, background, etc. in the image data of the content item. Object detection 334 may be or include a machine learning model.

[0129] The image of the detected object may be supplied to embedding 336. Embedding 336 may include converting one or more images to a low dimension. Embedding 336 may include providing one or more images to a dimensionality reduction model. The dimensionality reduction model may be a machine learning model. The dimensionality reduction model may be configured to reduce the dimensionality of similar images in a similar manner. For example, embedding 336 may receive an image as input and generate a value vector as output. Embedding 336 may be configured, trained, etc. so that similar images (e.g., images of the same or similar products) are similarly represented in the reduced dimensionality vector space (e.g., by Cartesian distance, by cosine distance, by another distance metric, etc.). Embedding 336 may generate reduced dimensionality data.

[0130] The reduced dimensionality image data may be provided to product recognition 338. Product recognition 338 may identify one or more products associated with the reduced dimensionality representation provided by embedding 336. Product recognition 338 may compare the reduced dimensionality image data (e.g., provided by embedding 336) with reduced dimensionality image data of products included in product image index 339 (e.g., generated from product images by the same machine learning model as used by embedding 336). Product image index 339 may be stored as part of a data repository. Product image index 339 may include, for example, many products (e.g., hundreds of products, thousands of products, or more). Product image index 339 may include associations between stored image data (e.g., reduced dimensionality image data) and product identifiers, product indicators, and the like. Product image index 339 may be segmented, for example, the stored reduced dimensionality data may be classified into one or more categories, classes, and the like. For example, product recognition 338 may compare data received from embedding 336 with products of a particular category, type, classification, and the like. In some embodiments, the category, type, classification, and the like may be provided by one or more users, one or more content creators, may be automatically detected (e.g., by one or more machine learning models), and the like. The content item or one or more products associated with the content item may be associated with a category (e.g., a general category such as electronic equipment, a more limited category such as screen devices, a product classification such as a tablet computer, a manufacturer or brand, a model, etc.). Product recognition 338 may generate one or more indications of products detected in the image of the content item (e.g., a list of products that can be matched to the products represented in the product image index 339) and one or more confidence value indications (e.g., the confidence of each product in the product list being accurately detected). The output of image recognition 330 may be used to update the metadata of the content item (e.g., including an association with one or more products, including one or more product identifiers or indicators, etc.). The output of image recognition 330 may be provided to image verification 340, for example, to verify the presence of the image identified by image recognition 330 in the content item. The output of image recognition 330 may be provided to fusion 360, for example, to generate an overall and / or multi-input determination of the products included in the content item via a fusion model 366. The output of image recognition 330 may be provided to text recognition 350 (not shown), for example, to limit the space of products queried, searched, compared, etc. by the text recognition module 350. In some embodiments, image recognition 330 is used to identify products found in one or more frames of a video content item. For example, image recognition module 330 can be configured to generate a list of all products detected in any selected frame and provide a confidence value for each product in each selected frame. Image recognition module 330 can generate image-based product data, such as one or more identifiers of a product that is identified based on the image of the content item.

[0131] The image verification module 340 may be configured to verify the presence of the identified product of the content item using one or more images of the content item. For example, the image verification module 340 may include a model configured to confirm the presence of products identified by other models. The image verification module 340 may include auxiliary recognition 345. Auxiliary recognition 345 may include components similar to image recognition 330. In some embodiments, the image recognition module 330 may communicate directly with the product candidate image index 344 as an alternative or in addition to the auxiliary recognition 345 communicating with the product candidate image index 344. In some embodiments, the auxiliary recognition 345 may perform a similar role to the image recognition 330, but may include a different model, a model trained using different training data, a model configured to select different frames or detect different objects, etc.

[0132] Image verification 340 may include synthesis model 341. Synthesis model 341 may receive indications of products identified by image recognition module 330, text recognition module 350, auxiliary recognition 345 (data flow not shown), etc. Synthesis model 341 may select an object detection model 342 (e.g., an object detection model specifically configured for a product category or classification) for providing data to. Synthesis model 341 may include image selection, for example, synthesis model 341 may provide one or more images to object detection 342, may select one or more frames to provide to object detection 334, etc. For example, synthesis model 341 may provide one or more frames that may include a product to object detection 342 based on data received from image recognition 330 and text recognition 350. Object detection model 342 may perform similar functions to object detection model 334, for example, modified by the functions of synthesis model 341. Embedding 343 may perform similar functions to embedding 336, for example, to reduce the dimensionality of an image of a detected product. In some embodiments, product candidate image index 344 may include reduced dimension image data (e.g., vectors of values) detected by other modules (e.g., image recognition 330, text recognition 350, etc.). Auxiliary recognition 345 may compare the reduced dimension image data (e.g., embedded image data) with candidate data of product candidate image index 344 (e.g., products recognized by modules other than image verification module 340) to verify the presence of the product in the content item.

[0133] Text recognition 350 may be configured to identify one or more products from text data (e.g., metadata) associated with a content item. Text recognition 350 may generate metadata-based product data, such as one or more product identifiers based on the metadata of the content item. Text recognition 350 may generate text-based product data, such as one or more product identifiers based on the text data associated with the content item. Text recognition 350 may identify products from one or more of a content item title, a content item description, a subtitle associated with the content item (e.g., a machine-generated subtitle), a comment associated with the content item, and / or other text data or metadata of the content item. Text data (e.g., metadata) associated with the content item may be provided to a text parsing model 352. The text parsing model 352 may be a machine learning model. The text parsing model 352 may be configured to detect or predict products from text data associated with the content item. The text parsing model 352 may be configured to detect one or more products having a product identifier stored in a product identifier 354. The text parsing model 352 may provide an output (e.g., a list of detected candidate products, associated confidence values, etc.) to the image verification module 340. The text parsing model 352 may provide outputs to the synthesis model 341. The text parsing model 352 may provide outputs that affect products for image verification, for example, the outputs of the text parsing model 352 may cause products detected by the text recognition module 350 to be added to the product candidate image index 344. The image verification module 340 may query an index (e.g., the product candidate image index 344) that includes products detected by other modules (e.g., the image recognition module 330, the text recognition module 350, etc.).

[0134] The fusion module 360 ​​may receive output data (e.g., detected products, associated confidence values) from one or more sources (e.g., image recognition module 330, image verification module 340, text recognition module 350, etc.). The fusion module 360 ​​may further receive data from other sources (e.g., context term extraction 362 or additional feature extraction 363). The context term extraction 362 may, for example, provide context for potential products of the content item, such as categories or subjects associated with some products. The context term extraction 362 may be performed by one or more machine learning models. The context term extraction 362 may detect context information from text associated with the content item, metadata associated with the content item, etc. The additional feature extraction 363 may provide additional details that may be used to determine whether one or more products appear in the content item. The additional features may include video embeddings. The additional features may include other metadata of the content item, such as the date the content item was uploaded to the content providing platform (e.g., compared to the release date of the product), the classification of the content item (e.g., shopping or product review videos may be more likely to include products than other types of videos), and the like.

[0135] Data from multiple sources can be provided to the fusion model 366. The fusion model 366 can be configured to receive data (e.g., including one or more products with confidence values) and determine one or more products with confidence values ​​indicating the likelihood that the product appears in the content item. In some embodiments, the content item can be a video. In some embodiments, the content item can be a live feed, such as a live video feed (e.g., a product review stream, an unboxing stream, etc.). The content item can be a short video.

[0136] Figure 4A-4E Depicted is an example UI presented on a user device including a UI element indicating associated products in accordance with some embodiments. Figure 4A-4E The UI may be provided as part of an application of the device 400A-E (eg, a web browser application, a mobile application associated with / provided by a content platform, etc.). Figure 4A-4EUser interaction with various elements of a UI can cause changes in the presented UI elements. For example, interaction with a UI element indicating that a content item has associated products can cause a second UI element to be displayed (e.g., replacing the first UI element, expanding the first UI element, etc.), thereby presenting additional information (e.g., about the associated products). UI elements may include elements that, when interacted with, cause the UI to display less information about the associated products (e.g., folding a panel describing one or more associated products). Interacting with UI elements associated with one or more products can result in different effects, such as switching to a UI environment for presenting content items. The new UI environment may include one or more UI elements associated with one or more products of a content item. Interacting with UI elements associated with products can cause the display of UI elements that facilitate transactions (e.g., purchases) of products. Figure 4A-4E Various interactions between the UI and UI elements presented in the sample UI are possible (e.g., interacting with elements of a first UI layout may result in a transition to a second UI layout), and any transitions between sample UIs, similar UIs, inclusion of similar UI elements, etc. are within the scope of the present disclosure. Figure 4A-4E The description is combined with the video content item, and other types of content items (for example, image content, text content, audio content, etc.) can also be presented in a similar UI. Figure 4A-4E Any optional features, elements etc. presented in relation to one or more of the Figures may be suitably included in a system similar to another of these Figures.

[0137] Figure 4A Device 400A is depicted presenting example UI 402 including UI element 404 indicating one or more associated products in accordance with some embodiments. Device 400A may present a UI including elements of UI 402 and / or similar UI elements as part of an operation to present one or more content items for user selection. UI element 404 may be displayed in the form of a text or image. Figure 4A is depicted in a collapsed state, such as a collapsed default state.

[0138] UI 402 includes a first content item selector 406 (e.g., a video thumbnail) and a second content item selector 408. In some embodiments, more or fewer content items may be selected, UI 402 may be scrolled to view additional content items, etc. UI element 404 is associated with the content item indicated by content item selector 406. UI element 404 (and Figure 4A-4DIn some embodiments, product information (e.g., product / content item associations) may be provided by a content creator. Product information may be provided by one or more users. Product information may be provided by an administrator. Product information may be retrieved from a content item, for example, via one or more machine learning models, via a system such as system 300B, and the like.

[0139] In some embodiments, a user may interact with UI element 404 to be presented with a replaced UI element, an updated UI element, etc. For example, a user may interact with expansion element 410 to display more information about a product associated with a content item. In some embodiments, expanding UI element 404 may open a panel that includes additional information about one or more products associated with the content item. Expanding UI element 404, interacting with UI element 404, etc. may adjust the presentation of UI 402, for example, to include Figure 4B-4D The elements depicted in .

[0140] UI element 404 may include, for example, an expansion element 410, an indication of associated products (e.g., how many products are associated with the content item), a visual indication of product 412 (e.g., a visual indication that a transaction or purchase is available, a visual indication that a merchant link for the product is available, etc.), etc. User interaction with one or more components of UI element 404 may cause device 400A to modify the presentation of UI element 404, for example, user selection of expansion element 410 may cause the presentation of UI element 404 to be modified to an expanded state.

[0141] In response to device 400A providing content to a platform (eg, Figure 1400A sends a request for a content item to a content providing platform 120 of the present invention, and a UI including UI element 404 may be presented. UI element 404 may be presented in a home feed (e.g., a list of suggested content items for a user or user account). UI element 404 may be presented in a suggested feed (e.g., a list of suggested content items based on one or more recently presented content items). UI element 404 may be presented as part of a playlist (e.g., a list of content items to be presented populated by a user, a creator, etc.). UI element 404 may be presented in a search feed (e.g., in response to a user-generated search query sent to a content providing platform by device 400A). For example, UI elements associated with one or more products may be presented based on the inclusion of products, product categories, product brands, etc. included in the search query. UI element 404 may be presented in a product-focused feed (e.g., a shopping content feed). UI element 404 may be responsive to a user's selection of a content item, for example, it may be displayed after a user selects to watch an associated video, presented together with an associated content item, etc. UI element 404 may be presented based on a detected user's interest in a content item. For example, a user may pause on a thumbnail of a video (e.g., a user may pause a cursor on a thumbnail, a user may pause scrolling of the presented thumbnails, etc.). When the pause satisfies one or more conditions (e.g., a duration condition, a thumbnail position condition, etc.), UI element 404 may be presented. UI element 404 may be presented in response to additional data (e.g., user account history, user settings, user preferences, etc.). UI element 404 may be presented with a list of content items to be presented, UI element 404 may be presented when a content item is presented (e.g., when a video associated with a product of UI element 404 is played), etc.

[0142] Figure 4B An apparatus 400B is depicted presenting an example UI 420 including a UI element 422 indicating associated products in accordance with some embodiments. The UI element 422 includes information about one or more products associated with a content item. The UI element 422 includes information about one or more products associated with a content item. Figure 4B424 is presented in an expanded state. UI element 422 may include multiple components. For example, a first component may include information about a first product (e.g., a picture, a product name, a price, a timestamp, etc.), a second component may include information about a second product, etc. In some embodiments, UI element 422 may be scrollable, for example, to access information about additional products. UI element 422 may include multiple tabs (e.g., UI element 422 may be associated with product information and one or more other types of information). For example, UI element 422 may include a product tab 424 and a chapter tab 426. In some embodiments, product tab 424 may be open by default (e.g., the content of product tab 424 may be presented by default). In some embodiments, chapter tab 426 may be open by default (e.g., the content of chapter tab 426 may be presented by default). In some embodiments, another tab may be opened by default. For example, section tabs 426 may be opened by default in addition to content items with associated products, and the tabs that are opened by default may be selected based on a user's history (e.g., interactions with elements of section tabs, product tabs, etc.), based on a search query (e.g., a search including a product name, a related term or phrase such as "product review", etc.), etc. UI elements 422 may include additional elements, such as elements that a user may use to control the presentation of UI 420, UI elements 422, etc. For example, UI elements 422 may include a "close" element to present UI 420 without UI element 422, may include a "collapse" element to display less information (e.g., collapse the panel to display the equivalent of Figure 4A UI element 404, modifying the presentation of UI element 422 to a collapsed state, etc.), one or more elements to display more information (for example, one or more listed products, listed product icons, etc. can be selected to present additional information about the products to facilitate purchasing the products, etc.), and the like.

[0143] The product tab 424 of the UI element 422 may include one or more product images, information about the product (e.g., product name, product description, etc.), one or more prices of the product, timestamps of content items related to the product, etc. In some embodiments, the product information may be provided by a content creator, one or more users, a system administrator, etc. In some embodiments, the product information may be retrieved by one or more models (e.g., machine learning models). For example, the presence of a product in a content item, the association of a product in a content item, the timestamp or location where a product appears in a content item, etc. may be determined by one or more machine learning models. Figure 3BThe system 300B of the system 300B determines one or more products associated with a content item (e.g., a video). A portion of the content item related to the content item can be determined (e.g., a timestamp), for example, via the time of a subtitle associated with the product, the display time of an image or frame including the product in the video, etc. In some embodiments, selecting a product can cause presentation of a portion of the content item associated with the product, can cause presentation of a content item starting at a time indicated by a timestamp associated with the product, and so on.

[0144] In some embodiments, the portion of UI element 422 associated with a particular product (e.g., a visual component) can be displayed by default, displayed in a different manner (e.g., highlighted), etc. For example, upon receiving a search query from a user that includes a product name, UI element 422 can be displayed, including a display associated with the searched product.

[0145] In some embodiments, one or more associations between content items and products may be stored, for example, as metadata associated with the content items (the metadata associated with the content items may further include content item titles, descriptions, presentation history, subtitles associated with the content items, etc.). In response to the device (e.g., device 400B) executing instructions to display a list of content items for presentation, presenting content items to a user, presenting a UI element (e.g., UI element 422) including information about one or more products, etc., the device may retrieve information about the products based on the metadata associating the products with the content items. Information about the products (e.g., images, associated products (such as color variants), availability, prices, etc.) may be retrieved from a data repository. The data repository may include information about the products and may be updated, for example, when information such as product prices changes, the UI may retrieve updated information based on the content item / product association and display the updated information.

[0146] UI element 422 may be presented as part of a home feed (e.g., a list of suggested content items for a user or user account), a suggestion feed (e.g., a list of suggested content items based on one or more recently presented content items), a playlist, a search result list, a shopping content page, etc. UI element 422 may be presented when a content item is selected for presentation, when a content item is presented, etc. UI element 422 may be presented when a content item is paused (e.g., pausing scrolling on a content thumbnail, when scrolling on a less detailed element (e.g., Figure 4AIn some embodiments, UI element 422 may be removed or replaced based on user actions, for example, while scrolling, UI element 422 may be collapsed to be equivalent to UI element 404 (e.g., to facilitate selection of content items from a list of content items, to simplify scrolling a list of content items, etc.). UI element 422 may be displayed when presenting a list of content items, when presenting a single content item (e.g., when playing a video associated with a product of UI element 422), etc.

[0147] Figure 4C Depicted is an apparatus 400C presenting an example UI 430 including a UI element 432 presenting a content item and a UI element 434 presenting information about an associated product in accordance with some embodiments. Figure 4B In some embodiments, UI element 434 may include one or more control elements, such as a back button for returning to a previous view, a close button for closing UI element 434 and viewing a different set of UI elements (e.g., unrelated to the product), and the like. In some embodiments, UI element 434 may be removed from UI 430 in response to another user action (e.g., a user scrolling past an associated content item). In some embodiments, a user selecting a product presented via UI 434 may prompt the presentation of a UI element that facilitates the purchase of the product. In some embodiments, UI element 434 may be displayed in response to determining a user's interest in one or more products associated with a content item (e.g., based on user history, based on one or more terms in a user's search query, based on a user selection to browse and / or present shopping content items, based on a user selection of a product or a UI element associated with a product, etc.).

[0148] UI element 434 may include one or more pictures and / or additional information about one or more products associated with a content item (e.g., a content item presented via UI element 432). Pictures and / or information may be provided by a content creator, one or more users, and may be retrieved from a database (e.g., based on product / content item associated metadata), etc. In some embodiments, UI element 432 may scroll automatically. For example, UI element 432 may scroll when a content item is presented, such as making a product associated with a part of a currently presented content item visible. UI element 432 may be presented in response to a user's selection of a content item to be presented. UI element 432 may be presented in response to other factors (e.g., user history). UI element 432 may be presented when a user stays on an associated content item, an associated UI element, etc.

[0149] In some embodiments, UI element 434 may present information about a single product, such as that requested by a user (e.g., via Figure 4B UI element 422) can be a selected product. UI element 434 can be product-focused, can focus on a single product, can display product variations (e.g., such as combining Figure 4D ), or may include other elements, components and / or information described with respect to other UI elements described herein.

[0150] Figure 4D An apparatus 400D is depicted presenting an example UI 440 according to some embodiments, the example UI including a UI element 442 presenting a content item and a UI element 444 facilitating a transaction associated with a product. The UI element 444 may provide one or more fields associated with a user conducting a transaction (e.g., purchasing a product). For example, the UI element 444 may include an alternative product panel 446 that may include information, pictures, prices, etc. about alternative products (e.g., products related to one or more products associated with the content item, such as color variations, size variations, variations in product bundling, related products (such as similar products from another brand, etc.)).

[0151] UI elements 444 may include transaction elements 448. In some embodiments, transaction elements 448 may facilitate a transaction (e.g., a purchase) within application-provided UI 440. In some embodiments, transaction elements 448 may facilitate a transaction via another application, another website, etc. For example, interaction with transaction elements 448 may direct the user to a merchant website, may direct device 400D to open an application associated with purchasing a product, and so on.

[0152] UI element 444 may be navigable, scrollable, etc. UI element 444 may include one or more control elements, such as a back button for returning to a previous view, a close button for closing UI element 444 and displaying a different set of UI elements via UI 440, etc. In some embodiments, UI element 444 may be displayed as part of a list of content items for the user to select, as part of UI 440 presenting content items, etc. UI element 444 may be responsive to determining a user's interest in a transaction (e.g., purchase) associated with one or more products included in the content item (e.g., from a UI element such as Figure 4C The UI element 434 may be displayed based on selecting a product in a shopping list, selecting a content item, hovering over a content item, including a product or product-related term in a search query, the user navigating to a shopping-focused content item list, etc. The UI element 444 may receive information about one or more products from a database, such as based on metadata associating a content item with one or more products.

[0153] Figure 4A-4D The UI elements depicted in can be integrated into various configurations. For example, a UI element such as UI element 404 can be presented, an element such as UI element 422 can be presented when a user interacts with UI element 404, UI element 434 can be presented when a user interacts with UI element 422, UI element 444 can be presented when a user interacts with UI element 434, and the like. One or more UI elements may include navigation elements for guiding the device to display different UI elements, for example, interacting with expansion element 410 may cause a UI element such as UI element 422 to be presented, and interacting with different elements of UI element 404 may cause UI elements such as UI element 434, UI element 444, and the like to be presented.

[0154] Other connections between UI elements are possible, for example, interaction with a UI element such as UI element 404 may cause a UI element such as UI element 422, such as UI element 434, such as UI element 444, etc. to be displayed. Interaction with a UI element such as UI element 422 or a portion thereof may cause a UI element such as UI element 404, such as UI element 434, such as UI element 444, etc. to be presented. User interaction with a UI element such as UI element 434 or a portion thereof may cause a UI element such as UI element 404, UI element 422, such as UI element 444, etc. to be presented. User interaction with a UI element such as UI element 444 or a portion thereof may cause a UI element such as UI element 404, such as UI element 434, such as UI element 444, etc. to be presented. The default UI element presented may depend on the context in which the UI element is presented (e.g., a list of content items presented in response to a search, a home feed, a shopping feed, a viewing feed, etc.; an context including content items being presented; etc.). For example, the selection of the form of a UI element may be based on multiple factors. In some embodiments, including a product name, category, etc. in a search query can change the default UI elements, e.g., can cause the UI to default to presenting a UI element that includes information about the product, a UI element that includes a purchase option for the product, etc. The determination of the form of the UI element to be displayed can be based on user history, user account history, user actions (e.g., opening a home feed, presenting a viewing feed, transmitting a search query, selecting a shopping feed, etc.). Transitions between forms of UI elements associated with a product can be determined by additional data similar to the data used to determine the form of the UI element presented.

[0155] Figure 4E An example device 400E is depicted with UI elements superimposed on content presentation elements 452 in accordance with some embodiments. Device 400E includes UI 450. UI 450 may be provided by an application (e.g., an application associated with a content providing platform). Presentation element 452 may present a content item (e.g., a video). UI 450 may present a list of additional content items, additional information associated with the presented content item (e.g., a title, description, comments, live chat, etc.), additional UI elements associated with a product (e.g., such as UI elements 404, 422, 434, 444, or variations of UI elements), and the like.

[0156] Presentation element 452 may be overlaid with one or more UI elements. UI element 454 may indicate a product included in the content item (e.g., shown in the video). UI element 454 may perform actions related to Figure 4A-4DThe UI element 454 may have similar functions to other UI elements in the content item, for example, information about the product may be presented, further information about the product may be displayed, the product may be promoted for purchase, etc. The placement of the overlay element may be determined by one or more users, content creators, models (e.g., a machine learning model configured to detect products or objects in an image may be used to avoid content item display areas including objects), etc. UI element 454 may include a visual indicator of where one or more related products are located in the content item (e.g., where in the video). UI element 454 may be presented during the presentation of the content item (e.g., while the video is playing). UI element 454 may be displayed and / or removed in response to the presence of related products in the content item, for example, it may be displayed when the content item is in the video. UI element 454 may indicate multiple products, may identify these products (e.g., may name one or more products), may display information about the product, etc. Multiple UI elements (such as UI element 454) may be displayed, for example, on a thumbnail of the video, during the entire presentation of the video, during the video, etc.

[0157] Overlaid UI elements (such as UI element 454) may be presented in combination with other UI elements associated with products, for example, UI element 456 may open a panel including information about multiple products associated with a content item, UI element 454 may cause a UI element including information about a pictured product to be displayed, and so on.

[0158] The superimposed UI element 454 can be displayed on the visual representation of the content item (e.g., in front of the visual representation of the content item, with visual priority, etc.). For example, UI element 454 can be superimposed on a video thumbnail. UI element 454 can be displayed on the content item. For example, UI element 454 can be superimposed on the video being played. The presentation of UI element 454 can be performed in response to a user action. For example, after determining that the user is interested in one or more products (e.g., via a search query, via interaction with a product-related UI element, via user history, etc.), UI element 454 can be superimposed on another UI element. In some embodiments, the content item can be a live video. In some embodiments, the content item can be a short video.

[0159] In some embodiments, UI element 454 may perform similarly to the performance description of UI element 404, for example, a user may be notified that one or more products are associated with a content item. UI element 454 may respond to interactions from a user similarly to UI element 404, for example, a panel including product information may be opened or expanded, the presentation of UI element 404 may be modified to display more or different information, the UI element may be expanded to include more information, the presentation of a content item may be initiated, etc. UI element 454 may respond to interactions from a user similarly to UI element 434, for example, a panel that facilitates a transaction may be opened or expanded.

[0160] Figure 5A-5F 1 is a flow chart of methods 500A-F related to content items having associated products according to some embodiments. Methods 500A-F may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, processing devices, etc.), software (such as instructions running on a processing device, a general purpose computer system, or a dedicated machine), firmware, microcode, or a combination thereof. In some embodiments, methods 500A-F may be performed in part by Figure 1 The method 500A may be performed in part by the product identification system 175 (e.g., Figure 1 The server machine 170 and the data set generator 172, Figure 2 According to an embodiment of the present disclosure, the product identification system 175 may use method 500A to generate a data set to at least one of train, validate, or test a machine learning model. Methods 500B-D may be performed by the product identification system 175 (e.g., Figure 3B The method 500E may be performed by the client device 110. The client device 110 may utilize the method 500E to display one or more UI elements associated with a product, for example, to facilitate a user to identify a product included in a content item. The method 500F may be performed by the content platform system 102, for example, it may be performed by the processing logic of the content providing platform 120 to facilitate the client device 110 to present one or more UI elements associated with a product. In some embodiments, a non-transitory machine-readable storage medium stores instructions that, when executed by a processing device (e.g., of the product identification system 175, the server machine 180, etc.), cause the processing device to perform one or more of the methods 500A-F.

[0161] For ease of explanation, methods 500A-F are depicted and described as a series of operations. However, operations according to the present disclosure may occur in various orders and / or concurrently with other operations not presented and described herein. In addition, not all of the operations shown may be performed to implement methods 500A-F according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that methods 500A-F may alternatively be represented as a series of interrelated states via state diagrams or events.

[0162] Figure 5A is a flow chart of a method 500A for generating a data set for a machine learning model according to some embodiments. Figure 5A In some embodiments, at block 401, the processing logic implementing method 500A initializes the training set T to an empty set.

[0163] At block 502, processing logic generates a first data input (e.g., a first training input, a first validation input), which may include one or more of product data, image data, metadata, text data, confidence data, etc. In some embodiments, the first data input may include a first feature set for a data type, and the second data input may include a second feature set for a data type (e.g., a first training input, a first validation input). Figure 3A In some embodiments, the input data may include historical data.

[0164] In some embodiments, at block 503, the processing logic optionally generates a first target output for one or more data inputs (e.g., a first data input). In some embodiments, the input includes one or more predicted products detected in the content item and an associated confidence interval, and the target output may include a label of a product included in the content item. In some embodiments, the input includes one or more data sets associated with the content item (e.g., image data such as a video frame or a portion of a video frame, metadata such as title text or subtitle text, etc.), and the target output is a list of products included in the content item. In some embodiments, the first target output is the predicted data. In some embodiments, for a machine learning model configured to correct subtitles by including product information, the input data may be in the form of subtitle text data, and the target output may be a list of possible corrections to the subtitles to include product names / references. In some embodiments, no target output is generated (e.g., an unsupervised machine learning model is able to group or find correlations in the input data, rather than requiring a target output to be provided).

[0165] At block 504, processing logic optionally generates mapping data indicating an input / output mapping. The input / output mapping (or mapping data) may refer to a data input (e.g., one or more data inputs described herein), a target output for the data input, and an association between the data input and the target output. In some embodiments, such as when associated with a machine learning model that does not provide a target output, block 504 may not be performed.

[0166] In some embodiments, at block 505 , processing logic adds the mapping data generated at block 504 to data set T.

[0167] At block 506, processing logic determines whether the dataset T is sufficient for training, validating, and / or testing a machine learning model (such as Figure 1 If so, execution continues to block 507, otherwise, execution continues back to block 502. It should be noted that in some embodiments, the adequacy of the dataset T can be determined simply based on the number of inputs in the dataset (which are mapped to outputs in some embodiments), while in some other embodiments, the adequacy of the dataset T can be determined based on one or more other criteria (e.g., diversity measures of data examples, accuracy, etc.) in addition to or instead of the number of inputs.

[0168] At block 507, processing logic provides a data set T (e.g., Figure 1180) to train, validate and / or test the machine learning model 190. In some embodiments, the data set T is a training set and is provided to the training engine 182 of the server machine 180 to perform training. In some embodiments, the data set T is a validation set and is provided to the validation engine 184 of the server machine 180 to perform validation. In some embodiments, the data set T is a test set and is provided to the testing engine 186 of the server machine 180 to perform testing. For example, in the case of a neural network, the input values ​​of a given input / output mapping (e.g., the numerical values ​​associated with the data input 210) are input to the neural network, and the output values ​​of the input / output mapping (e.g., the numerical values ​​associated with the target output 220) are stored in the output nodes of the neural network. The connection weights in the neural network are then adjusted according to a learning algorithm (e.g., back propagation, etc.), and the process is repeated for other input / output mappings in the data set T. After block 507, the model (e.g., model 190) may undergo at least one of the following operations: being trained using the training engine 182 of the server machine 180, being validated using the validation engine 184 of the server machine 180, or being tested using the testing engine 186 of the server machine 180. The trained model may be implemented by the product recognition system 175 to generate output data, e.g., used by the product information platform 161 to provide product data to a user, provided to a fusion model, used to update metadata for a content item to include one or more product associations, etc.

[0169] Figure 5B 5 is a flow chart of a method 500B for updating metadata for a content item according to some embodiments. At block 510, processing logic (e.g., a processing device, a computer processor, etc.) receives first data. The first data includes a first identifier of a first product determined in association with the content item based on the metadata of the content item (e.g., an indicator, a pointer to further data, a code identifying the product, etc.). The content item can be or include visual content, audio content, textual content, video content, etc. The metadata of the content item can be provided to one or more trained machine learning models (e.g., Figure 3B The first data may further include a first confidence value associated with the first product and the content item. The first confidence value may indicate the likelihood that the first product is associated with the first content item, for example, the likelihood that the first product appears or is referenced in the metadata of the content item. The first data may further include an identifier of a second product, and a second confidence value associated with the second product. The first data may include a list of products and associated confidence values.

[0170] At block 512, processing logic receives second data including a second identifier of the first product. The second identifier has been determined based on image data of the content item (e.g., one or more frames of a video, portions of one or more images, etc.) associated with the content item. The second data also includes a second confidence value associated with the first product and the content item. The confidence value and the identifier may be generated by one or more machine learning models (e.g., Figure 3B The machine learning model may include a system configured to reduce the dimensionality of the image data. One or more candidate product images may be subjected to dimensionality reduction (e.g., converting the image to a vector of values ​​via a trained machine learning model). The machine learning model may perform operations including comparing the reduced dimensionality image data from the content item with the reduced dimensionality product images of the data repository, for example, to determine a likelihood that the image includes a product. The second data may include a list of products (e.g., product identifiers) and a list of confidence values, which include at least the first product. The confidence value may indicate the likelihood that the associated product appears in the content item, is referenced by the content item, etc.

[0171] In some embodiments, the images are analyzed for potential products included in one or more images of the content item. The existence of the potential product can be verified, for example, by providing an image of the content item for further product image detection analysis (e.g., different frames, additional frames, etc. of the video content item), by text or metadata verification, etc. For example, further analysis of verification for the candidate product can be performed after the candidate product is found, for example, other evidence of the identified product can be searched. In some embodiments, text data and / or metadata associated with the content item can be analyzed for potential / candidate products. The existence of the potential product can be verified, for example, by image-based verification, text verification, etc.

[0172] In some embodiments, the processing logic may be further provided with one or more timestamps, such as a timestamp of a frame in a video having a detected candidate product, a timestamp of a subtitle associated with the video or audio content in which the detected candidate product appears, etc. The processing logic may utilize the timestamps for further analysis to adjust metadata for the content item, to generate a UI element associated with the presentation of the content item, and the like.

[0173] At block 514, processing logic provides the first data and the second data to a trained machine learning model. The trained machine learning model may be a fusion model. The trained machine learning model may be provided with one or more product lists with associated confidence values.

[0174] At block 516, processing logic receives a third confidence value associated with the first product from the trained machine learning model.In some embodiments, processing logic may receive a list of confidence values ​​associated with a list of products including the first product.

[0175] At block 518, processing logic adjusts metadata associated with the content item in view of the third confidence value. In some embodiments, adjusting the metadata may include adding one or more connections between the content item and the product to the metadata. For example, adjusting the metadata may include adding an indication that a particular product is associated with the content item, is featured in the content item, is included in the content item, is advertised by the content item, etc. Adjusting the metadata may include adjusting the subtitles of the content item, for example, to include one or more references to the product that was erroneously transcribed during subtitle generation.

[0176] Figure 5C 5 is a flow chart of a method 500C for training a machine learning model associated with a content item product pairing according to some embodiments. In some embodiments, the machine learning model trained using the method 500C can be a fusion model. Similar methods can be utilized to train different models associated with a media item product pairing, such as an image recognition model, an image verification model, a text recognition model, a subtitle update model, etc.

[0177] At block 520, processing logic receives product image data associated with a plurality of content items. The product image data may include data associating products with content items, where the association may be derived from one or more images (e.g., video frames). The product image data includes an indication of one or more products (e.g., potential products, candidate products) detected (e.g., determined) in the images and one or more product image confidence values.

[0178] At block 522, processing logic receives product text data associated with a plurality of content items. The product text data may include data associating products with the content items. The association may be derived from text associated with the content items (e.g., metadata associated with the content items). The product text data includes indications of one or more products (associated with the content items) detected in the text and one or more product text confidence values.

[0179] The data received (or in some embodiments acquired) by the processing logic at blocks 520 and 522 may be used as training input to train the fusion model. Training the machine learning model to perform different functions may include the processing logic receiving different data as training input.

[0180] At block 524, processing logic receives data indicating products included in the plurality of content items. For example, each of the plurality of content items used to train the model (e.g., data associated with the content items may be used to train the model) may include a list of associated products, e.g., tagged by one or more users, tagged by a content creator, etc. The data received by processing logic at block 524 may be used as a target output for training the fusion model. Training the machine learning model to perform different functions may include processing logic receiving different data as target outputs.

[0181] At block 526, processing logic provides the product image data and the product text data to the machine learning model as training inputs. Processing logic may provide different types of data to train different machine learning models. In some embodiments, a machine learning model for frame selection may be trained by providing video frames to the model as training inputs. In some embodiments, a machine learning model for object detection may be trained by providing images (possibly including products) to the machine learning model as training inputs. A machine learning model for embedding may be trained by providing one or more images of an object (e.g., a product) to the model as training inputs. In some embodiments, a text parsing model may be trained by providing text associated with a content item (e.g., metadata) as training input. In some embodiments, a model configured to correct subtitles may be provided with machine-generated subtitles as training inputs.

[0182] At block 528, the processing logic provides data indicating products included in the plurality of content items (e.g., a list of products included in each of the plurality of content items) to the machine learning model as a target output. The processing logic may provide different types of data to train different machine learning models. In some embodiments, a machine learning model for frame selection may be trained by providing data indicating which frames of one or more videos include products as a target output. In some embodiments, a machine learning model for object detection may be trained by providing labels of objects in an image to the model as a target output. In some embodiments, a text parsing model may be trained by providing content items referenced by the text of a content item as a target output. In some embodiments, a corrected subtitle (e.g., including one or more products) may be provided as a target output to a model configured to correct subtitles. In some embodiments, no target output is provided to train a machine learning model (e.g., an unsupervised machine learning model).

[0183] Figure 5D5 is a flow chart of a method 500D for adjusting metadata associated with a content item according to some embodiments. At block 530, processing logic obtains first metadata associated with the content item. The metadata may include text data. The metadata may include a content item title, description, subtitles, comments, real-time chat, etc. At block 531, processing logic provides the first metadata to a first model. In some embodiments, the model is a trained machine learning model. In some embodiments, the model is a product detection model, for example, the model is configured to receive metadata and generate an indication of a product associated with the content item (e.g., in view of the metadata).

[0184] At block 532, processing logic obtains a first product identifier based on the first metadata and a first confidence value associated with the first product identifier as an output of the first model. The product identifier can be an ID number, an indicator, a product name, or any data that (uniquely) distinguishes a product. The first product identifier can identify the first product. In some embodiments, processing logic can obtain a list of products (e.g., candidate products, potential products) and associated confidence values.

[0185] At block 533, processing logic obtains image data of the content item. In some embodiments, the image data may include or be extracted from one or more video frames. In some embodiments, the image data may be obtained from an object detection model. In some embodiments, the image data may include one or more products associated with the content item.

[0186] At block 534, processing logic provides the image data to a second model. In some embodiments, the second model is a machine learning model. In some embodiments, the second model is a model configured to identify a product from an image. In some embodiments, the second model is a model configured to verify the presence of an identified product from an image. In some embodiments, the second model may reduce the dimensionality of the provided image data. In some embodiments, the second model may compare the reduced dimensionality image data to second reduced dimensionality image data (e.g., retrieved from a data repository, output by a machine learning model, etc.).

[0187] At block 535, processing logic obtains a second product identifier based on the image data and a second confidence value associated with the second product identifier as an output of the second model. In some embodiments, the second product identifier indicates a second product. In some embodiments, the second product is the same as the first product. In some embodiments, processing logic may obtain a list of products and associated confidence values.

[0188] At block 536, processing logic provides data including the first product identifier, the first confidence value, the second product identifier, and the second confidence value as input to a third model. The third model may be a fusion model.

[0189] At block 537, processing logic obtains a third product identifier and a third confidence value as an output of the third model. In some embodiments, the third confidence value may indicate the likelihood that the product indicated by the third product identifier is associated with (e.g., present in) the content item. In some embodiments, the third model may output a list of products and associated confidence values. In some embodiments, the third product identifier identifies a third product. In some embodiments, the third product is the same as the second product. In some embodiments, the third product is the same as the first product. In some embodiments, the first, second, and third products are all the same product.

[0190] At block 538, processing logic adjusts second metadata associated with the content item in view of the third product identifier and the third confidence value. Adjusting the metadata may include supplementing the metadata with one or more product associations (e.g., indications of the associated products). Adjusting the metadata may include updating the subtitles, for example, to include products that may have been erroneously transcribed (e.g., erroneously transcribed by a machine-generated subtitle model). In some embodiments, processing logic may further receive one or more timestamps associated with the content item and the one or more products (e.g., video times at which the products were detected in the video image). Updating the metadata may include adding to the metadata an indication of the time at which the products were found in the content item.

[0191] Figure 5E 5 is a flowchart of a method 500E for presenting a UI element associated with one or more products according to some embodiments. At block 540, processing logic (e.g., of a user device, a client device, etc.) presents a UI. The UI includes one or more graphical representations of one or more content items (e.g., a video). A graphical representation of a content item (e.g., a video thumbnail) can be selected to start the presentation of an associated content item. One or more graphical representations of a content item can be displayed using a UI element associated with one or more products. Each graphical representation of a corresponding content item can be displayed together with a UI element associated with one or more products. The UI element can be presented / displayed in a folded state (e.g., a folded default state). The UI element includes information identifying a plurality of products covered by the corresponding content item. The UI element can identify one or more products associated with the content item. The UI element can identify one or more products associated with the content item (e.g., covered in a video) (e.g., via name, picture, etc.).

[0192] The UI may present a selectable graphical representation of a content item. The represented content item may be part of a home feed, provided in response to a search, may be part of a watch list, may be part of a play list, may be part of a shopping feed, and the like. In some embodiments, the UI element may be superimposed on top of and / or in front of one or more other elements of the UI. For example, the UI element (e.g., in a collapsed state) may be superimposed on a graphical representation of a content item, may be superimposed on a content item (e.g., while the content item is being presented), and the like.

[0193] At block 542, in response to the user's interaction with the UI element in the folded state, the processing logic continues to facilitate the presentation of the graphical representation of the corresponding video, while modifying the presentation of the UI element from the folded state to the expanded state. Interaction with the UI element may include selecting the UI element. Interaction with the UI element may include staying on the UI element (e.g., placing a cursor on the UI element, scrolling to the UI element and pausing scrolling, etc.). The UI element in the expanded state may include multiple visual components. Each visual component may be associated with a product in a plurality of products. Visual components may include pictures, descriptions, prices, timestamps, etc. associated with various products.

[0194] In some embodiments, the UI element (e.g., in an expanded state) may include multiple tabs. For example, the UI element may include tabs for products, tabs for chapters or sections of content items, and so on. For content items with associated products, the UI element may display / open tabs for products by default. The UI element may display tabs for products by default in response to user actions and / or history.

[0195] At block 544, in response to a user selecting one of the multiple visual components of the UI element in the expanded state, processing logic initiates presentation of a corresponding content item covering a product associated with the selected visual component. Processing logic may initiate playing of a video covering the product associated with the selected visual component. Processing logic may initiate presentation of a portion of a content item associated with the product of the selected visual component (e.g., initiate playing of a portion of a video).

[0196] In some embodiments, interaction with a UI element may cause the UI element to be modified to a product focused state. The product focused state may present additional information, detailed information, etc. about one or more products. Interaction with a UI element in a collapsed state may cause the UI element to be modified to a product focused state. Interaction with a UI element in an expanded state (e.g., interaction with a visual component of a UI element associated with a product) may cause the UI element to be modified to a product focused state.

[0197] In some embodiments, interaction with a UI element may cause the presentation of the UI element to be modified to a transaction state. Interaction with a UI element in a collapsed state may cause the presentation of the UI element to be modified to a transaction state. Interaction with a UI element in an expanded state (e.g., selecting a component associated with a product) may cause the presentation of the UI element to be modified to a transaction state. Interaction with a UI element in a product focused state may cause the presentation of the UI element to be modified to a transaction state. Determining whether a selection or interaction of a UI element, UI element component, etc. causes a conversion to a transaction state may be performed based on user history, user preferences, content items, content item feeds (e.g., search results, viewing feeds, etc.), etc.

[0198] In some embodiments, UI elements can be superimposed on the presented content items. For example, when a video is playing, when a video is showing one or more products, etc., a UI element identifying one or more products can be superimposed and displayed on the video. Selection of a superimposed UI element can cause additional UI elements to be displayed, can cause the superimposed UI element to be modified to a different state, can cause a separate UI element to be modified to a different state, and so on. The superimposed UI element can be in a folded state, an expanded state, a product focus state, a transaction state, etc. The presence and / or location of the superimposed UI element can be determined by one or more trained machine learning models (e.g., one or more models configured to detect products).

[0199] Fig. 5F 5 is a flowchart of a method 500F for instructing a device to present one or more UI elements associated with a product according to some embodiments. At block 550, processing logic provides a UI including one or more graphical representations of one or more content items to the device. The graphical representation is provided by the UI of the device for display / presentation. Each graphical representation of the corresponding content item can be selected to initiate the presentation of the corresponding content item. The content item may include a video. The content item may include a live video. The graphical representation may be provided in response to a request of the device. The graphical representation may include a home feed, a viewing feed, a playlist, a search result list, a shopping feed, etc. Instructions sent to the device (e.g., including instructions associated with any step of method 500F), a UI sent to the device, a UI element sent to the device, etc. may be determined / selected based on obtaining the user's history. The user's history may include historical interactions and / or selections of content items, including content items with associated products. The user's history may include historical interactions and / or selections of UI elements or UI element components associated with the product. The user's history may include one or more searches of the user, such as searches including product names. Instructions may be provided to the device in response to the processing logic receiving the user's history.

[0200] One or more of the graphical representations of the content items are displayed as a UI element in a collapsed state. In some embodiments, each graphical representation is displayed as a UI element in a collapsed state. In some embodiments, a subset of the graphical representations are displayed as a UI element in a collapsed state. The UI element in the collapsed state is presented / displayed as a first graphical representation of a first content item. The UI element includes information identifying a plurality of products covered by the first content item. The UI element may identify how many products are associated with the content item, may identify one or more products by name, may identify a category or classification of products covered by the content item, and so on. In some embodiments, the plurality of products are obtained as output from one or more trained machine learning models. The trained machine learning models may be combined with Figure 3B Similar to those described.

[0201] At block 554, in response to receiving an indication that a user interacted with a UI element in a folded state, processing logic causes the device to modify the presentation of the UI element. The presentation of the UI element can be modified from a folded state to an expanded state. The UI element in the expanded state can include multiple visual components, each visual component associated with a product in the multiple products. The visual components can include pictures, names, descriptions, prices, timestamps, etc.

[0202] At block 556, in response to receiving an indication that a user selected one of the multiple visual components of the UI element in the expanded state, processing logic facilitates presentation of a first content item. Processing logic may provide instructions to facilitate presentation of a portion of the first content item associated with a first product (e.g., a product associated with one of the multiple visual components). Processing logic may provide instructions to display a portion of a video associated with the product (e.g., play the video starting from a selected point in the video based on a timestamp associated with the product).

[0203] In some embodiments, the processing logic may further provide instructions to the device to modify the presentation of the UI element to a product focus state. For example, upon selecting a visual component of the UI element in an expanded state, the UI element may be modified to a product focus state. The product focus state may include additional details about one or more products covered by, included in, associated with, and the like, the content item.

[0204] In some embodiments, the processing logic may further provide instructions to the device to modify the presentation of the UI element to a transaction state. The transaction state may facilitate the user to initiate a transaction associated with the product (e.g., purchase the product). The transaction state may be presented in response to a user action, user history, user selection of one or more UI elements, etc. The UI element in the transaction state may include one or more components that facilitate a transaction associated with one or more products.

[0205] Figure 6 600 is a block diagram showing a computer system 600 according to some embodiments. In some embodiments, the computer system 600 can be connected (e.g., via a network, such as a local area network (LAN), an intranet, an extranet, or the Internet) to other computer systems. The computer system 600 can operate in a client-server environment with the capabilities of a server or client computer, or as a peer machine in a peer-to-peer or distributed network environment. The computer system 600 can be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web device, a server, a network router, a switch or a bridge, or any device capable of executing an instruction set (sequential instructions or other instructions) specifying the action to be taken by the device. Further, the term "computer" should include any collection of computers that execute one (or more) instruction sets individually or jointly to execute any one or more of the methods described herein.

[0206] In a further aspect, the computer system 600 may include a processing device 602, a volatile memory 604 (e.g., a random access memory (RAM)), a non-volatile memory 606 (e.g., a read-only memory (ROM) or an electrically erasable programmable ROM (EEPROM)), and a data storage device 618, which may communicate with each other via a bus 608.

[0207] The processing device 602 may be provided by one or more processors, such as a general-purpose processor (such as, for example, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor that implements other types of instruction sets, or a microprocessor that implements a combination of types of instruction sets) or a special-purpose processor (such as, for example, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0208] The computer system 600 may further include a network interface device 622 (e.g., coupled to the network 674). The computer system 600 may also include a video display unit 610 (e.g., LCD), an alphanumeric input device 612 (e.g., keyboard), a cursor control device 614 (e.g., mouse), and a signal generating device 620.

[0209] In some embodiments, the data storage device 618 may include a non-transitory computer-readable storage medium 624 (e.g., a non-transitory machine-readable medium) on which instructions 626 encoding any one or more of the methods or functions described herein may be stored, including instructions for Figure 1Components (e.g., content providing platform 120, other platforms of content platform system 102, communication application 115, model 190, etc.) are encoded and used to implement instructions of the methods described herein.

[0210] The instructions 626 may also reside, completely or partially, within the volatile memory 604 and / or within the processing device 602 during execution of the instructions 626 by the computer system 600, and thus, the volatile memory 604 and the processing device 602 may also constitute machine-readable storage media.

[0211] Although the computer-readable storage medium 624 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" shall include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more executable instruction sets. The term "computer-readable storage medium" shall also include any tangible medium that can store or encode a set of instructions for execution by a computer, which instructions cause the computer to perform one or more of the methods described herein. The term "computer-readable storage medium" shall include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0212] The methods, components, and features described herein may be implemented by discrete hardware components, or may be integrated into the functionality of other hardware components (such as ASICS, FPGAs, DSPs, or similar devices). In addition, these methods, components, and features may be implemented by firmware modules or functional circuits within a hardware device. Further, these methods, components, and features may be implemented by any combination of hardware devices and computer program components or by a computer program.

[0213] Unless otherwise specifically indicated, terms such as "receive," "execute," "provide," "acquire," "cause," "access," "determine," "add," "use," "train," "reduce," "generate," "correct," and the like refer to actions and processes performed or implemented by a computer system that manipulate and transform data represented as physical quantities (electronic quantities) within computer system registers and memories into other data similarly represented as physical quantities within computer system memories or registers or other such information storage, transmission, or display devices. Also, the terms "first," "second," "third," "fourth," and the like as used herein are intended as marks for distinguishing different elements and may not have ordinal meanings according to their numerical designations.

[0214] The examples described herein also relate to a device for performing the methods described herein. The device may be specially constructed to perform the methods described herein, or the device may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0215] The methods and illustrative examples described herein are not inherently related to any particular computer or other device. Various general purpose systems may be used according to the teachings described herein, or more specialized devices may be conveniently constructed to perform each of the methods described herein and / or their respective functions, routines, subroutines, or operations. Examples of structures for a variety of these systems are set forth in the above description.

[0216] The above description is intended to be illustrative rather than restrictive. Although the present disclosure has been described with reference to specific illustrative examples and embodiments, it will be appreciated that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the following claims and the full scope of equivalents to which the claims are authorized.

[0217] References throughout this specification to "one implementation" or "an implementation" mean that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Thus, the appearances of the phrase "in one implementation" or "in an implementation" in various places throughout this specification may, but do not necessarily, refer to the same implementation, depending on the circumstances. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0218] To the extent that the terms "includes," "has," "including," variations thereof, and other similar words are used in the detailed description or claims, these terms are intended to be inclusive in a manner similar to the term "comprising" as an open transitional word and do not exclude any additional or other elements.

[0219] As used in this application, the terms "component", "module", "system", etc. are generally intended to refer to a computer-related entity, which is hardware (e.g., circuitry), software, a combination of hardware and software, or an entity associated with an operating machine having one or more specific functions. For example, a component can be, but is not limited to, a process, a processor, an object, an executable program, an execution thread, a program, and / or a computer running on a processor (e.g., a digital signal processor). For example, both an application running on a controller and the controller can be components. One or more components can reside within a process and / or execution thread, and a component can be confined to one computer and / or distributed between two or more computers. Further, a "device" can appear in the following forms: specially designed hardware; general-purpose hardware specialized by executing software thereon that enables the hardware to perform specific functions (e.g., generating points of interest and / or descriptors); software on a computer-readable medium; or a combination thereof.

[0220] The foregoing systems, circuits, modules, etc. have been described with respect to the interaction between several components and / or blocks. It will be appreciated that such systems, circuits, components, blocks, etc. may include those components or specified subcomponents, some of the specified components or subcomponents, and / or additional components, and according to various permutations and combinations of the foregoing. Subcomponents may also be implemented as components (hierarchical structures) that are communicatively coupled to other components rather than included within a parent component. Additionally, it should be noted that one or more components may be combined into a single component that provides aggregation functionality or divided into several separate subcomponents, and any one or more intermediate layers such as a management layer may be provided to communicatively couple to such subcomponents in order to provide integration functionality. Any component described herein may also interact with one or more other components that are not specifically described herein but are known to those skilled in the art.

[0221] In addition, the words "example" or "exemplary" are used herein to mean serving as an example, instance or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be interpreted as being preferred or advantageous over other aspects or designs. On the contrary, the use of the words "example" or "exemplary" is intended to present concepts in a specific way. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X adopts A or B" is intended to mean any one of the natural inclusive arrangements. That is, if X adopts A; X adopts B; or X adopts both A and B, then "X adopts A or B" is satisfied under any of the aforementioned examples. In addition, unless otherwise specified or clear from the context that it is for a singular form, the articles "one" and "a kind" used in this application and the appended claims should generally be interpreted as meaning "one or more".

Claims

1. A method comprising: presenting a user interface (UI) including one or more graphical representations of one or more videos, wherein each graphical representation of a respective video is selectable to initiate playback of the respective video and is displayed with a UI element in a collapsed state, the UI element including information identifying a plurality of products related to the respective video; In response to a user interaction with the UI element in the collapsed state, continuing to present the graphical representation of the corresponding video while modifying the presentation of the UI element from the collapsed state to an expanded state, wherein the UI element in the expanded state includes a plurality of visual components, each visual component being associated with a product in the plurality of products; and In response to a user selecting one of the plurality of visual components of the UI element in the expanded state, initiating playback of a corresponding video covering a product associated with the selected visual component.

2. The method of claim 1, further comprising, in response to the user selecting one of the multiple visual components of the UI element in the expanded state, modifying the presentation of the UI element from the expanded state to a product focus state, wherein the UI element in the product focus state includes detailed information about the product associated with the selected visual component.

3. The method of claim 1, further comprising: Provide search queries to content platforms; as well as The one or more graphical representations of the one or more videos are received from the content platform.

4. The method of claim 1, further comprising: providing a user request to present a first video to a content platform; as well as The one or more graphical representations of the one or more videos are received from the content platform as recommended videos associated with the first video.

5. The method of claim 1, wherein: The UI element in the expanded state further includes: a first tab associated with the plurality of products; and A second tab page is associated with the plurality of portions of the corresponding video, wherein the content of the first tab page is presented by default.

6. The method of claim 1, further comprising modifying the presentation of the UI element to a transaction state in response to a user selecting a component of the UI element associated with the one of the plurality of products, wherein The UI element in the transaction state facilitates a transaction associated with the one of the plurality of products.

7. The method of claim 1, wherein: The UI element in the collapsed state is superimposed on one or more of the graphical representation of the corresponding video or the corresponding video.

8. The method of claim 1, wherein: The user interaction with the UI element in the collapsed state includes dwelling on the UI element in the collapsed state.

9. A method comprising: providing a user interface (UI) comprising one or more graphical representations of one or more content items for presentation on a user device, wherein each graphical representation of a respective content item is selectable to initiate presentation of the respective content item and is displayed as a UI element in a collapsed state, the UI element comprising information identifying a plurality of products related to the respective video; In response to receiving an indication of a user interaction with the UI element in the collapsed state, causing presentation of the UI element to be modified from the collapsed state to an expanded state, wherein the UI element in the expanded state comprises a plurality of visual components, each visual component being associated with a product in the plurality of products; and In response to receiving an indication of a user selection of one of the plurality of visual components of the UI element in the expanded state, presentation of a first content item is facilitated.

10. The method of claim 9, further comprising providing instructions to the device to facilitate modifying the presentation of the UI element from the expanded state to a product focus state in response to receiving an indication that the user has selected one of the multiple visual components of the UI element in the expanded state, wherein the UI element in the product focus state includes detailed information about the product associated with the selected visual component.

11. The method of claim 9, wherein: Obtain identifiers of the plurality of products related to the first content item as output from a trained machine learning model, wherein the trained machine learning model is configured to receive data associated with the first content item as input.

12. The method of claim 9, further comprising providing instructions to the device to facilitate modifying the presentation of the UI element to a transaction state in response to a user selecting a component of the UI element associated with one of the plurality of products, wherein The UI element in the transaction state includes one or more components that facilitate a transaction associated with the one of the plurality of products.

13. The method of claim 9, wherein: The instructions facilitating presentation of the first content item include an indication of a portion of the first content item to be presented, the portion of the first content item encompassing a product of the plurality of products associated with one of the plurality of visual components.

14. The method of claim 9, further comprising obtaining a history of a user, the history of the user comprising one or more interactions with content items having associated products, wherein Instructions for providing display of the UI element are executed in response to obtaining the history of the user.

15. The method of claim 9, wherein: The first content item includes a live video.

16. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a processing device to perform operations comprising: Presenting a user interface UI including one or more graphical representations of one or more videos, wherein Each graphical representation of a respective video is selectable to initiate playback of the respective video and is displayed with the UI element in a collapsed state, the UI element including information identifying a plurality of products related to the respective video; In response to a user interaction with the UI element in the collapsed state, continuing to present the graphical representation of the corresponding video while modifying the presentation of the UI element from the collapsed state to an expanded state, wherein the UI element in the expanded state includes a plurality of visual components, each visual component being associated with a product in the plurality of products; and In response to a user selecting a selected one of the plurality of visual components of the UI element in the expanded state, initiating playback of the corresponding video covering a product associated with the selected visual component.

17. The non-transitory machine-readable storage medium of claim 16, wherein: The operation further includes, in response to the user selecting one of the multiple visual components of the UI element in the expanded state, modifying the presentation of the UI element from the expanded state to a product focus state, wherein the UI element in the product focus state includes detailed information about the product associated with the selected visual component.

18. The non-transitory machine-readable storage medium of claim 16, wherein: The operations further include modifying the presentation of the UI element to a transaction state in response to a user selecting a component of the UI element associated with the one of the multiple products, wherein the UI element in the transaction state facilitates a transaction associated with the one of the multiple products.

19. The non-transitory machine-readable storage medium of claim 16, wherein: The UI element in the expanded state further includes: a first tab associated with the plurality of products; and A second tab page is associated with the plurality of portions of the corresponding video, wherein the content of the first tab page is presented by default.

20. The non-transitory machine-readable storage medium of claim 16, wherein: Initiating playback of the corresponding video includes presenting a portion of the corresponding video depicting the product associated with the selected visual component.

Images