Content matching system
The content matching system automatically identifies products in video frames using AI and object recognition, providing an instant and user-friendly shopping experience within video content, addressing the inefficiencies of manual tagging and cookie reliance.
Patent Information
- Application Number
- PCT/AU2024/050582
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
Current systems for identifying products in video content are labor-intensive, not user-friendly, and lack instant gratification, relying on manual tagging and cookies for advertising, which are no longer reliable.
A content matching system that identifies product data in video frames by analyzing selected portions against a product database, using object recognition and AI to match visual characteristics, allowing for real-time, automatic product identification and presentation to users without requiring personal data or cookies.
Enables instant, user-friendly product identification and shopping experiences within video content, enhancing user engagement and revenue generation for digital platforms without relying on manual tagging or cookies.
Smart Images

Figure AU2024050582_05122024_PF_FP_ABST
Abstract
Description
CONTENT MATCHING SYSTEMField of the Invention
[0001] The invention generally relates to a content matching system and, in particular to a system for matching content to video.Background to the Invention
[0002] Consumer content engagement and shopping trends are changing. Consumers want instant gratification from media content such as video streamed from online sources, and also the ability to shop what they want when they see it. Revenue generation is important for digital content platforms and creators, however, changing habits mean that reliance cannot be made solely on old revenue generation techniques such as “programmatic advertising” and “subscriptions”. New revenue streams are therefore desirable to ensure a continued capacity to generate revenue. In addition, brands can no longer rely on cookies and personal data for advertising and marketing purposes.
[0003] Media content, such as video, can present various different products, such as the clothing worn by actors and models, and items shown in on set such as consumer goods. Media content can also show a representation of an experience (e.g., a destination or tour) or service (e.g., tradespersons, legal professionals). Current systems which allow for users viewing such media content, in particular within video media, rely on a separate step of manually identifying what products are shown in the media content and finding specific products which matches it, then manually ‘tagging’ the matching products to a specific point in the video. This is labour intensive, not user friendly, and not instantaneous for the consumer.Summary of the Invention
[0004] According to an aspect of the present invention, there is provided a product data identification method for identifying product data associated with a video presented on a first user device, comprising: identifying a selected portion of a video corresponding to a portion of the video determined based at least in part on a received content identifier, wherein the selected portion comprises at least one video frame of the portion of the video; analysing the selected portion to identify one or more objects present within the at least one video frame of the selected portion; comparing at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product; identifying one or more matching products based on a comparison between one or more object characteristics identified forthe one or more objects and the one or more product characteristics of each of the one or more matching products; and communicating the product data of the, or each, matching product to the first user device or a second user device.
[0005] Optionally, the content identifier comprises a video portion identifier comprising image data corresponding to the at least one video frame of the portion of the video to thereby identify the selected portion.
[0006] Optionally, the content identifier comprises a video identifier comprising information enabling access to the video and a video portion identifier identifying the at least one video frame of the portion of the video, and wherein the method comprises: using the video identifier to access the video; and obtaining the at least one video frame of the portion of the video according to the video portion identifier to thereby identify the selected portion.
[0007] According to another aspect of the present invention, there is provided a product data identification method for identifying product data associated with a video presented on a first user device, comprising: accessing the video identified by a received content identifier; determining at least two selected portions of the video by analysing the video according to a predefined trigger rule; for the at least two selected portions: analysing the selected portion to identify one or more objects present within the at least one video frame of the selected portion, comparing at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product, and identifying one or more matching products based on a comparison between one or more object characteristics identified for the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicating the product data of each matching product identified for each of the at least two selected portions to the first user device or a second user device.
[0008] The product data may comprise a selected portion identifier for each product to thereby enable presentation of the product data of a particular product in association with presentation of the selected portion of the video.
[0009] In the above methods, the video is optionally provided to the first user device by a publisher server, and identifying a selected portion of a video may correspond to a portion of the video comprises accessing the video on the publisher server. The method may comprise receiving the content identifier from the publisher server. The method may comprise communicating the product data of the, or each, matching product to the first user device or a second user device via a communication with the publisher server.
[0010] The above methods optionally comprise: receiving the content identifier from the first user device; and communicating the product data of the, or each, matching product to the first user device.
[0011] The above methods optionally comprise: receiving the content identifier from the first user device; and communicating the product data of the, or each, matching product to the second user device.
[0012] In the above methods, the product data of the, or each, matching product is optionally communicated to the first user device or a second user device via a website accessed by the first user device or the second user device. The website may include the video such that the video is presented on the first user device in response to the first user device accessing the website, in which case the method may comprise: updating the website to include a visual representation of the product data for the, or each, matching product. The visual representation of the product data for the, or each, matching product may be presented in a pop-up on the website and / or adjacent the video. The method may comprise: configuring the website to present the visual representation of each product in dependence on a current playback location of the video, in accordance with the selected portion identifier associated with each product.
[0013] In the above methods, the content identifier is optionally generated in response to a user action on the first user device. The user action may correspond to the user pausing the video, and the content identifier may identify the currently displayed portion of the video. The first user device may present an interactive element selectable by the user, the user action may correspond to the user selecting the interactive element, and the content identifier may identify the displayed portion of the video at the time of the user selecting the interactive element.
[0014] In the above methods, each product record optionally comprises a product characteristic corresponding to a product image, and the, or each, object optionally comprises an object characteristic corresponding to a visual representation determined for the object by reference to the selected portion, and matching products may be identified at least in part in accordance with a visual similarity between the one or more matching products and the visual representations of the one or more objects.
[0015] In the above methods, each product record optionally comprises one or more product characteristics each corresponding to a product property, and the, or each, object is optionally analysed to determine one or more object characteristics corresponding to additional information regarding the object, and the one or more matching products may be identified at least in part in accordance with a similarity between the one or more product characteristics of the one or more matching products and the determined additional information of the one or more objects.
[0016] The above methods may be implemented by a product server.
[0017] According to another aspect of the present invention, there is provided a product server comprising one or more processors interfaced with a memory and a network interface, wherein the one or more processors are configured to communicate with a data network via the network interface, and wherein the memory comprises program instructions configured to cause the one or more processors to: identify aselected portion of a video corresponding to a portion of the video determined based at least in part on a received content identifier, wherein the selected portion comprises at least one video frame of the portion of the video; analyse the selected portion to identify one or more objects present within the at least one video frame of the selected portion; compare at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product; identify one or more matching products based on a comparison between one or more object characteristics identified for between the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicate, via the data network, the product data of the, or each, matching product to a first user device on which the video is presented or a second user device.
[0018] Optionally, the content identifier comprises a video portion identifier comprising image data corresponding to the at least one video frame of the portion of the video to thereby identify the selected portion. The content identifier may comprise a video identifier comprising information enabling access to the video and a video portion identifier identifying the at least one video frame of the portion of the video, and the program instructions may be configured to cause the one or more processors to: use the video identifier to access the video; and obtain the at least one video frame of the portion of the video according to the video portion identifier to thereby identify the selected portion.
[0019] According to still yet another aspect of the present invention, there is provided a product server comprising one or more processors interfaced with a memory and a network interface, wherein the one or more processors are configured to communicate with a data network via the network interface, and wherein the memory comprises program instructions configured to cause the one or more processors to: access a video identified by a received content identifier; determine at least two selected portions of the video by analysing the video according to a predefined trigger rule; for the at least two selected portions: analyse the selected portion to identify one or more objects present within the at least one video frame of the selected portion, compare at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product, and identify one or more matching products based on a comparison between one or more object characteristics identified for the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicate the product data of each matching product identified for each of the at least two selected portions to a first user device on which the video is presented or a second user device.
[0020] Optionally, the product data comprises a selected portion identifier for each product to thereby enable presentation of the product data of a particular product in association with presentation of the selected portion of the video.
[0021] In the above product servers, the video is optionally provided to the first user device by a publisher server, and the product server may be configured to access the video on the publisher server in order to determine a selected portion of the video. The program instructions may be configured to cause the one or more processors to: receive the content identifier from the publisher server. The program instructions may be configured to cause the one or more processors to: communicate the product data of the, or each, matching product to the first user device or a second user device via a communication with the publisher server.
[0022] In the above product servers, the program instructions are optionally configured to cause the one or more processors to: receive the content identifier from the first user device; and communicate the product data of the, or each, matching product to the first user device.
[0023] In the above product servers, the program instructions are optionally configured to cause the one or more processors to: receive the content identifier from the first user device; and communicate the product data of the, or each, matching product to the second user device.
[0024] In the above product servers, the product data of the, or each, matching product is optionally communicated to the first user device or a second user device via a website accessed by the first user device or a second user device. The website may include the video such that the video is presented on the first user device in response to the first user device accessing the website, and the program instructions may be configured to cause the one or more processors to: update the website to include a visual representation of the product data for the, or each, matching product.
[0025] In the above product servers, the visual representation of the product data for the, or each, matching product is optionally presented in a pop-up on the website and / or adjacent the video. The program instructions may be configured to cause the one or more processors to: configure the website to present the visual representation of each product in dependence on a current playback location of the video, in accordance with the selected portion identifier associated with each product.
[0026] In the above product servers, the content identifier is optionally generated in response to a user action on the first user device. The user action may correspond to the user pausing the video, and the content identifier may identify the currently displayed portion of the video. The first user device may present an interactive element selectable by the user, the user action may correspond to the user selecting the interactive element, and the content identifier may identify the displayed portion of the video at the time of the user selecting the interactive element.
[0027] In the above product servers, the program instructions are optionally configured to cause the one or more processors to maintain the product database in the memory.
[0028] In the above product servers, each product record optionally comprises a product characteristic corresponding to a product image, and the, or each, object optionally comprises an object characteristic corresponding to a visual representation determined for the object by reference to the selected portion, and the one or more matching products may be identified at least in part in accordance with a visual similarity between the matching products and the visual representations of the one or more objects.
[0029] In the above product servers, each product record optionally comprises one or more product characteristics each corresponding to a product property, and the, or each, object is optionally analysed to determine one or more object characteristics corresponding to additional information regarding the object, and the one or more matching products may be identified at least in part in accordance with a similarity between the one or more product characteristics of the one or more matching products and the determined additional information of the one or more objects.
[0030] According to still yet another aspect of the present invention, there is provide a product data identification system comprising: a product server as described above; and a first client device associated with a user, the first client device in data communication with the product server via a network, wherein the first client device is configured to: present a video for viewing by the user; and communicate a content identifier to the product server identifying the video and / or a portion of the video being presented on the user device.
[0031] The system optionally comprises a second client device associated with the user in data communication with the product server via the network, and the product server may be configured to communicate the product data of the, or each, matching product to the second user device in response to receiving a content identifier from the first client device. The second client device may execute a web browser application and may be configured to load a website comprising one or more product representations, and each product representation may be associated with a matching product, to thereby receive the product data from the product server. The second client device may execute a native application configured to present on the second user device one or more product representations, and each product representation may be associated with a matching product, to thereby receive the product data from the product server. The first client device may be configured to: present a visible code (such as a QR-code) in response to a user action, the QR-code encoding the content identifier, and the second client device may be configured to: capture an image of the visible code in order to decode the content identifier; and communicate the content identifier to the product server, thereby enabling the first client device to communicate the content identifier to the product server.
[0032] Optionally, the product server is configured to communicate the product data of the, or each, matching product to the first user device in response to receiving a content identifier from the first client device. The video may be presented on a website accessed by the first client device, and the first client device may receive the product data as an update to the website, such that the first user device may present one or more product representations, each product representation may be associated with a matching product, as a pop-up on the website or adjacent to the video.
[0033] The system optionally comprises a publisher server, and the first client device optionally obtains the video from the publisher server.
[0034] Embodiments herein may advantageously enable digital content platforms (such as media content publishers, for example streaming services, online blogs, etc.) to make their media content “come to life”, by making it instantly shoppable. Embodiment may advantageously scan media content (such as a video) being viewed by a user (i.e., a potential customer) to identify one or more objects within the media content (e.g., one or more items discernible within a video), then identifying one or more “contextually relevant” products matching the identified one or more objects to present as advertisements to the user allowing the user a richer experience.
[0035] Advantageously, in an embodiment, this process of identifying one or more objects within a video and identifying one or more matching products is automatic and fed back to the user instantaneously, which may improve their user experience and allow media content publishers to continue making media content (such as videos), without the effort of manually matching products to the media content.
[0036] Embodiments described herein can advantageously leverage artificial intelligence (Al) technology to deliver the one or more matching products by identifying individual objects within an instance of media content (e.g., a particular video being viewed by a user) with similar products available for purchase.
[0037] Advantageously, identifying one or more matching products is not reliant on the use of browser cookies (or other means of tracking a user’s activities on the Internet), or having a record of the user’s personal information.
[0038] The following two broad examples assist in illustrating possible user experiences in relation to embodiments herein:A user may be watching a television show on their television, about home interiors. They have the ability to pause the show and view or purchase similar products (which can be goods, and, in an implementation, can include experiences or services) to what they see on the paused frame.During another show they spot an outfit they like and wish to purchase, and they pause the video. The system identifies the point at which the video has been paused and similar items of clothing are then displayed, which can be view or transacted with.
[0039] In relation to both examples, embodiments may advantageously detect the point at which the video has been paused and compare that portion of the video with a product database to identify similar products (e.g., comprising one or more of other media, product items, goods, services or experiences) to object(s) shown in the paused video, and then serve to the user via dynamic promotions (for example, but not limited to pop-up ads, landing pages, and websites).
[0040] Advantageously, one or more embodiments described herein may facilitate dynamic and real-time identification of products appearing in a video. Embodiments may allow a user to interact with the video to uniquely identify video frames (for example, a single video frame or one or more video frames) or sequences of video frames or other parts of the video, and to receive product information or other information associated with the uniquely identified video frame(s). The product information or other information may relate to products or other information appearing in the identified portion of the video. The product information or other information matches objects or other information which are identified in the uniquely identified video frame(s) (for example, using an object-identification algorithm on the video frame). For example, clothes or homewares may be identified as appearing in the video frame(s).
[0041] Advantageously, one or more embodiments provide the user with an independent and unique interactive experience by identifying objects or information within the video frame(s) that the user has selected. The system may operate dynamically in response to user engagement with the video. This provides the user with complete freedom to select any portion of the video and be provided with up-to-date product information relating to objects appearing in the selected portion of the video in real-time, advantageously providing an interactive user experience. Some embodiments may advantageously operate independently of user interaction, for example, by determining one or more video frames for analysis according to a predefined trigger, and present products or other information relevant to the one or more video frames.
[0042] As used herein, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.Brief Description of the Drawings
[0043] In order that the invention may be more clearly understood, embodiments will now be described, by way of example, with reference to the accompanying drawing, in which:Figure 1 shows a content matching system according to an embodiment;Figure 2 shows a schematic representation of the product server;Figure 3 shows product data identification method implemented by a product server, according to an embodiment;Figures 4A-4E show different configurations of the content matching system according to different embodiments;Figure 5 shows a video analysis method for identifying selected portions of a video, according to an embodiment;Figure 6 shows an example implementation utilised a QR code;Figure 7 shows an example implementation involving online streaming;Figure 8 shows an example architecture flow diagram of the steps performed in an example system; andFigures 9 and 10 show is a flow diagram exemplifying the customer experience during interaction with an example system.Description of Embodiments
[0044] Figure 1 shows a content matching system 10 according to an embodiment. The system 10 comprises a product server 11 is data communication with a network 15. Typically, the network 15 comprises the Internet. Also shown are one or more publisher servers 12 (in the example shown, there are three publisher servers 12a, 12b, 12c). Relevantly, the product server 11 is in data communication with the one or more publisher servers 12 via the network 15. The one or more publisher servers 12 are typically external to the product server 11 (i.e., can be external computing systems). Also shown are one or more user devices 13 (in the example shown, there are three publisher servers 13a, 13b, 13c) is data communication with the network 15, such that the user devices 13 can undertake data communication with the one or more publisher servers 12 and, depending on the embodiment, the product server 11. In general, the one or more user devices 13 are each associated with a user (two or more user devices 13 can be associated with the same user). Users can utilise their associated user device(s) 13 to experience media content so can also be understood to be consumers or customers of media content.
[0045] According to embodiments described herein, the product server 11 is configured to generate product data for display on a user device 12 associated with a user. The generated product data can be associated with one or more products identified by the product server 11, such that the product data comprises information related to the, or each, associated product. A specific instance of product data is generated in part in accordance with media content provided by an associated publisher server 12. A publisher server 12 can be associated with a publisher of media content. In embodiments described herein, the same user device 12a is configured to present both a video provided by a publisher server 12 and product data provided by the product server 11 in dependence on the particular video. In other embodiments described herein, a first user device 12a is configured to present a video provided by a publisher server 12 and a second user device 12b is configured to present the product data provided by the product server 11 in dependence on the particular video. In said embodiments, the first user device 12a is typically associated with the same user as the second user device 12b.
[0046] Regarding terminology as used herein, with reference to the figures, a general reference to a feature of the described embodiments is indicated by a number. In cases where it is necessary to distinguish between two instances of a feature, a lowercase suffix is appended to the number. Similarly, labels such as “first” and “second” may be utilised where necessary to distinguish equivalent features. For example, reference can be made to a user device 13 where it is not required to distinguish between several different user devices 13. In contrast, the present disclosure may refer to a first user device 13a and a second user device 13b where necessary to make such a distinction.
[0047] In a general sense, the product server 11 corresponds to a computer executing program code arranged to cause the product server 11 to implement the functionality herein described. Figure 2 shows a schematic representation of the product server 11, which comprises one or more processors 20 interface with a memory 21 and a network interface 22. The one or more processors 20 can correspond to a single central processing unit (CPU) microchip or a plurality of CPU microchips sharing a high-speed data bus within a dedicated physical server environment. The one or more processors 20 can also represent a distributed computer server environment, for example, the product server 11 can be implemented as a logical or virtual server within a cloud computing environment such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure. The one or more processors 20 can optionally include one or more graphic processing units (GPU), in particular, in terms of implementing machine learning (e.g., artificial intelligence) algorithms where applicable.
[0048] The memory 21 typically comprises both a volatile working memory into which computer programs are loaded for execution by the one or more processors 20 and in which runtime data is stored, and a non-volatile storage memory (for example, selected from one or more of a magnetic memory, solid state memory, and optical memory) for storing data related to operation of the product server 11.
[0049] The network interface 22 provides for data communication between the one or more processors 20 and the one or more publisher servers 12 and / or the one or more user devices 13. Typically, the data communication is with the network 15, being a packet switching network for example implementing the Internet Protocol. For example, the network 15 can comprise (at least in part) the Internet implementing TCP / IP. The network interface 22 can comprise one or more wired (such as Ethernet) and / or one or more wireless network interfaces (such as according to the various IEEE 802.i l standards). The network interface 22 can also, or alternatively, provide data communication via a non-packet network interface, such as a serial data bus (for example, USB). However, generally, it is expected that at least a portion of the data communication between the one or more processors 20 and the one or more publisher servers 12 and / or the one or more user devices 13 utilises a packet switching network.
[0050] Referring back to Figure 1, in a general sense, a user device 13 is a computing device configured for presenting either or both received product data and received video. The user device 13 typically comprises at least one display configured to present visually the product data and received video. The user device 13 also typically comprises other presentation means including one or more speakers for output audio. In an implementation, the video comprises video data including both a visual display component and an audio display component. The user device 13 comprises a user input means, such as one or more of a touchscreen display (typically a feature of the display), a keyboard, and a mouse. In an implementation, a user device 13 is a general -purpose computing device such as a personal computer, television (in particular, a so-called “smart TV”), smartphone, or tablet (although these examples are not intended to be limiting). The functionality of the user device 13 described herein can be associated with one or more discrete computer programs, such as one or more “apps” obtained from one or more “app stores”, such that it is the execution of the computer program (or programs) which causes the executing general-purpose computing device to function as a user device 13 as described herein.
[0051] Still referring to Figure 1, the one or more publisher servers 12 are representative of online content providers, for example, providers of entertainment media (e.g., movies, TV shows, live sports, video blogs, etc.). In terms of the embodiments described herein, it is assumed that the one or more publisher servers 12 are configured to provide videos to a user device 13 (in some cases, this can be termed “streaming”, “streaming content”, or “video stream”), which is communicated from the one or more publisher servers 12 to the one or more user devices 13 when required (e.g., “on demand”). Although shown in Figure 1 as discrete elements, the embodiments described herein are not intended to be limited to any specific configuration of the one or more publisher servers 12. Instead, the specific publisher servers 12a- 12c shouldbe understood as representative of different providers of media content (e.g., as understood by the consumer of the media content — the “identity” of a specific publisher server 12a- 12c can be associated with a company or other organisation or even a specific brand such that a single company or organisation can be associated with several different specific publisher servers 12a- 12c).
[0052] Referring to Figure 3, one or more embodiments described herein relate to a product data identification method implemented by the product server 11.
[0053] At step SI 00, the method comprises identifying at least one portion of a video (for example a video frame, a sequence of two or more video frames, a timestamp of the video, or a range of timestamps) for processing at step 101. A specific identified portion of a video is referred to herein as a “selected portion” (i.e., therefore, a “selected portion” represents both a specific video and a specific portion of the video). Relevantly, in relation to a particular video, in a case where there are two or more selected portions, no two selected portions relate to the same portion of the video. In an embodiment, no two selected portions comprise an “overlapping” part of the video. In an embodiment, the (or each) selected portion comprises a continuous sequence of two or more video frames (or, equivalently, a particular portion is continuous between a first timestamp and a second timestamp).
[0054] Typically, at least one user device 13 associated with a user is configured to present a video sourced from a publisher server 12. In this way, the user is able to experience the video. The video can be provided as a transient “video stream” (i.e., the user device 13 does not store the entire video before or during playback), although other implementations are envisaged (for example, the entire video can be downloaded before playback begins or an entire download completed during playback). A video can be understood as a visual media which changes over time (i.e., during video playback). A video can comprise a soundtrack as well as the visual media. Therefore, step SI 00 can correspond to identifying one or more selected portions of a currently presented video on the user device 13.
[0055] At step S 101, the method comprises analysing a selected portion in order to determine one or more matching products. The one or more matching products determined in accordance with the selected portion such that, as a general rule, the one or more matching products determined in accordance with a first selected portion can be expected to be different to the one or more matching products determined in accordance with a second selected portion. For example, at least one matching product determined in accordance with the first selected portion is not associated with the second portion. It should be noted that this is not intended as a restriction as such, it is simply the case that it is expected to be unlikely that the one or more matching products determined in accordance with two different selected portions comprise exactly the same matching products. Each matching product is itself associated with product data, such that step S101 results in the determination of said product data. The product data corresponds to informationabout its associated product and typically comprises at least a visual representation of its associated product (e.g., an image of the product). The product data can also comprise additional information associated with the product, such as a current price of the product, a hyperlink to a landing page of an online store associated with the product, a textual description of the product, and / or a visual identifier the online store associated with the product. The product data is suitable for enabling a receiving user device 13 to generate a product representation for presentation to a user of the user device 13.
[0056] As used herein, a “product” can be understood as corresponding to one or more goods and / or services — usually, it is a single good or service per product, but this is not required. In an embodiment, a product is required to be visually representable such as to enable the product server 11 to identify the product as being visually similar to an object identified in a video. A “product” can, in an embodiment, also be understood as unique to a particular provider (e.g., seller of the good or provider of the service). For example, a particular good sold by (or otherwise associated with) a first provider can be considered to be a first product and the same good sold by (or otherwise associated with) a second provider can be considered to be a second product; that is, the first product is different to the second product despite relating to the same good.
[0057] Generally, the one or more matching products determined for a particular selected portion of the video are based on analysing the selected portion to identify one or more objects. Typically, the objects are visual features present within the selected portion of the video. In the case of a selected portion corresponding to a single frame of the video, one or more objects can be identified utilising an object recognition algorithm suitable for analysing still images. In a case of a selected portion corresponding to several frames of a video (e.g., defined by a range of video frames or a time range), each individual frame can be analysed separately or the video segment thereby defined can be analysed using a suitable object recognition algorithm. The object recognition algorithm can be based on a machine learning model.
[0058] Reference is made to the present Applicant’s earlier United States patent publications US 2023 / 0044463 Al (published 9 February 2023) and US 2023 / 0259695 Al (published 17 August 2023), both of which are incorporated herein in the entirety be reference. In a general sense, these prior publications include techniques for enabling extraction of objects from an image (i.e., object recognition) and matching products to these objects. In particular, said disclosures may be applicable to step S 101.
[0059] The one or more objects are analysed to identify one or more object characteristics associated with the, or each, object. The one or more object characteristics of one or more identified objects are then compared to product characteristics of a plurality of products stored within a product database maintained by the product server 11. The product database comprises product records, and each product record is associated with a product and comprises at least one product characteristic. The product server 11 identifiesone or more matching products for a particular object by comparing the one or more object characteristics of the object to the product characteristics of the products in the product database. In a general sense, the product server 11 is configured to identify the one or matching products based on a similarity between the one or more matching products and the particular object, based on a closeness in similarity between the or more object characteristics of the object and respective one or more product characteristics of the matching products.
[0060] The product server 11 can implement a scoring algorithm in order to determine the one or more matching products identified for a particular object or, alternatively, a particular selected portion (which can comprise one or more objects). The scoring algorithm in a general sense is arranged to provide a similarity score between a particular object and the products of the product database. The scoring algorithm can include providing a weighting for different object characteristics to prefer a similarity measure in accordance with a first object characteristic compared to a similarity measure in accordance with a second object characteristic. The scoring algorithm can also, or alternatively, include other considerations, for example, preferring a variety of product types, preferring products associated with one class of provider to another class (e.g., where a provider has an option of paying an additional amount to have its products preferred), and any other consideration.
[0061] In an embodiment, the, or each, object has an associated object characteristic corresponding to a visual representation of the object. Typically, the visual representation is derived from the selected portion of the video itself. For example, the visual representation can correspond to a cropped extract of the selected portion in which the object is located. The visual representation can correspond to a further processed extract of the selected portion. For example, a background removal algorithm can be applied to the cropped extract of the selected portion comprising the object to remove or at least minimise the presence of visual features not corresponding to the object. In one example embodiment, an initial composite object is separated into component objects, with each component object assigned an object characteristic corresponding to a visual representation of the component object. For example, product server 11 can identify features of a shirt (the composite object), including collar, round next, v-neck, as corresponding to component objects. Each object (e.g., the composite object, such as the shirt, and the component objects, such as the collar, round next, v-neck) can be treated as an identified object and subsequently analysed. Herein, reference to an “object” should therefore be understood as including both composite objects and component objects where applicable.
[0062] The product server 11 then performs a similarity comparison in which the visual representations of one or more identified objects are compared to visual representations (“product images”) of the plurality of products stored within the product database maintained by the product server 11. A product image can beunderstood as a product characteristic. Therefore, each product record comprises at least one product image to in association with a specific product. The product server 11 therefore is configured to identify one or more visually similar products, based on the product image for the, or each product, for each of the one or more objects identified by the object comparison algorithm. These identified visually similar products correspond to the one or more identified matching products (see step S101).
[0063] In an embodiment, one or more identified objects are analysed to determine one or more object characteristics corresponding to additional properties of the object. Here, “additional properties” refers to properties of the object which are not the visual appearance of the object. For example, one or more additional properties can comprise one or more object-types. For example, the product server 11 can associate an image of a shirt with an object-type being a “shirt” (as opposed to the visual appearance of the shirt). An identified object can be associated with multiple object-types, for example, each of: a “shirt”; “upper body clothing”; “clothing”; and “formal wear”, “casual wear”, or “sportswear” (as appropriate). The available object-types for assigning to an object as the one or more additional properties can be preconfigured (e.g., by a system administrator associated with the product server 11). It is generally expected that the collection of available object-types can be changed as required.
[0064] In this embodiment, the one or more product characteristics of a product record comprise one or more product properties. A product property provides information regarding the product different to its visual appearance. The product properties can correspond (at least in part) to the possible object-types assignable to an object. The product server 11 performs a similarity comparison to identify one or more similar products based on a comparison to identify similarities between the determined one or more additional properties of the object and the one or more product properties of the product records within the product database. These identified similar products correspond to the one or more identified matching products (see step S101).
[0065] The product characteristics can comprise both product images as described above and one or more product properties for each product. Similarly, the object characteristics can comprise both the visual representation of the object obtained from the video at the selected portion and the one or more additional properties determined by analysing the object. Therefore, the product server 11 is configured to perform a similarity comparison including comparing both visual representation of the object to the product images stored in the product database and comparing the determined one or more additional properties of the object to the one or more product properties stored in the product database. For example, the scoring algorithm can produce a score representing both a visual similarity and an object-type similarity.
[0066] Advantageously, utilising a similarity comparison which includes a comparison of product properties to additional properties may allow for determining matching products which are quite differentin appearance to the object. In one example, an object is identified as being an object-type of “smartphone”. The similarity comparison in this embodiment can allow for identifying a matching product (e.g., good or service) which is not a smartphone (which could be the result of a visual comparison). In this example, a determined matching product could be an advertisement for a mobile phone plan, an advertisement for a particular smartphone “app” (e.g., downloadable from an “App Store”), or an advertisement for a mobile phone repair service. In another example, an object is, visually, a visually distinctive building in a city (e.g., the Empire State Building). In this case, the determined matching product could be an advertisement for a vacation package to New York.
[0067] As previously mentioned, the scoring algorithm can apply a weighting. The scoring algorithm can also (or alternatively) be configured to prefer certain combinations of products to be determined as two or more matching products. For example, the scoring algorithm can be configured to prefer primarily results that are based on a high visual similarity between a particular object and the products of the product database, but which will allow for a minority of results to be non-visually similar (for example, the mobile phone example presented above), or vice versa. It is envisaged that the scoring algorithm depend on the particular publisher server 12 from which the video is derived (or more generally, a brand identity associated with the publisher server 12); for example, for a first publisher server 12a the scoring algorithm is configured to only consider visual similarities but for a second publisher server 12b the scoring algorithm is configured to prefer an equal balance between visually similar matching products and non-visual based similarities of the matching products to the object..
[0068] At step SI 02, the method comprises providing the product data of the determined one or more matching products to a user device 13. Typically, step SI 00 is initiated due to a request for product data communicated to the product server 11. The request for product data includes a user device identifier suitable for identifying a user device 13 associated with the user. Step SI 02 therefore corresponds to the product server 11 communicating the product data of the determined one or more matching products to the identified user device 13 as a product data package, typically via a data communication over the network 15. In an embodiment, the product data package includes a selected portion identifier suitable to enable to receiving user device 13 to determine the portion of the video utilised at step S 101 to determine the product data of the one or more matching products.
[0069] In a case where, at step S 100, two or more selected portions are identified, step S 101 can be repeated for each of the selected portions. In this case, step SI 02 can also be repeated such that a separate product data package is communicated to the identified user device 13 for each selected portion. In another embodiment, in the case of two or more selected portions being identified, a combined product data package comprising at least two product data packages is communicated to the identified user device 13 (i.e., eachbeing associated with a selected portion). In either case, the user device 13 receives a product data package for each selected portion, wherein each product data package is uniquely associated with one of the selected portions and comprises product data of the one or more matching products determined in accordance with its associated selected portion. Typically, each product data package further comprises a selected portion identifier of its associated selected portion.
[0070] In a variation, according to an embodiment, a product data package is instead communicated to an intermediary. For example, the identified user device 13 can be in data communication with a web server (not shown), such that the user device 13 presents a webpage (typically a dynamic webpage) on a web browser application. In this case, the product data package is communicated to the web server which is itself configured to generate (or, equivalently, update) a webpage for presentation on the user device 13 in which one or more product representations, generated in accordance with the product data, are presented. The user device 13 therefore renders and presents the webpage thereby presenting the one or more product representations to the user. For the purposes of the present disclosure, unless noted otherwise, reference to the user device 13 receiving, representing, or otherwise interacting with a product data package should be understood as applying to a case where an intermediary receives the product data package.
[0071] According to an embodiment, with reference to Figure 4A, the same user device 13 is configured to present a video sourced from a publisher server 12 as well as to present product data sourced from the product server 11. Typically, the user device 13 communicates with the publisher server 12 (e.g., as a data communication via network 15) to request a video. In an example, the request for video can be in response to a user selection of a video, for example, the user choosing a movie, TV show, or other video media that the user would like to view via an interaction with a user interface of the user device 13. In another example, the request for video can be in response to the user visiting, on the user device 13, a website in which the video is presented; that is, the user’s primary interest is in visiting the website rather than the specific video in question. In response, the publisher server 12 communicates video data to the user device 13 to enable the user device 13 to present the video to the user.
[0072] After receiving the video data, the user device 13 communicates to the product server I l a content identifier comprising either or both of a video identifier and video portion identifier (depending on the embodiment).
[0073] A video identifier comprises information enabling the product server 11 to access the video. In an implementation, the product server 11 uses the video identifier to identify a copy of the video data of the video stored in its memory 21. In an implementation, the product server 11 communicates with a remote data storage identified by reference to the video identifier in which the video data of the video is located (this can, for example, correspond to the publisher server 12 accessed by the user device 13). These twoimplementations can be combined, for example, the product server 11 can store (e.g., cache) video data for a relatively small number of videos at any one time (for example, the product server 11 can store video data associated with currently popular videos) and access the remote data storage when the video data for a particular video is found not to be stored by the product server 11.
[0074] A video portion identifier enables the product server 11 to identify an associated selected portion of a particular video.
[0075] In an embodiment, the video portion identifier comprises a location identifier enabling the product server 11 to identify the location (e.g., relative position within the video) of the associated specific portion. In order to identify the associated selected portion, the product server 11 is required to access the video and, therefore, in this embodiment the video identifier is provided to the product server 11 in combination with the video portion identifier. The product server 11 identifies the associated selected portion by determining the portion of the video corresponding to the location identifier. For example, the location identifier can correspond to a frame number, a frame range, a timestamp, or a time range defined by a first timestamp and a second timestamp. Therefore, according to this embodiment, the video portion identifier may be understood as comprising metadata related to the video.
[0076] In another embodiment, the video portion identifier comprises image data corresponding to the selected portion. For example, the video portion identifier can comprise a “screenshot” of the video corresponding to the selected portion. In this embodiment, the video portion identifier is not necessarily combined with a video identifier. In a typical implementation, the image data corresponds to single image. However, it is anticipated that the image data can correspond to a collection of images, for example, where the selected portion is a segment of the video.
[0077] The product server 11 utilises the received video identifier and / or video portion identifier to determine one or more selected positions of an associated video in order to identify product data for communicating to the user device 13 (for example, as according to the method of Figure 3). As described, the product data can be communicated as one or more product data packages.
[0078] The user device 13 is configured to present the product data to the user (e.g., as one or more product representations displayed via a display of the user device 13). Optionally, the product data (or at least a portion thereof) can be presented simultaneously with the video.
[0079] In an embodiment, the user device 13 is configured to identify that playback of the video has been paused. In response, the user device 13 is configured to determine a video portion identifier that identifies the currently presented video frame of the video (i.e., representing the appearance of the paused video). This video portion identifier is then communicated to the product server 11 as described.
[0080] In an embodiment, the user device 13 is configured to identify user selection of a particular portion of the video, the user selection being an indication that the user desires to receive one or more instances of product data associated with the video. More specifically, the user selection can correspond to a user input to the user device 13 (in some cases, for example where the user device 13 corresponds to a smart TV, the user input can be via a remote control coupled with the user device 13 — for example, a remote control can be provided with a physical button to enable the user input) indicating that a currently presented portion of the video is to be utilised by the product server 11 to generate the product data. Optionally, the user input can also cause playback of the video to be paused. Optionally, the video can be required to be paused before enabling the user to make the user input. The user input can comprise the user interacting with a visible element, such as a labelled button (for example, the button may be labelled “Pause-to-Shop”) presented on the user device 13 as an overlay to the video or close to the video. In response, the client device 13 is configured to determine a video portion identifier (and, depending on the embodiment, a video identifier) that identifies the currently present video frame of the video (i.e., representing the appearance of the paused video). This video portion identifier and optionally video identifier is then communicated to the product server 11.
[0081] Figure 4B shows another embodiment, which can be understood as a variation to the embodiment of Figure 4A. In this case, a first user device 13a communicates with the publisher server 12 (thereby receiving video data to present to an associated user). As with the embodiment of Figure 4A, the first user device 13a also communicates with the product server 11 to thereby provide a content identifier comprising a video identifier and / or a video portion identifier.
[0082] The embodiment of Figure 4B differs from that of Figure 4A in that the product data is provided to a second user device 13b rather than the first user device 13a. This embodiment may be suitable for a case where the product data cannot be presented on the same display as the video. For example, in a case where the first user device 13a is a smart TV, it may be that the video corresponds to a TV show, movie, or other media provided by a streaming service. For example, a smart TV (although not necessarily limited to smart TVs) may not provide functionality to enable the presentation of product representations based on product data sourced from a product server 11 at the same time as a video sourced from the publisher server 12.
[0083] In embodiments, the first user device 13a and the second user device 13b are known to the product server 11 as being associated with one another. In an embodiment, the video identifier and / or video portion identifier communicated from the first user device 13a to the product server 13b includes a user device identifier suitable to enable the product server 11 to communicate with the second user device 13b (for example, comprising a TCP / IP address of the second user device 13b or information to enable the productserver 11 to determine the TCP / IP address of the second user device 13b). In another embodiment, the first user device 13a and second user device 13b are both pre-registered with the product server 11, for example, due to a prior registration and / or login procedure undertaken by the user with each user device 13a, 13b. This pre-registration and / or login can effectively provide the product server 11 with the user device identifier suitable to enable the product server 11 to communicate with the second user device 13b.
[0084] Figure 4C shows another embodiment, which can be understood as a variation to the embodiment of Figure 4B. In this case, a first user device 13a communicates with the publisher server 12 (thereby receiving video data to present as a video to an associated user). The embodiment of Figure 4C differs from that of Figure 4B in that the video identifier and / or video portion identifier is communicated to the product server 11 from the second user device 13b. Therefore, the first user device 13a is not required to undertake data communication with the product server 11. The second user device 13b is configured to obtain the video identifier and / or video portion identifier from the first user device 13a (or, at least, data to enable the second user device to determine the video identifier and / or video portion identifier) — represented by a broken line in the figure.
[0085] In an implementation, the first user device 13a is configured to display a visible code (such as a QR- code) on its display encoding the video identifier and / or video portion identifier. The second user device 13b is configured to capture an image of the visible code and decode it, thereby obtaining the video identifier and / or video portion identifier. The second user device 13b is then configured to communicate the video identifier and / or video portion identifier to the product server 11 (e.g., in a similar manner to as described with reference to Figures 4A and 4B).
[0086] In another implementation, the first user device 13a and second user device 13b are in data communication with one another. The data communication can be via a local data network (e.g., set up by a home WiFi router) and / or via the Internet. The data communication can be via a direct data connection between the first user device 13a and second user device 13b, typically a wireless data connection for example, a Bluetooth connection (where the first user device 13a has been paired with the second user device 13b).
[0087] Figures 4D and 4E show two embodiments, each of which can be understood as a variation to the embodiments of Figures 4A-4C. In each case, a user device 13 communicates with the publisher server 12 and receives video data to present as a video to an associated user. Unlike the embodiments of Figures 4A to 4C, the user device 13 does not communicate a video identifier and / or video portion identifier to the product server 11. Instead, the publisher server 12 communicates the video identifier and / or video portion identifier to the product server 11. In this way, the user device 13 is not required to initiate communication with the product server 11.
[0088] In an embodiment, the publisher server 12 is configured to identify that playback of the video has been paused on the user device 13. In response, the publisher server 12 is configured to determine a video portion identifier that identifies the currently present video frame of the video (i.e., representing the appearance of the paused video). This video portion identifier and, depending on the embodiment, video identifier is then communicated to the product server 11.
[0089] In an embodiment, the publisher server 12 is configured to identify user selection of a particular portion of the video, the user selection being an indication that the user desires to receive one or more instances of product data associated with the video. More specifically, the user selection can correspond to a user input to the user device 13 (in some cases, for example where the user device 13 corresponds to a smart TV, the user input can be via a remote control coupled with the user device 13) indicating that a currently presented portion of the video is to be utilised by the product server 11 to generate the product data. This user input is communicated the publisher server 12 from the user device 13. Optionally, the user input can also cause playback of the video to be paused. Optionally, the video can be required to be paused before enabling the user to make the user input. The user input can comprise the user interacting with a visible element, such as a labelled button (for example, the button may be labelled “Pause-to-Shop”) presented on the user device 13 as an overlay to the video or close to the video. In response, the client device 13 is configured to determine a video portion identifier (and, depending on the embodiment, a video identifier) that identifies the currently present video frame of the video (i.e., representing the appearance of the paused video). This video portion identifier and optionally video identifier is then communicated to the product server 11.
[0090] In the embodiment of Figure 4D, the publisher server 12 also communicates a user device identifier suitable to enable the product server 11 to communicate with the user device 13 directly (for example, comprising a TCP / IP address of the user device 13 or information to enable the product server 11 to determine the TCP / IP address of the user device 13). In a variation, the user device identifier identifies a second user device 13b as described with reference to Figure 4B.
[0091] In the embodiment of Figure 4E, the product server 11 is configured to reply (that is, communicate to the one or more instances of product data) to the publisher server 12. The publisher server 12 is configured to then provide the one or more instances of product data to the user device 13. For example, in the case of the video being streamed from the publisher server 12 to the user device 13, the one or more instances of product data can be communicated with the video to the user device 13 (for example, as an overlay or pop-up presented along with the video). In another example (not shown), the publisher server 12 communicates the one or more instances of product data to a second user device 13b for presentation to the user (e.g., as described with reference to Figure 4B).
[0092] In a general sense, the content matching system 10 described herein enables one or more user devices 13 associated with a particular user to present both a video obtained from a publisher server 12 and product data obtained from a product server 11. Relevantly, the product server 11 determines the product data to provide to a user device 13 of a user based on the visual content of the video being presented to the user. Advantageously, therefore, the content matching system 10 provides product data to a user that has an improved likelihood of being of interest to the user, as it has been determined based on a current video viewing experience of the user.
[0093] Figure 5 shows a video analysis method according to an embodiment. The video analysis method analyses a provided video to identify one or more selected portions. Relevantly, the identified selected portions are not determined by a user action as such (e.g., the identified selected portions due to the method of Figure 5 are not related to the user pausing the video or taking another user action such as interacting with a button). The one or more selected portions identified by this method can optionally be combined with one or more selected portions identified due to a user action. For a particular video, the one or more selected portions identified by this method are determined according to a trigger rule which is applied to the video.
[0094] At step S200, in response to a video being presented on a user device 13, the product server 11 receives a video identifier associated with the video. For example, depending on the particular embodiment, the video identifier can be communicated to the product server 11 from the publisher server 12 or the user device 13. At step 201, the product server 11 access the video identified by the video identifier (e.g., by communicating with the associated publisher server 12, or in a case where the video is stored in the memory 21 of the product server 11, by accessing said memory 21).
[0095] At step S202, the product server 11 analyses the video by applying the trigger rule to the video. Depending on the particular trigger rule, the product server 11 can as a result identify one or more selected portions of the video. At step S203, for each identified selected portion, the product server 11 analyses the selected portion in order to determine one or more instances of product data (for example, as described with reference to Figure 3).
[0096] In an implementation, the trigger rule specifies a number of video frames between selected portions. For example, the trigger can define every Xthframe (e.g., the trigger can define every 50thframe, corresponding to frame numbers 50, 100, 150, and so on; for a 50 frames per second video, this corresponds to identifying a selected portion every one second). Generally, the value of X can be selected based on considerations including product server 11 load (e.g., the smaller the value of A, the more processing is required by the product server 11) and usefulness to the user (e.g., larger values of A may result in objects in the video being missed). It is expected that the value of A is a settable parameter. In a similarimplementation, the trigger rule defines an amount of time (in terms of video playback) between each selected portion. Therefore, rather than defining a number of frames X. the trigger defines an elapsed time T between selected portions (e.g., the trigger can define every 1-second, corresponding to the displayed portion of the video at 1-second, 2-seconds, 3 -seconds, and so on). The same considerations in selecting X apply to selecting Y.
[0097] In an implementation, the trigger rue defines assigning selected portions to scene changes in the video. Algorithms are known for predicting a change in scene in a video; therefore, a suitable algorithm can be implemented to determine the one or more selected portions. In a variation, the one or more selected portions are identified as portion(s) of the video a predetermined time after (or, equivalently, before) each scene change, so as to improve the chance that one or more objects are visible within the video at the selected portion.
[0098] Optionally, the trigger can comprise multiple different triggers. For example, both every Xhframe as well as every identified scene change.
[0099] Still referring to Figure 5, at step S204, the resulting one or more instances of product data is communicated to the user device 13 (or, in a variation, a second user device 13b as described with reference to Figure 4B).
[0100] The embodiments described herein may be suitable for utilisation in a number of different implementations. The following non-limiting examples are provided to assist in understanding different configurations of the content matching system 10.Example 1 : TV Streaming - OR Code
[0101] A first example is now described with reference to Figure 6, in which a TV show is rendered on smart TV 30 (i.e., corresponding to a first user device 13a), for example via a streaming platform (i.e., corresponding to a publisher server 12). In this example, the streaming platform is Binge. Other example streaming platforms include Netflix, Amazon Prime Video, Hulu, and Paramount-l-. Streaming platforms can provide “on demand” video selected by a user when desired to be watched. The streaming platform facilitates user interaction with a video being presented on the smart TV 30, for example the user may pause the video using the remote control for the smart TV 30.
[0102] The content matching system 10 allows users to watch a video and indicate objects within the video for which they want to be provided with further information. One example allows a user to select a particular video frame, or sequence of video frames, or scene within the video stream in which the objects of interest is present to thereby indicate at least one object of interest. In one example, the user may wish to indicateclothes worn by a character in a scene. In another example, or the user may wish to indicate homeware products in an interior scene.
[0103] Generally, the video is identified by the streaming platform with a particular video stream identifier (“stream ID”) corresponding to a video identifier. The stream ID may relate to a URL or other digital address where the video may be retrieved or some other identification of the video to enable it to be identified or retrieved from a video stream database of the streaming platform. The stream ID can comprise other data to identify the video stream to allow identification of the video stream and retrieval
[0104] When the video is paused, the system 10 generates a visible code in this case in the form of QR code 31. The QR code 31 includes data identifying the video frame displayed at the point of pausing. The QR code 31 can encode the stream ID in order to uniquely identify the particular video being watched by the user. The QR code 31 also encodes video portion identifier to enable the particular location in the video at which the video is paused to be identified. This video portion identifier can be a timestamp identifying the time within the video associated with the paused video frame or a specific frame number of the paused video frame. In this way, the QR code 31 encodes both a video identifier and a video portion identifier. The QR code 31 can therefore be expected to be unique for the particular paused video frame.
[0105] With the QR code 31 displayed on the smart TV 30, the user can scan the QR code 31 using a second user device 13b associated with the user (in the example, the second user device 13b is a smartphone 32). Typically, the user captures an image of the QR code 31 using a QR code scanner application running on the smartphone 32.
[0106] The QR-code 31 (or other visible code) can be generated by the first user device 13a. Alternatively, the QR-code 31 (or other visible code) can be generated by the publisher server 12 and communicated, via the network 15, the first user device 13a such that the first user device 13a displays the QR-code. The publisher server 12 can be configured to include the QR-code 31 with the video to thereby communicate the QR-code 31 to the first user device 13a.
[0107] The smartphone 32 can have installed a dedicated application (“app”) configured to cause the smartphone 32 to undertake data communication with the product server 11. The smartphone 32 can instead communicate with a webpage via a web browser of the smartphone 32, wherein the webpage is configured to enable communication between the smartphone 32 and the product server 11 (for example, the webpage address can also be encoded by the QR code 31). In either case, the smartphone 32 is configured to provide the video identifier and the video portion identifier obtained by decoding the QR code 31 to the product server 11.
[0108] The product server 11 retrieves the video frame (e.g., from the publisher server 12) and executes a matching algorithm (e.g., as described with reference to Figure 3) to identify one or more products matching those the one or more objects within the video frame.
[0109] Product data associated with the one or more matching products is returned to the user by communicating a product data package comprising the product data to the smartphone 32. In the example, the smartphone 32 presents the paused video frame (this can be provided by the product server 11 with the product data package or the smartphone 32 can be configured to obtain an image of the video frame separately, for example, through a data communication with the publisher server 12 based on the video identifier and the video portion identifier decoded from the QR code 31). The smartphone 32 presents the one or more product representations determined from the product data in corresponding one or more result windows 33. In the example, a first product representation is presented in a first result window 33a, and a second product representation is presented in a second result window 33b. It may be that further result windows 33 are located such as to not currently be displayed on the smartphone 32 (e.g., these further result windows 33 can be accessed by “scrolling” the display).
[0110] In the example show, each product representation comprises a product image 34 and several pieces of associated content 40-43. In the present example, each instance of product data comprises associated content including: a text label 40 associated with the particular identified product, a price 41 of the particular identified product, and a brand image 42 indicating an online store from which the particular identified product can be purchased. Also, a hyperlink (or other network identifier) is provided for each product; in the example, the hyperlink is associated with a button 43 located with the relevant result window 33. In a typical implementation, each hyperlink enables a user to visit an online store from which the user can purchase the particular identified product.Example 2: TV Streaming - display on TV or notification to user e.g., SMS, Bluetooth, or email
[0111] In a second example, the user interacts with the video playing on a first user device 13a (e.g., the smart TV 30 shown in Figure 6), for example by pausing the video stream. In response, the video portion identifier identifying the video frame at which the video has been paused is provided automatically to the product server 11. For example, the video identifier and video portion identifier can be pushed to the product server 11 by the smart TV 30. In another example, the publisher server 12 is configured to detect that the video has been paused on the smart TV 30. The publisher server 12 can have one or more software development kits (SDKs) installed configured to automatically detect that the video has been paused. In response, the publisher server 12 determines the appropriate video portion identifier corresponding to the video frame at which video has been paused. The publisher server 12 then communicates the video portion identifier to the product server 11.
[0112] The product server 11 communicates a product data package associated with one or more identified products to the user. For example, the product data package is directed back to the publisher server 12 which then communicates the product data package to the smart TV 30 for display on the smart TV 30 (for example, alongside or as an overlay to the video). In another variation, the product data package is directed to the user’s smartphone 32. The user can be alerted that the product data package is available or has been received by the smartphone 32 by a notification, for example an email, Bluetooth or SMS. In such cases the user watching the video would need to be identified (or at least, the smartphone 32 is identified as being associated with the smart TV 30), for example by the account details for the streaming service or by other means of logging into the streaming platform.Example 3: Online streaming - display on digital channel or device
[0113] In a third example, with reference to Figure 7, the video is embedded within a webpage 34 within video window 38 of a webpage publisher. The video is sourced from a publisher server 12 (this can be associated with the webpage publisher or can be a third-party video source). Optionally, there can be integration between the webpage publisher and the product server 11.
[0114] The webpage of the webpage publisher (or the webserver of the webpage publisher) can have an SDK loaded which detects when a user opens the webpage and sees a particular video. The SDK detects that the video has been viewed or paused and automatically determines an appropriate video identifier and video portion identifier. The SDK then communicates the video identifier or video portion identifier to the product server 11. The product server 11 identifies one or more matching products which match identified one or more objects within the video portion defined by the video identifier and video portion identifier and provides a product data package associated with one or more identified products to the user. The product data of the product data package can comprise web links. The product representation(s) can be presented in a pop-up 35 displayed on the viewport of the device (the change in appearance of the webpage is indicated by the broken arrow). The pop-up 35 can be a lightbox presenting an arrangement (e.g., grid) of result windows 33. As shown, the pop-up 35 can effectively “block” the video window 38 and therefore the video, depending on the configuration of the website, in which case the video can be paused when the pop-up 35 is visible.
[0115] Figure 7 also shows a dynamic results window 36 in which one or more matching products are displayed in corresponding result windows 33. The dynamic results window 36 can be understood as showing a “preview” or sub-set of the matching products determined in accordance with a selected portion of the video. The dynamic results window 36 is presented in proximity to the video but not blocking the video (as may occur with a pop-up 35 as shown); in this case, the dynamic results window 36 is located below the video window 38.
[0116] The system 10 can be configured to change the matching products displayed within the dynamic results window 36. For example, in an embodiment such as described with reference to Figure 5, the product server 11 can be configured to automatically identify a plurality of selected portions. The system 10 can be configured to present matching products for a first selected portion when the corresponding portion of the video plays and retain these matching products until the portion of the video of a second selected portion plays, in which case (at least some of) the matching products determined in relation to the first selected portion are removed and new matching products determined in accordance with the second selected portion are presented. From a user experience perspective, the dynamic results window 36 is effectively regularly updated with new product representations relevant to the current portion of the video being played (i.e., presented in the relevant user device 13).
[0117] Figure 7 also shows a selected portion indicator 37 which provides a visual indication of the one or more selected portions automatically determined for the video. In this case, the selected portion indicator 37 display still images of the video associated with the various selected portions. In an implementation, the selected portion indicator 37 is dynamically updated such that it indicates selected portions that have already been (and optionally are currently being) displayed in the video. The selected portion indicator 37 can allow the user to choose one of the indicated selected portions (which can, for example, cause a popup 35 or other display of the matching products associated with the chosen selected portion).
[0118] Figure 8 shows an example architecture flow diagram of the steps performed in an example system 10 integrated with a webpage 34 (e.g., see Example 3). At S310 the webpage 34 is loaded onto a user device 13. Loading the webpage 34 triggers loading of an SDK configured to cause the user device 13 to communicate with product server 11 (either directly or via the publisher server 12, depending on the embodiment). The SDK can cause the user device 13 to perform several functions, such as determining the page configuration, including identification of any video streams appearing on the webpage, identifying any video stream elements associated with the video stream(s), and connecting these video stream elements into a video elements API. The SDK can also load algorithm(s) used to identify user interaction with the video streams.
[0119] At S320 the user pauses the video stream. At S330 information about the paused video stream is captured and provided to the product server 11. The information may include the video frame and / or video stream id or other data and the timestamp for the video frame (e.g., corresponding to the video identifier and / or video portion identifier).
[0120] At S340 the product server 11 processes the video frame and identifies matching products or other matching information. In some cases, at S342, if a link or other video identifier is provided, the product server 11 may process other video frames of the video stream and identify matching products or othermatching information for each of the other video frames. These additional matching products or other matching information may be stored at the product server 11 against the relevant video frames. This allows faster retrieval of relevant products and information forthose frames in the future.
[0121] At 350 the identified matching products or other matching information is provided from the product server 11 to be displayed in a popup 35 on the webpage 34, for example in an adunit.
[0122] Throughout the process the user may continue to interact with the video, for example the user may un-pause the video. The product data may be stored along with the relevant video frames and accessed by the user at a later time at 352.Example 4: Outline of user experience
[0123] Figure 9 is a flow diagram exemplifying the customer experience during interaction with an example system 10. At 410, a media provider distributes a video (e.g., the media provider is associated with a publisher server 12). The video may be distributed via a website, smart TV, TV application, or other media delivery platform, for example augmented reality (AR) or virtual reality (VR) experiences. The user accesses the video stream at 420, for example via an associated user device 13, for example a computer, television, mobile phone, tablet. At 430 the user interacts with the video, for example by pausing the video or by providing some other interaction to identify a video frame for selection as a selected portion. The selection is analysed by a product server 11 and product information (e.g., as product data provided with a product data package) is provided to the user at 440. For example, the matched products may be displayed to the user on the user device 13 alongside the video (e.g., directly to the side of, above, or below the video, or as an overlay over the video). At 450 if the user wishes to interact with a particular matched product, then the user can interact with the corresponding product representation for the matched product (this can be understood as the user “selecting” the particular product). In an example where the matched product is a clothing product the user device 13 can present a link to a clothes brand website where the user may view or purchase the product at 460.
[0124] An example user experience is shown in further detail in Figure 10, corresponding to a flow diagram showing the customer experience during interaction with an example system 10 in a little more detail than Figure 9. At 510 the media provider distributes the video (e.g., the media provider is associated with a publisher server 12). The video may be distributed via a website, TV, TV application, or other media delivery platform, for example AR or VR experiences. The user accesses the video at 520, for example via a user device 13, for example a computer, television, mobile phone, tablet. The video is played on the user device 13 either automatically at 522 or on selection by the user at 524. The video plays at 526. At 530 the user interacts with the video, for example by pausing the video or by providing some other interaction to identify a video frame for selection as a selected portion. The selected portion is analysed by a productserver 11 and product information (e.g., as product data provided with a product data package) is provided to the user. For example, at 542 the user may engage further on detected and matched products. At 544 the user is presented with the top detected and matched products based on objects identified in relation to the selected portion of the viewed video. In some examples at 532 the user can continually interact with the video. For example, the user may pause the video at multiple points. In some examples at 546 the product server 11 may automatically detect, match and present products for display to the user during video streaming. This automatic detection may be periodic. Matched products may be displayed to the user on the user device 13 alongside the video stream. At 550 if the user wishes to interact with the matched products, then the user can interact with the matched items at 560, for example by selecting the product as described above. Selection of the product may direct the user differently depending on the system 10 setup or the product or link type. For example, at 562 the user is redirected to an intermediary site which is dedicated to the specific product. At 564 the user is directed to an end destination, for example, a brand website to enable the user to engage directly with the associated brand. At 566 the user may remain on the publisher platform and open a new UI experience for the user to allow purchase a selected product. The user may purchase the product at 570.
[0125] The examples described above describe interaction of a user with a video. The video may be a digital video. The digital video may or may not have accompanying audio. Although the present disclosure has been in reference to “video”, other embodiments are envisaged where the video is substituted for interactive media, for example a video game.
[0126] Advantageously, in the described examples, user interaction with the video may be detected by the system 10 and the system 10 dynamically responds to the user interaction by automatically retrieving product data for products similar to objects identified within a video frame interacted with by the user, for example a viewed or paused video frame. The system 10 advantageously may allow a user complete freedom of choice when selecting video frames for matching. Example systems 10 may advantageously respond in real-time and retrieve “live” product data to provide a user with real-time links to view or purchase options associated with objects in the video frame. Here, real-time refers to the user experience in which product representations are made available to the user at the same time (from the user’s perspective) as the related objects are visible on the relevant user device 13. Real-time therefore includes small delays which, from the user experience perspective, are insufficient to stop the user from associating the resulting product representations from the video portion resulting in the provision of the product representations.
[0127] Other embodiments may be configured to identify selected portions automatically (e.g., selected video frames of a video) without user interaction. Such embodiments may be configured to match products to objects identified within video frames periodically, for example every 1 second, 10 seconds, 60 seconds,or any other desired period during the video, or configured to match products each time a scene change is detected in the video stream. These systems may provide these automatic product updates (i.e., automatic presentation of product representations while the user is watching the video) in addition to facilitating user interaction.
[0128] Further modifications can be made without departing from the spirit and scope of the specification.For example, the product server 11 can be configured to store its memory 21 product data for a particular video or selected portion after determining the product data (e.g., for each product identified based on the video or selected portion). The product data can be stored such that, in a case when another user or user device 13 views the same video, the product server 11 can obtain the previously stored product data forthat video rather than performing a new analysis of the video. The product server 11 can be configured to keep the stored product data for some amount of time, for example, for 24-hours after the video was most recently watched by any user of the system 10. In this way, popular videos are not analysed every time they are viewed, which can reduce the total load on the product server 11 without having an undue effect on the user experience.
Claims
Claims:
1. A product data identification method for identifying product data associated with a video presented on a first user device, comprising: identifying a selected portion of a video corresponding to a portion of the video determined based at least in part on a received content identifier, wherein the selected portion comprises at least one video frame of the portion of the video; analysing the selected portion to identify one or more objects present within the at least one video frame of the selected portion; comparing at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product; identifying one or more matching products based on a comparison between one or more object characteristics identified for the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicating the product data of the, or each, matching product to the first user device or a second user device.
2. A method as claimed in claim 1, wherein the content identifier comprises a video portion identifier comprising image data corresponding to the at least one video frame of the portion of the video to thereby identify the selected portion.
3. A method as claimed in claim 1, wherein the content identifier comprises a video identifier comprising information enabling access to the video and a video portion identifier identifying the at least one video frame of the portion of the video, and wherein the method comprises: using the video identifier to access the video; and obtaining the at least one video frame of the portion of the video according to the video portion identifier to thereby identify the selected portion.
4. A product data identification method for identifying product data associated with a video presented on a first user device, comprising: accessing the video identified by a received content identifier; determining at least two selected portions of the video by analysing the video according to a predefined trigger rule;for the at least two selected portions: analysing the selected portion to identify one or more objects present within the at least one video frame of the selected portion, comparing at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product, and identifying one or more matching products based on a comparison between one or more object characteristics identified for the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicating the product data of each matching product identified for each of the at least two selected portions to the first user device or a second user device.
5. A method as claimed in claim 4, wherein the product data comprises a selected portion identifier for each product to thereby enable presentation of the product data of a particular product in association with presentation of the selected portion of the video.
6. A method as claimed in any one of claims 1 to 5, wherein the video is provided to the first user device by a publisher server, wherein identifying a selected portion of a video corresponding to a portion of the video comprises accessing the video on the publisher server.
7. A method as claimed in claim 6, comprising: receiving the content identifier from the publisher server.
8. A method as claimed in claim 7, comprising: communicating the product data of the, or each, matching product to the first user device or a second user device via a communication with the publisher server.
9. A method as claimed in any one of claims 1 to 6, comprising: receiving the content identifier from the first user device; and communicating the product data of the, or each, matching product to the first user device.
10. A method as claimed in any one of claims 1 to 6, comprising: receiving the content identifier from the first user device; and communicating the product data of the, or each, matching product to the second user device.
11. A method as claimed in any one of claims 1 to 10, wherein the product data of the, or each, matching product is communicated to the first user device or a second user device via a website accessed by the first user device or the second user device.
12. A method as claimed in claim 11, wherein the website includes the video such that the video is presented on the first user device in response to the first user device accessing the website, the method comprising: updating the website to include a visual representation of the product data for the, or each, matching product.
13. A method as claimed in claim 12, wherein the visual representation of the product data for the, or each, matching product is presented in a pop-up on the website and / or adjacent the video.
14. A method as claimed in claim 13 when dependent on claim 5, comprising: configuring the website to present the visual representation of each product in dependence on a current playback location of the video, in accordance with the selected portion identifier associated with each product.
15. A method as claimed in any one of claims 1 to 14, wherein the content identifier is generated in response to a user action on the first user device.
16. A method as claimed in claim 15, wherein: the user action corresponds to the user pausing the video, and wherein the content identifier identifies the currently displayed portion of the video; and / or the first user device presents an interactive element selectable by the user, wherein the user action corresponds to the user selecting the interactive element, and wherein the content identifier identifies the displayed portion of the video at the time of the user selecting the interactive element.
17. A method as claimed in any one of claims 1 to 16, wherein each product record comprises a product characteristic corresponding to a product image, and wherein the, or each, object comprises an object characteristic corresponding to a visual representation determined for the object by reference to the selected portion, and wherein matching products are identified at least in part in accordance with a visual similarity between the one or more matching products and the visual representations of the one or more objects.
18. A method as claimed in any one of claims 1 to 17, wherein each product record comprises one or more product characteristics each corresponding to a product property, and wherein the, or each, object is analysed to determine one or more object characteristics corresponding to additional information regarding the object, and wherein the one or more matching products are identified at least in part in accordance with a similarity between the one or more product characteristics of the one or more matching products and the determined additional information of the one or more objects.
19. A method as claimed in any one of claims 1 to 18, implemented by a product server.
20. A product server comprising one or more processors interfaced with a memory and a network interface, wherein the one or more processors are configured to communicate with a data network via the network interface, and wherein the memory comprises program instructions configured to cause the one or more processors to: identify a selected portion of a video corresponding to a portion of the video determined based at least in part on a received content identifier, wherein the selected portion comprises at least one video frame of the portion of the video; analyse the selected portion to identify one or more objects present within the at least one video frame of the selected portion; compare at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product; identify one or more matching products based on a comparison between one or more object characteristics identified for between the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicate, via the data network, the product data of the, or each, matching product to a first user device on which the video is presented or a second user device.
21. A product server as claimed in claim 20, wherein the content identifier comprises a video portion identifier comprising image data corresponding to the at least one video frame of the portion of the video to thereby identify the selected portion.
22. A product server as claimed in claim 20, wherein the content identifier comprises a video identifier comprising information enabling access to the video and a video portion identifieridentifying the at least one video frame of the portion of the video, and wherein the program instructions are configured to cause the one or more processors to: use the video identifier to access the video; and obtain the at least one video frame of the portion of the video according to the video portion identifier to thereby identify the selected portion.
23. A product server comprising one or more processors interfaced with a memory and a network interface, wherein the one or more processors are configured to communicate with a data network via the network interface, and wherein the memory comprises program instructions configured to cause the one or more processors to: access a video identified by a received content identifier; determine at least two selected portions of the video by analysing the video according to a predefined trigger rule; for the at least two selected portions: analyse the selected portion to identify one or more objects present within the at least one video frame of the selected portion, compare at least one of the one or more objects to a product database comprising product records for each of a plurality of products, wherein each product record comprises one or more product characteristics and product data of its associated product, and identify one or more matching products based on a comparison between one or more object characteristics identified for the one or more objects and the one or more product characteristics of each of the one or more matching products; and communicate the product data of each matching product identified for each of the at least two selected portions to a first user device on which the video is presented or a second user device.
24. A product server as claimed in claim 23, wherein the product data comprises a selected portion identifier for each product to thereby enable presentation of the product data of a particular product in association with presentation of the selected portion of the video.
25. A product server as claimed in any one of claims 20 to 24, wherein the video is provided to the first user device by a publisher server, wherein the product server is configured to access the video on the publisher server in order to determine a selected portion of the video.
26. A product server as claimed in claim 25, wherein the program instructions are configured to cause the one or more processors to: receive the content identifier from the publisher server.
27. A product server as claimed in claim 26, wherein the program instructions are configured to cause the one or more processors to: communicate the product data of the, or each, matching product to the first user device or a second user device via a communication with the publisher server.
28. A product server as claimed in any one of claims 20 to 28, wherein the program instructions are configured to cause the one or more processors to: receive the content identifier from the first user device; and communicate the product data of the, or each, matching product to the first user device.
29. A product server as claimed in any one of claims 20 to 25, wherein the program instructions are configured to cause the one or more processors to: receive the content identifier from the first user device; and communicate the product data of the, or each, matching product to the second user device.
30. A product server as claimed in any one of claims 20 to 29, wherein the product data of the, or each, matching product is communicated to the first user device or a second user device via a website accessed by the first user device or a second user device.
31. A product server as claimed in claim 30, wherein the website includes the video such that the video is presented on the first user device in response to the first user device accessing the website, wherein the program instructions are configured to cause the one or more processors to: update the website to include a visual representation of the product data for the, or each, matching product.
32. A product server as claimed in claim 31, wherein the visual representation of the product data for the, or each, matching product is presented in a pop-up on the website and / or adjacent the video.
33. A product server as claimed in claim 32 when dependent on claim 24, wherein the program instructions are configured to cause the one or more processors to: configure the website to present the visual representation of each product in dependence on a current playback location of the video, in accordance with the selected portion identifier associated with each product.
34. A product server as claimed in any one of claims 20 to 33, wherein the content identifier is generated in response to a user action on the first user device.
35. A product server as claimed in claim 34, wherein: the user action corresponds to the user pausing the video, and wherein the content identifier identifies the currently displayed portion of the video; and / or the first user device presents an interactive element selectable by the user, wherein the user action corresponds to the user selecting the interactive element, and wherein the content identifier identifies the displayed portion of the video at the time of the user selecting the interactive element.
36. A product server as claimed in any one of claims 20 to 35, wherein the program instructions are configured to cause the one or more processors to maintain the product database in the memory.
37. A product server as claimed in any one of claims 20 to 36, wherein each product record comprises a product characteristic corresponding to a product image, and wherein the, or each, object comprises an object characteristic corresponding to a visual representation determined for the object by reference to the selected portion, and wherein the one or more matching products are identified at least in part in accordance with a visual similarity between the matching products and the visual representations of the one or more objects.
38. A product server as claimed in any one of claims 20 to 37, wherein each product record comprises one or more product characteristics each corresponding to a product property, and wherein the, or each, object is analysed to determine one or more object characteristics corresponding to additional information regarding the object, and wherein the one or more matching products are identified at least in part in accordance with a similarity between the one or more product characteristics of the one or more matching products and the determined additional information of the one or more objects.
39. A product data identification system comprising: the product server of any one of claims 20 to 38; and a first client device associated with a user, the first client device in data communication with the product server via a network, wherein the first client device is configured to: present a video for viewing by the user; and communicate a content identifier to the product server identifying the video and / or a portion of the video being presented on the user device.
40. A system as claimed in claim 39, comprising a second client device associated with the user in data communication with the product server via the network, wherein product server is configured to communicate the product data of the, or each, matching product to the second user device in response to receiving a content identifier from the first client device.
41. A system as claimed in claim 40, wherein the second client device: executes a web browser application and is configured to load a website comprising one or more product representations, each product representation associated with a matching product, to thereby receive the product data from the product server; and / or executes a native application configured to present on the second user device one or more product representations, each product representation associated with a matching product, to thereby receive the product data from the product server.
42. A system as claimed in either claim 40 or claim 41, wherein the first client device is configured to: present a visible code (such as a QR-code) in response to a user action, the QR-code encoding the content identifier, and wherein the second client device is configured to: capture an image of the visible code in order to decode the content identifier; and communicate the content identifier to the product server, thereby enabling the first client device to communicate the content identifier to the product server.
43. A system as claimed in claim 39, wherein product server is configured to communicate the product data of the, or each, matching product to the first user device in response to receiving a content identifier from the first client device.
44. A system as claimed in claim 43, wherein the video is presented on a website accessed by the first client device, and wherein the first client device receives the product data as an update to the website, such that the first user device presents one or more product representations, each product representation associated with a matching product, as a pop-up on the website or adjacent to the video.
45. A system as claimed in any one of claims 39 to 44, comprising a publisher server, wherein the first client device obtains the video from the publisher server.
Citation Information
Patent Citations
Machine-based object recognition of video content
US11317159B2
Method, system, and apparatus for identifying and revealing selected objects from video
US11416714B2
Second Screen Recipes Function
US20150170325A1
Mobile visual commerce system
US20160225053A1
Image Recognition System
US20190122384A1
