A system and method for automated video content enhancement for tagging products, locations, and experiences
Patent Information
- Application Number
- IN202541050215
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-05-26
- Publication Date
- 2026-08-04
- Estimated Expiration
- 2045-05-26
Abstract
Description
TECHNICAL FIELD
[0001] The claimed subject matter in general relates to enhancing video content.The subject matter in particular relates to automatically identifying elements in thevideo content and assigning tags to the elements.BACKGROUND
[0002] Video content, ranging from movies and TV shows to advertisements,vlogs, and documentaries, frequently features products, locations, and experiencesthat are of interest to viewers. Experience may be a kind of activity like para-sailing,bungee jumping etc. However, there is no scalable, automated system forextracting, identifying, and tagging these elements within video data.
[0003] Current methods rely heavily on manual annotation, which is time-consuming,error-prone, and inconsistent, especially as the volume of video contentgrows exponentially. This lack of automation results in several missed opportunitiesmentioned below:
[0004] For viewers: Difficulty in discovering, learning about, or purchasing items,visiting places, or having experiences seen in videos.
[0005] For content creators and OTT platforms: Inability to monetize viewer'sdesires for items, places, experiences or enhance content with interactive featuresat scale.
[0006] For brands and marketers: Missed chances for product or place orexperience placement analytics, targeted advertising, and commerce enablement.
[0007] Furthermore, the absence of standardized, actionable metadata for products,locations, and experiences in video content hinders downstream applications suchas shoppable video, location-based recommendations or travel opportunities, andexperience booking.
[0008] The manual techniques of enhancing video content have following shortcomings:
[0009] Manual processes are not scalable for large video libraries. Existing AI toolsfocus on generic object / scene recognition, not actionable tagging or commerceenablement. There is no unified system for identifying, tagging, and exportingactionable metadata for products, locations, and experiences for future use. Thereis a lack of temporal mapping and context-aware tagging. There is no support foroffline batch processing by third parties.
[0010] The US 9516251 B2 discloses a method for identifying a product whichappears in a video stream for which product information and possible purchase isdesired. A video pixel cluster is selected within a video frame corresponding to theproduct. The selected pixel cluster is stored in an area definition database. A productidentifier is associated with the stored pixel cluster. The selection of the cluster ismade manually by a technician.
[0011] Manually identifying products by marking the cluster on the display, is timeconsuming and prone to error. Large and repetitive videos pose a challenge inmanually identifying the clusters and linking them to a product.
[0012] The claimed subject matter addresses the above discussed issues.STATEMENT OF INVENTION
[0013] The claimed subject matter is defined in the independent claims. Furtherdetails are defined in the dependent claims.
[0014] The subject matter discloses a system and method for automated videocontent enhancement for tagging products, locations, and experiences.
[0015] The method for automated video content enhancement comprises accessinga video file, extracting data from the video file, the data comprising video frames,processing the extracted data to perform, at least one of detecting objects;recognizing scenes; and recognizing an activity. The method comprises identifyingat least one element from the group of elements comprising a product from thedetected objects, a location from the recognized scenes and an experience from therecognized activity. The method comprises generating and assigning a tag for eachof the identified elements, the tag comprising at least one of an icon, a QR code anda bounding box. The method comprises determining optimal timing and spatialplacement for the tags, extracting metadata for each element, structuring metadatain standardized formats and exporting enhanced video file, the enhanced video filecomprising embedded tags and associated metadata.
[0016] The subject matter discloses a system for automated video contentenhancement, the system comprising a processor, a memory coupled to theprocessor. The processor is configured to perform the above method steps.BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The detailed description is described with reference to the accompanyingfigures. In the figures, similar reference numerals are used throughout the drawingsto reference similar features and components.Figure 1 illustrates a system for automated video content enhancement for taggingproducts, locations, and experiencesFigure 2 illustrates Video Content Enhancement ManagerFigure 3 illustrates a method for video content enhancement for tagging products,locations, and experiencesDETAILED DESCRIPTION
[0018] The claimed subject matter now will be described with exemplaryembodiments. However, the claimed subject matter may be embodied in manydifferent forms and should not be construed as limited to the embodimentsdescribed herein. These embodiments are provided only as examples so that thisdisclosure is clear and concise.
[0019] Some of the terms used in this document may be used with differentexpressions and interchangeably. Some examples are given below.
[0020] The terms 'object', 'element' and 'product' may be used interchangeably.The object indicates a broader category like 'Apparel' while 'product' or 'element'may indicate a shirt, a vase etc. The term 'element' refers to the group i.e. Product,Location and Experience. The terms 'activity' and 'experience' may be usedinterchangeably. The terms 'Tags' and 'Icons' may be used interchangeably.
[0021] Only the details / components required to describe the claimed subject matterin specific, are disclosed in this document. The details / components which arecommonly known or understood by people skilled in the art may not be covered inthis document.
[0022] The subject matter proposes a modular, automated system for enhancingvideo content by identifying, tagging, and generating metadata for products,locations, and experiences. The system is designed to operate at scale, supportingboth real-time and offline (batch) processing, and to output standardized, actionablemetadata for downstream applications.
[0023] Figure 1 illustrates the System 100 for automated video contentenhancement for tagging products, locations, and experiences. The System 100comprises a Processor 102, a Memory 104 coupled to the Processor 102, a VideoContent Enhancement Manager 106, at least one Video File 108, Input OutputInterfaces 112 and an optional Communication Unit 114 etc. The Video File 108 isconverted into an Enhanced Video File 110 by the Video Content EnhancementManager 106.
[0024] In an embodiment the Communication Unit 114 may be external to theSystem 100 and may be connected to the System 100 using standard interfaces. TheInput Output Interfaces 112 may comprise input ports and output ports, connectedto keyboard, display and other input output devices, as required.
[0025] Some of the units referred herein may be implemented in hardware orfirmware or a combination. For example, the Communication Unit 114 maycomprise standard radio frequency circuits, filters, firmware, input output ports,protocol implementations or a combination of these. In an embodiment, theCommunication Unit 114 establishes communication with any external device, forexchange of data. In an embodiment the System 100 may receive Video File 108from an external device or from a server through the Communication Unit 114.
[0026] The Processor 102 may comprise one or more of central processing units(CPU), graphical processing units (GPU), digital signal processors (DSP),application specific integrated circuits (ASIC), a controller, field programmablegate arrays (FPGA), or any other hardware device, a firmware device, or anycombination of these. The Processor 102 is configured using its control registers,input output ports, firmware, input output interfaces etc. to perform the operationsdescribed herein.
[0027] In an embodiment, the Memory 104 comprises one or more volatile andnon-volatile memory components which can store data and instructions to beexecuted.
[0028] The Video Content Enhancement Manager 106 in the Memory 104 managesenhancement of the video content.
[0029] Figure 2 illustrates a block diagram of the Video Content EnhancementManager 106. The Video Content Enhancement Manager 106 is referred as VideoManager 106 henceforth.
[0030] The Video Manager 106 comprises below modules and the respectivefunctionalities:
[0031] Pre-processing Module 200:
[0032] The Pre-processing Module 200 accepts Video Files 108 in various formats(MP4 (MPEG-4 Part 14), MOV (Apple QuickTime Movie), Matroska Video -MKV, AVI (Audio Video Interleave), WMV (Windows Media Video) etc.). ThePre-processing Module 200 extracts video frames, audio tracks, and subtitle / captiondata from the Video File 108. It performs frame sampling and scene segmentationto optimize processing.
[0033] Multi-Modal Recognition Engine 202:
[0034] The Multi-Modal Recognition Engine 202, referred to as Engine 202,further processes the extracted data. The engine performs audio, visual, and textualanalysis on the extracted data. The Engine 202 comprises Visual Analysis Module202A, an Audio Analysis Module 202B and a Text Analysis Module 202C.
[0035] The Visual Analysis Module 202A uses deep learning models (e.g.,Convolutional Neural Network, CNN, transformers) for object detection, scenerecognition, and activity recognition. Deep learning models, such as ConvolutionalNeural Networks (CNNs) and transformers, are powerful tools for visual analysisbecause they automatically learn hierarchical representations from large datasets.CNNs excel at processing images by capturing spatial hierarchies of features, whichenables them to detect objects and recognize scenes with high accuracy.Transformers, meanwhile, are effective at capturing long-range dependencies andcontextual information, making them well-suited for understanding complexactivities within videos. By learning intricate patterns and relationships in data,these models can accurately identify, classify, and understand various elements inimages and videos. Thus deep learning models identify products (e.g., clothing,electronics), locations (e.g., landmarks, cityscapes), and experiences (e.g., sports,festivals). They employ fine-grained classification for product types, brand logos,and location features. Fine-grained classification is achieved by analyzing subtlevisual differences to distinguish between highly similar categories, going beyondbroad classifications. This involves focusing on minute details like logo placement,stitching patterns, or color accents for product identification. In locationrecognition, architectural features, signage, or background arrangements are usedto differentiate similar cityscapes. By leveraging both these minute visual detailsand the broader context, models achieve higher specificity and accuracy. Thisallows for precise and nuanced identification of products, brands, and locations.
[0036] The Audio Analysis Module 202B processes audio tracks for spokenreferences to products, locations, or experiences (e.g., "This is the new XYZcamera"). It uses speech-to-text and keyword extraction.
[0037] The Text Analysis Module 202C analyzes subtitles, captions, and on-screentext for additional cues. It extracts named entities and context.
[0038] Using the combinational analysis of visual data, audio data and the text data,the engine 202 detects and identifies the elements / objects like products, locationsand experiences accurately.
[0039] Contextual Correlation and Disambiguation Module 204:
[0040] The Contextual Correlation and Disambiguation Module 204 correlatesvisual, audio, and textual cues from the extracted data to improve identificationaccuracy of the objects, e.g. products, locations and experiences. It resolvesambiguities (e.g., distinguishing between similar-looking products or locations). Ituses knowledge graphs and external databases (e.g., product catalogs, geo-locationAPIs) for entity resolution.
[0041] Tag Generation and Placement Module 206:
[0042] For each identified object / element, the Tag Generation and PlacementModule 206 generates a tag. The tags comprise Icons, QR codes and Boundingboxes / Hotspots corresponding to the objects / elements. A "hotspot" is an interactiveregion overlaid on an object in an image or video. Unlike a bounding box, a hotspottriggers a specific action when interacted with, such as displaying information orlinking to a product page. Hotspots offer a precise and interactive way to engagewith identified elements. Some examples of tags are: a cart symbol for a product toindicate 'Buy' option, a destination symbol for a location, a like symbol for anexperience etc.
[0043] The icons are clickable overlays for products, locations and experiences.The QR Codes encode URLs or metadata for mobile interaction. The BoundingBoxes / Hotspots spatially anchor tags within the video frame. Tag Generation andPlacement Module 206 determines optimal timing (start / end frames) and spatialplacement (coordinates) for each tag.
[0044] Metadata Extraction and Structuring Module 208:
[0045] Metadata Extraction and Structuring Module 208 extracts rich metadata foreach element. "Extracting / Extraction / Extract" in this context refers to automaticallyidentifying and retrieving relevant information, such as names, descriptions, prices,and locations, associated with each identified element from various sources. Forproducts it extracts name, brand, model, color, size, price (if available), e-commercelinks etc. For locations, it extracts place name, GPS coordinates, description, pointsof interest. For experiences, it extracts event name, type, organizer, booking links.It structures the metadata in standardized formats (e.g., MPEG-7, JavaScript ObjectNotation for Linked Data, JSON-LD).
[0046] Quality Assurance and Human-in-the-Loop Review Module 210
[0047] Quality Assurance and Human-in-the-Loop Review Module 210 providesan interface for human editors to review, validate, and edit automatically generatedtags and metadata. It supports batch approval, correction, and enrichmentworkflows.
[0048] Export and Integration Module 212:
[0049] The Export and Integration Module 212 exports Enhanced Video File 110(with embedded tags / overlays) and associated metadata. In this context, "export"means generating and outputting the enhanced video file - now containingembedded tags or overlays - along with its associated metadata, so that it can beused by other platforms, applications, or systems. It supports integration with Over-The-Top, OTT, platforms, interactive video players, and commerce / bookingsystems. It provides APIs and SDKs for downstream consumption.
[0050] Logging, Analytics, and Monitoring Module 214:
[0051] The Logging, Analytics, and Monitoring Module 214 tracks processingstatus, error rates, and system performance. It provides analytics on tag density,recognition accuracy, and content enrichment metrics.
[0052] The Enhanced Video File 110 is stored in Memory 104. It may be sent toany external device or broadcast by an OTT channel. When the Enhanced VideoFile 110 is played back on any display device, the content displayed on the displaydevice comprises the video data, tags / icons for the products, locations andexperiences, QR codes, bounding boxes / hotspots etc. The tags may be displayed onthe objects / elements or near the boundary of the objects / elements.
[0053] Figure 3 illustrates a method 300 for automated video content enhancementfor tagging products, locations, and experiences. Step 302 discloses accessing aVideo File 108. Step 304 discloses extracting data from the Video File 108, the datacomprising video frames. Steps 304 and 306 may be performed by the PreprocessingModule 200. Step 306 discloses processing the extracted data toperform, at least one of detecting objects, recognizing scenes and recognizing anactivity. Step 308 discloses identifying at least one element from the elementscomprising a product from the detected objects, a location from the recognizedscenes and an experience from the recognized activity. Steps 306 and 308 may beperformed by the Engine 202. Step 310 discloses generating and assigning a tag foreach of the identified elements. Step 312 discloses determining optimal timing andspatial placement for the tags. Steps 310 and 312 may be performed by the TagGeneration and Placement Module 206. Step 314 discloses extracting metadata foreach element. Step 316 discloses structuring metadata in standardized formats. Step318 discloses exporting Enhanced Video File 110. Steps 314, 316 and 318 may beperformed by Metadata Extraction and Structuring Module 208.
[0054] Further details of the method are described below:
[0055] The method comprises processing the extracted data using deep learningmodels.
[0056] The method comprises frame sampling and scene segmentation.
[0057] The method comprises processing audio tracks for spoken references toidentify products, locations and experiences.
[0058] The method comprises using speech-to-text and keyword extraction toidentify products, locations and experiences.
[0059] The method comprises classifying the products into product types,classifying the locations into location types and classifying the experiences intoexperience types.
[0060] The method comprises employing fine-grained classification for producttypes, location types and experience types.
[0061] The method comprises analyzing subtitles, captions and on-screen text fromthe Video File 108 for additional cues to identify products, locations andexperiences.
[0062] The method comprises extracting named entities and context from the onscreentext for additional cues to identify products, locations and experiences.
[0063] The method wherein extracting metadata comprises, at least one ofextracting name, brand, model, color, size, price, e-commerce links for the product;extracting place name, GPS coordinates, description, points of interest for thelocation; and extracting event name, event type, event organizer and event bookinglinks for the experience.
[0064] The method comprises correlating visual, audio and textual cues to improveidentification accuracy.
[0065] The method comprises using external databases for resolving ambiguities inidentifying products.
[0066] The method comprises providing an interface for editing to review, validateand edit the generated tags and metadata.
[0067] The method comprises tracking processing status, error rates, and systemperformance.
[0068] The method comprises providing analytics on tag density, recognitionaccuracy and content enrichment metrics.
[0069] The claimed subject matter has following technical advantages:
[0070] The subject matter uses a combination of computer vision, audio analysis,and natural language processing (NLP) to accurately identify products, locations,and experiences in the video content, leveraging both visual and contextual cues.
[0071] The subject matter generates tags (icons, QR codes, clickable overlays) foreach identified element, with context specific metadata (e.g., product type, locationcoordinates, event details). It associates each tag with precise timing (start / endframes or timestamps) and, where relevant, spatial coordinates within the videoframe (bounding boxes, regions of interest). It extracts and structures metadata foreach identified element, including product attributes, location data (GPS, placenames), and experience descriptors, supporting downstream search, analytics, andcommerce. Thus, the subject matter provides accurate and enough information tothe user for further actions.
[0072] The subject matter supports both real-time and offline (batch) processing,enabling third-party content creators or OTT platforms to enhance video librariesat scale before distribution.
[0073] The subject matter exports Enhanced Video Files 110 and associatedmetadata in standardized formats (e.g., JSON, XML, MPEG-7), enabling seamlessintegration with OTT platforms, interactive video players, and commerce systems.
[0074] The subject matter incorporates optional human review workflows forquality assurance, allowing editors to validate, correct, or enrich automaticallygenerated tags and metadata.
[0075] Reference numerals and their description:100 System102 Processor104 Memory106 Video Content Enhancement Manager108 Video File110 Enhanced Video File112 Input Output Interfaces114 Communication Unit200 Pre-processing Module202 Multimodal Recognition Module202A Visual Analysis Module202B Audio Analysis Module202C Text Analysis Module204 Contextual Correlation and Disambiguation Module206 Tag Generation and Placement Module208 Metadata Extraction and Structuring Module210 Quality Assurance and Human-in-the-loop Review Module212 Export and Integration Module214 Logging, Analytics and Monitoring Module
Claims
1. A method (300) for automated video content enhancement, the method (300) comprising: accessing (302) a Video File (108); extracting (304) data from the Video File (108), the data comprising video frames; processing (306) the extracted data to perform, at least one of: detecting objects; recognizing scenes; and recognizing an activity; identifying (308) at least one element from the group of elements comprising: a product from the detected objects; a location from the recognized scenes; and an experience from the recognized activity; generating (310) and assigning a tag for each of the identified elements, the tag comprising at least one of an icon, a QR code and a bounding box; determining (312) optimal timing and spatial placement for the tags; extracting (314) metadata for each element; structuring (316) metadata in standardized formats; and exporting (318) Enhanced Video File (110), the Enhanced Video File (110) comprising embedded tags and associated metadata.
2. The method (300) as claimed in claim 1, wherein processing the extracted data comprises processing the extracted data using deep learning models.
3. The method (300) as claimed in claim 1, wherein processing the extracted data comprises frame sampling and scene segmentation.
4. The method (300) as claimed in claim 1, comprising processing audio tracks for spoken references to identify products, locations and experiences.
5. The method (300) as claimed in claim 1, comprising using speech-to-text and keyword extraction to identify products, locations and experiences.
6. The method (300) as claimed in claim 1, comprising: classifying the products into product types; classifying the locations into location types; and classifying the experiences into experience types.
7. The method (300) as claimed in claim 6, comprising employing finegrained classification for product types, location types and experience types.
8. The method (300) as claimed in claim 6, wherein the classification is stored as part of metadata.
9. The method (300) as claimed in claim 1, comprising analysing subtitles, captions and on-screen text from the Video File (108) for additional cues to identify products, locations and experiences.
10. The method (300) as claimed in claim 1, comprising extracting named entities and context from the on-screen text for additional cues to identify products, locations and experiences.
11. The method (300) as claimed in claim 1, wherein extracting metadata comprises, at least one of: extracting name, brand, model, color, size, price, e-commerce links for the product; extracting place name, GPS coordinates, description, points of interest for the location; and extracting event name, event type, event organizer and event booking links for the experience.
12. The method (300) as claimed in claim 1, comprising correlating visual, audio and textual cues to improve identification accuracy.
13. The method (300) as claimed in claim 1, comprising using external databases for resolving ambiguities in identifying products.
14. The method (300) as claimed in claim 1, comprising providing an interface for editing to review, validate and edit the generated tags and metadata.
15. The method (300) as claimed in claim 1 comprising tracking processing status, error rates, and system performance.
16. The method (300) as claimed in claim 1 comprising providing analytics on tag density, recognition accuracy and content enrichment metrics.
17. A System (100) for automated video content enhancement, the System (100) comprising: a Processor (102); a Memory (104) coupled to the Processor (102); the Processor (102) configured to: access a Video File (108); extract data from the Video File (108), the data comprising video frames; process the extracted data to perform, at least one of: detecting objects; recognizing scenes; and recognizing an activity; identify at least one element from the group of elements comprising: a product from the detected objects; a location from the recognized scenes; and an experience from the recognized activity; generate and assign a tag for each of the identified elements, the tag comprising at least one of an icon, a QR code and a bounding box; determine optimal timing and spatial placement for the tags; extract metadata for each element; structure metadata in standardized formats; and export Enhanced Video File (110), the Enhanced Video File (110) comprising embedded tags and associated metadata.
18. The System (100) as claimed in claim 17, the System (100) comprising: a pre-processing module (200) to extract data from the Video File (108); a multimodal recognition engine (202) to: detect objects; recognize scenes; recognize activities; identify elements, the elements comprising: products form the objects; locations from the recognized scenes; and experiences from the recognized activity; a tag generation and placement module (206) to: generate and assign a tag for each of the identified elements, the tag comprising at least one of an icon, a QR code and a bounding box; and determine optimal timing and spatial placement for the tags; a metadata extraction and structuring module (208) to: extract metadata for each element; structure metadata in standardized formats; and an export and integration module (212) to export Enhanced Video File (110), the Enhanced Video File (110) comprising embedded tags and associated metadata.
