Method and system for providing context aware media streaming content
Patent Information
- Application Number
- IN202221068891
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2026-07-13
- Estimated Expiration
- 2042-11-30
AI Technical Summary
Current streaming media platforms lack context awareness, making it difficult for users to accurately skip or access specific scenes or characters within media content, and fail to provide personalized content recommendations, artwork customization, and targeted content insertion based on object and character recognition.
A method and system that utilize character and object recognition to index media content, allowing for precise navigation, customized artwork generation, and context-aware content streaming, enabling enhanced search, recommendation, and targeted content placement.
Enables users to seamlessly access specific content, provides personalized recommendations and artwork, and facilitates non-intrusive targeted content insertion, improving user experience and reducing resource consumption.
Smart Images

Figure 00000052_0000 
Figure 00000053_0000 
Figure 00000054_0000
Abstract
Description
TECHNICAL FIELD:The present invention generally relates to streaming of media content and moreparticularly to methods and systems for providing enhanced context aware mediastreaming including enhanced content search, content recommendation, targetcontent placement, content artwork customization based on indexing of mediacontent and object and character recognition.BACKGROUND OF THE DISCLOSURE:The following description of the related art is intended to provide backgroundinformation pertaining to the field of the disclosure. This section may includecertain aspects of the art that may be related to various features of the presentdisclosure. However, it should be appreciated that this section is used only toenhance the understanding of the reader with respect to the present disclosure,and not as an admission of the prior art.Streaming media platforms providing media content over the Internet havebecome increasingly popular in the last few years. OTT platforms provide video ondemand as well as live streaming content to the users. There are times when auser who has previously viewed a certain piece of multimedia material needs orwants to access it again at a later time. For instance, a person who previouslywatched multimedia content would desire to continue watching from a desiredpoint of his interest, or directly jump to a scene of his interest or preference (forexample, some scene which the user enjoyed watching in the past and wants towatch again or someone may have recommended to the user to watch, etc.).Typically, media players on the OTT platforms or the user devices provide a featurewhereby a user can forward or rewind the video by a fixed number of seconds, say10 or 15 seconds, to quickly skip some content (such as ads) or replay somecontent. Also, the media players on these platforms provide a navigation bar usingwhich the user can forward or rewind to a particular point on the timeline, or startexactly from the point where the user left as the media players on the user deviceswould save the time point on which the user paused or stopped playing the mediacontent. The existing OTT platforms / solutions heavily rely on timestamping, i.e.,the OTT platform may store the timestamp where the user last left the video sothat it can be resumed from that timestamp; similarly, the OTT platform mayforward or rewind content by 15 seconds by moving ahead / behind 15 secondsfrom the current timestamp.It is also often the case that users wish to skip certain scenes or watch a particularscene or character in a video / media content, however currently known solutionshave limited capability of allowing the users to skip certain scenes as the currentlyknown solutions fail to provide accurate information about scenes / characters / objects in the video / media content. If the user wants to skip certain scenes / watchany particular scene or character, he / she is required to drag the cursor on thenavigation / time progress bar (however, there is no guarantee that the user landson exactly the point he / she wants to land).At best in the currently existing systems, there is the feature of including a tableof contents providing one or more items, that allows a viewer to navigate to aparticular item within the content. This allows a very limited use case by which theuser can land at a particular point in time of the media content. These existingsystems allow the user to go to a point in time in the video where the nextitem / scene is displayed. However, the user may not wish to watch the itemspredefined in the table of contents but may wish to watch or skip a particularcharacter or object in that video. The existing solutions provide no flexibility to thiseffect as they are solely based on a predefined table of contents that are hardcoded with the video primarily based on the scene boundaries in the content.Therefore, the existing methods of providing multimedia content via OTTplatforms or other means using the internet require viewers to either watch theentire content from beginning to end or to haphazardly advance or rewindthrough it using the video player's controls (such as fast forward, rewind, etc.) orto watch the complete scene from the items predefined or marked in the contentand provided as a table of contents to the user.One major drawback of the existing systems is that these are not fully contextaware based on characters and objects recognized in the frames, or in other wordsthey are not able to automatically become aware of the context of the contentthat is being streamed on the user device. In order to identify any context, theexisting systems rely on manual intervention, where a human operator wouldmanually have to review the content to understand what is the context of thecontent being played or on automated or semi‐automated processing thatrequires heavy resource consumption and still be inefficient in terms of time andcosts. For providing any other enhancement with respect to the context, theexisting solutions rely on manual intervention. The existing solutions are unable toautomatically interpret or understand the context of the content being displayedand thus do not provide any enhanced features based on the automaticidentification of the context of the content.Since the existing systems are unable to automatically identify the context of thecontent based on characters and objects recognized in the frames in the mediastream, they also do not provide for automatic in‐video feed targeted contentinsertions (such as advertisements) based on such context, without the need formanual intervention. Further, the existing systems are also unable to provide forsuggesting / recommending content of interest to the users, the content that isbased on recognition of certain character(s) or object(s) present in a mediastream. Also, the targeted content provided in live video feeds, are often done bymanual intervention or semi‐automated systems. These targeted content are notdependent and / or not related to the content being displayed during the live videofeed.Similarly, since the existing systems are unable to automatically identify thecontext of the content in the media stream, they also do not provide for enhancedsearching based on context. The existing solutions are able to provide searchfeature only based on the titles of the media, top content, etc. Since the existingsystems do not have any context of the content in the media stream, they areunable to provide search based on such context. And therefore, the search resultsmay not always be accurate using the existing systems.Furthermore, existing systems are unable to provide character‐basedrecommendations to the users as the existing systems are unable to automaticallyidentify the context of the content in the media stream. The existing solutions onlyknow those characters of a particular media such as a movie, that have been listedin the description of that media. However, the existing systems are unable tocapture all the characters of the media based on the broad description providedand thus are unable to provide character based recommendations to the users.Also, the existing solutions provide fixed artwork such as previews, thumbnails,teasers, trailers, etc., for each media content. That is to say that for a particularmedia content, say a movie, the existing solutions provide the same thumbnail,preview, trailer and teaser of the movie to all the users, irrespective of theinterests of that particular user. This is again because the existing solutions areunable to identify the context of the content and thus are unable to provide anycustomization on the artwork based on such context and user preferences.Additionally, there are existing systems that provide for dynamic or customizedartwork to some extent, but a lot of computational processing and relatedhardware resources are needed to implement the same.Yet another limitation of the existing solutions is that they are unable to providedetailed and relevant description of the characters and / or objects in a mediastream since they are unable to understand the context of the media stream.Similarly, the existing solutions are also unable to provide the user with shoppablee‐commerce linking in the media content due to the lack of automatic contextawareness of the media content.In order to solve the above and other related inherent problems, it is an imperativeneed to create a method and system that can provide enhanced context awaremedia streaming including enhanced content search, content recommendation,target content placement, content artwork customization, basis indexing of mediacontent based on object and character recognition.SUMMARY OF THE DISCLOSUREThis section is provided to introduce certain aspects of the present disclosure in asimplified form that are further described below in the detailed description. Thissummary is not intended to identify the key features or the scope of the claimedsubject matter.A first aspect of the present disclosure is related to a method for providing acontext aware media streaming content on one or more user devices. The methodcomprises receiving, at a character and object recognition module, a set of framesof a media streaming content. Further, the method encompasses analyzing, by thecharacter and object recognition module, at least one selected frame from the setof frames of the media streaming content. Furthermore, the method encompassesidentifying, by the character and object recognition module at least one of one ormore characters and one or more objects based on the analysis of the at least oneselected frame. Thereafter, the method comprises: indexing, by a processing unit,at least one frame from the at least one selected frame that encompasses at leastone of the one or more characters and / or the one or more objects; andgenerating, by the processing unit, one or more customized artworks based on theindexing of the at least one frame from the at least one selected frame and a setof user attributes. Subsequently, the method encompasses generating, by theprocessing unit, the context aware media streaming content based at least on theone or more customized artworks; and providing, by the processing unit, thecontext aware media streaming content on the one or more user devices.Another aspect of the present disclosure relates to a system for providing acontext aware media streaming content on one or more user devices. The systemcomprises a character and object recognition module that is configured to:receive, a set of frames of the media streaming content; and analyze, at least oneselected frame from the set of frames of the media streaming content. Further,the character and object recognition module is configured to identify at least oneof one or more characters and one or more objects based on the analysis of the atleast one selected frame. Thereafter, the system comprises a processing unit thatis configured to: index, at least one frame from the at least one selected framethat encompasses at least one of the one or more characters and the one or moreobjects; and generate one or more customized artworks based on the indexing ofthe at least one frame from the at least one selected frame and a set of userattributes. Subsequently, the processing unit is configured to: generate thecontext aware media streaming content based at least on the one or morecustomized artworks; and provide, the context aware media streaming content onthe one or more user devices.BRIEF DESCRIPTION OF DRAWINGSThe accompanying drawings, which are incorporated herein, and constitute a partof this disclosure, illustrate exemplary embodiments of the disclosed methods andsystems in which like reference numerals refer to the same parts throughout thedifferent drawings. Components in the drawings are not necessarily to scale,emphasis instead being placed upon clearly illustrating the principles of thepresent disclosure. Some drawings may indicate the components using blockdiagrams and may not represent the internal circuitry of each component. It willbe appreciated by those skilled in the art that disclosure of such drawings includesdisclosure of electrical components, electronic components or circuitry commonlyused to implement such components.Figure 1a illustrates an exemplary diagram depicting an exemplary networkarchitecture diagram [100a], in accordance with exemplary embodiments of thepresent disclosure.Figure 1 illustrates an exemplary block diagram of a system
[100] to provide acontext aware media streaming content on one or more user devices, inaccordance with exemplary embodiments of the present disclosure.Figure 2 illustrates an exemplary method flow diagram
[200] for providing acontext aware media streaming content on one or more user devices, shown inaccordance with exemplary embodiments of the present disclosure.Figure 3 illustrates an exemplary process
[300] for identifying the one or morecharacters in accordance with exemplary embodiments of the present disclosure.The foregoing shall be more apparent from the following more detaileddescription of the disclosure.DETAIL DESCRIPTION OF THE DISCLOSUREIn the following description, for the purposes of explanation, various specificdetails are set forth to provide a thorough understanding of embodiments of thepresent disclosure. It will be apparent, however, that embodiments of the presentdisclosure may be practiced without these specific details. Several featuresdescribed hereafter can each be used independently of one another or with anycombination of other features. An individual feature may not address any of theproblems discussed above or might address only some of the problems discussedabove.The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability, or configuration of the disclosure.Rather, the ensuing description of the exemplary embodiments will provide thoseskilled in the art with an enabling description for implementing an exemplaryembodiment. It should be understood that various changes may be made in thefunction and arrangement of elements without departing from the spirit andscope of the disclosure as set forth.Specific details are given in the following description to provide a thoroughunderstanding of the embodiments. However, it will be understood by one ofordinary skills in the art that the embodiments may be practiced without thesespecific details. For example, circuits, systems, processes, and other componentsmay be shown as components in block diagram form in order not to obscure theembodiments in unnecessary detail.Also, it is noted that individual embodiments may be described as a process whichis depicted as a flowchart, a flow diagram, a data flow diagram, a structurediagram, or a block diagram. Although a flowchart may describe the operations asa sequential process, many of the operations can be performed in parallel orconcurrently. In addition, the order of the operations may be re‐arranged. Aprocess is terminated when its operations are completed but could haveadditional steps not included in a figure.The word "exemplary" and / or "demonstrative" is used herein to mean serving asan example, instance, or illustration. For the avoidance of doubt, the subjectmatter disclosed herein is not limited by such examples. In addition, any aspect ordesign described herein as "exemplary" and / or "demonstrative" is not necessarilyto be construed as preferred or advantageous over other aspects or designs, noris it meant to preclude equivalent exemplary structures and techniques known tothose of ordinary skill in the art. Furthermore, to the extent that the terms"includes," "has," "contains," and other similar words are used in either thedetailed description or the claims, such terms are intended to be inclusive-in amanner similar to the term "comprising" as an open transition word-withoutprecluding any additional or other elements.To address the problems mentioned in the background section, solutions havebeen developed where indexing is done in the video content, but it is done on thescene boundaries, for example, to enable a user to skip or watch a particular scenein the video content. Also, some existing systems may implement solutionspertaining to object recognition and some other systems may implementcharacter recognition separately, but do not provide a combined solution thatdoes not have compatibility issues. Such compatibility issues may also arise due todifferent or separate levels of processing that a content has to go through forobject recognition and character recognition. Further, there exists no solution forproviding enhanced context aware media streaming including enhanced contentsearch, content recommendation, target content placement, content artworkcustomization, etc. basis indexing of media content based on object and characterrecognition. This indexing based on the characters and / or objects in the live orpre‐recorded video content may enable a person to obtain several applications,which are not possible to obtain with the currently existing systems, such asaccessing metadata related to the indexed characters and / or objects, insertingtargeted content related to the recognized characters and / or objects in livestreamed videos, generating and providing customized artwork to the users,allowing to search for and skip to a particular character or object in the mediastream, etc.The present invention relates to methods and systems for enhanced contextaware media streaming basis indexing of media content based on object andcharacter recognition in the media being streamed. The content can be prerecordedas well as live streamed. This indexing helps users to navigate throughthe media stream to select the exact instance at which the user wants to skipand / or play the media. The indexing of content provides various other features tothe users, as discussed above, which the currently existing solutions implementingindexing based on scene boundaries in the content are unable to offer. Thus, inthe present invention, the indexing of content may not be limited to sceneboundaries, but the indexing of the content can be applied to characters andobjects recognized in the frames of the media content as well. Thus, the presentdisclosure relates to providing a method and a system that indexes frames, basedon identified characters and objects within the timeline of a video content. So, ifthe user wants to skip a particular part of the content or wishes to skip or watch acontent of a particular character, he / she can easily do so by selecting the desireditem / frame from an indexed content.As used herein, a "processing unit" or "processor" or "operating processor"includes one or more processors, wherein processor refers to any logic circuitryfor processing instructions, said processing unit may reside at server side (i.e., thenetwork server of the OTT platform) or client side (i.e., the user device). Theprocessing unit may be centralized or decentralized (distributed). A processor maybe a general‐purpose processor, a special purpose processor, a conventionalprocessor, a digital signal processor, a plurality of microprocessors, one or moremicroprocessors in association with a DSP core, a controller, a microcontroller,Application Specific Integrated Circuits, Field Programmable Gate Array circuits,any other type of integrated circuits, a graphics processing unit etc. Also, theprocessor or the processing unit may comprise one or more processors for variousoperations. The processor may perform signal coding data processing,input / output processing, and / or any other functionality that enables the workingof the system according to the present disclosure. More specifically, the processoror processing unit is a hardware processor. Furthermore, to execute certainoperations, the processing unit / processor as disclosed in the present disclosuremay include one or more Central Processing Unit (CPU) and one or more GraphicsProcessing Unit (GPU), selected based on said certain operations. Furthermore,the graphics processing unit (GPU) is a specialized electronic circuit designed torapidly manipulate and alter a memory to accelerate a creation of images in aframe buffer intended for output to a display device.As used herein, "storage unit" or "memory unit" refers to a machine or computerreadablemedium including any mechanism for storing information in a formreadable by a computer or similar machine. For example, a computer‐readablemedium includes read‐only memory ("ROM"), random access memory ("RAM"),magnetic disk storage media, optical storage media, flash memory devices orother types of machine‐accessible storage media. The storage unit can be any typeof storage unit such as Cloud or CDN (content delivery network) storage, public,shared, private, telecommunications operator based storage, or any other type ofstorage known in the art or may be developed in future. The storage unit stores atleast the data that may be required by one or more units of theserver / system / user device to perform their respective functions.A 'smart computing device' or 'user device' refers to any electrical, electronic, orelectromechanical equipment or a combination of one or more of the abovedevices. Smart computing devices may include, but not limit to, a mobile phone,smart phone, pager, laptop, a general purpose computer, desktop, personal digitalassistant, tablet computer, mainframe computer, smart television, gamingconsoles, media streaming devices, or any other computing device as may beobvious to a person skilled in the art. In general, a smart computing device is adigital, user configured, computer networked device that can operateautonomously. A smart computing device is one of the appropriate systems forstoring data and other private / sensitive information.A "smartphone" is one type of "smart computing device" that refers to themobility wireless cellular connectivity device that allows end users to use serviceson 2G, 3G, 4G, 5G or other upcoming generations of mobile broadband Internetconnections with an advanced mobile operating system which combines featuresof a personal computer operating system with other features useful for mobile orhandheld use. These smartphones can access the Internet, have a touchscreenuser interface, can run third‐party apps including capturing images, and arecamera phones possessing high‐speed mobile broadband internet with videocalling, hotspot functionality, motion sensors, mobile payment mechanisms andenhanced security features with alarm and alert in emergency situations.The present disclosure proposes a unified and enhanced solution in the form of asystem and method for providing enhanced context aware media streamingincluding enhanced content search, content recommendation, target contentplacement, content artwork customization basis indexing of media content basedon object and character recognition.The present disclosure is further explained in detail below with reference now tothe diagrams so that those skilled in the art can easily carry out the presentdisclosure.Referring to figure 1a that illustrates an exemplary diagram depicting anexemplary network architecture diagram [100a], in accordance with exemplaryembodiments of the present disclosure. As shown in Figure 1a, the exemplarynetwork architecture diagram [100a] comprises a set of user devices (UD) [101(1)],[101(2)], ….[101(n)] (hereinafter collectively referred to as user device
[101] or theset of user device
[101] or the one or more user devices
[101] for clarity purpose)in communication with at least one server device
[107] , and a content deliverynetwork
[109] wherein in an implementation the server device
[107] incombination with the content delivery network
[109] further comprises a system
[100] configured to implement the feature of the present disclosure. In animplementation system
[100] may be in connection with the server device
[107] and the user device
[101] , in a manner as obvious to a person skilled in the art toimplement the features of the present disclosure.Also, in Figure 1a only a single server device
[107] is shown, however, there maybe multiple such server devices
[107] or there may be any such numbers of saidserver device
[107] as obvious to a person skilled in the art or as required toimplement the features of the present disclosure.In general, the content delivery network (CDN)
[109] is a geographicallydistributed network of servers strategically placed in various locations around theworld. CDNs facilitate efficient delivery of content, such as personalized contentlayout / or catalogue (whether dynamic or static), web pages, images, videos, andother digital assets, to end‐users, reducing latency and improving load times.The CDN
[109] is further connected to various other CDNs (not shown in figures)present at different geographical locations, creating a hierarchy of interconnectednetworks. Each CDN in the hierarchy can have its own set of servers (e.g., theserver device
[107] ) located in different regions, forming a distributed networkinfrastructure. It is to be noted that the CDN may be public CDN, private CDN,Telco CDN or the ISP CDN.When the user device
[101] communicates with the content delivery serversystem
[109] after initial authentication or validation of subscription of the uservia the user device, the communication request is routed through the CDNnetwork. The CDN
[109] then uses intelligent routing algorithms to direct thecommunication request to the nearest server or the server with the lowest load(e.g., the server device
[107] ), minimizing the distance and optimizing responsetimes. After the direction of the communication request, the content cataloguelayout is then routed through the CDN
[109] .Further, in the implementation where the system
[100] is present in the serverdevice
[107] , based on the implementation of the features of the presentdisclosure, a context aware media streaming content is generated via the system
[100] and then provided on the one or more user devices
[101] .Further, referring to figure 1 that illustrates an exemplary block diagram of asystem
[100] to provide a context aware media streaming content on one or moreuser devices, in accordance with exemplary embodiments of the presentdisclosure.The present invention includes a system
[100] comprising at least a character andobject recognition module
[102] ; a processing unit
[104] ; and a storage unit
[114] .The character and object recognition module
[102] comprises at least one of aface detector module
[106] , a face alignment module
[108] , a face recognitionmodule
[110] and an object recognition module
[112] . The system componentsmay be present at the same location or may be distributed at different locations.Also, a component of the system
[100] may comprise one or more subcomponentswhich may be centralized or distributed at various locations and maytogether be referred to as that particular component. In an exemplaryimplementation, the invention includes a system
[100] comprising, but not limitedto an input / output module, a communication module, and a repository ofcharacters / objects.The character and object recognition module
[102] is configured to receive, a setof frames of a media streaming content. The media streaming content is one of alive video content and a pre‐recorded video content. Further, the character andobject recognition module
[102] is configured to analyze at least one selectedframe from the set of frames of the media streaming content. The at least oneselected frame from the set of frames is selected based on at least one of a logicalselection and a random selection. Furthermore, the character and objectrecognition module
[102] is configured to identify at least one of one or morecharacters and one or more objects based on the analysis of the at least oneselected frame. Particularly, to identify the one or more characters, the facedetector module
[106] of the character and object recognition module
[102] isconfigured to localize one or more faces in the set of frames of the mediastreaming content. Thereafter, the facial alignment module
[108] of the characterand object recognition module
[102] , is configured to align the one or morelocalized faces to normalized canonical coordinates. Next, the face recognitionmodule
[110] of the character and object recognition module
[102] is configuredto implement a facial recognition method on the one or more aligned localizedfaces. Also, the face recognition module
[110] is then configured to identify theone or more characters based on the implementation of the facial recognitionmethod on the one or more aligned localized faces.Therefore, according to the implementation of the features of the presentdisclosure, the character and object recognition module
[102] is configured toidentify faces (i.e., characters from their faces or from their overall appearance),and objects, if any, for selected frames in a streaming video content (i.e., themedia streaming content). The present disclosure encompasses logically and / orrandomly selecting some frames in a video content (i.e., the media streamingcontent). For example, while logically selecting the frames, the character andobject recognition module
[102] may analyse a background and / or a foregroundof one or more consecutive frames in a content (i.e., media streaming content),may be through a quick screening of frames, and pick the frames when there is adrastic change in the background and / or the foreground of the frames or thechange is above a threshold level as specified in the system
[100] . A person skilledin the art would appreciate that the above example is provided for understandingpurposes only and does not restrict the invention in any possible manner, andaccordingly, any other logic is also possible for selecting one or more frames fromthe set of frames. In an implementation, each frame on which a process of objectand character recognition is performed, or a frame on which indexing is done,should be understood as a selected frame throughout this disclosure. Also,character and object recognition may not be done one by one for a same content,that is for example, by first identifying characters in each frame of the content andthen identifying objects in said each frame of the content, but may be donesimultaneously where both the objects as well as characters are identified at oncein a frame of the content.Therefore, in accordance with the above disclosure, the streaming video contentmay be a live video content or a pre‐recorded video content. The character andobject recognition module
[102] may further comprise the face detector module(face anti‐spoofing)
[106] , the face alignment module
[108] , and the facerecognition module
[110] . In an exemplary implementation, to implement thefeatures of the present disclosure, firstly, the face detector module
[106] receivesthe streaming video content and localizes faces in each frame of the streamingvideo content. The received streaming video content may be retrieved from thestorage unit
[114] of the system
[100] or from an external server. Second, the facialalignment module
[108] aligns the localized / identified faces to normalizedcanonical coordinates. Third, the face recognition module
[110] implements afacial recognition method on the aligned faces to identify one or more charactersin the streaming video content. In an implementation a character number isassigned to each of the identified one or more characters and stored in the storageunit
[114] for easy extraction of a data related to the identified one or morecharacters to implement the features of the present disclosure. The facerecognition module
[110] may implement face matching steps based on at leastone of a threshold matching, neural network identification, metric learning, etc. Inan implementation, the face recognition module
[110] may include a featureextraction module, and a loss function calculator.Also, in an implementation, the character and object recognition module
[102] isa pre‐trained system that is trained on a training dataset, wherein the trainingdataset comprises a data associated with at least one of a plurality of images ofcharacters and a plurality of objects.Also, in an implementation the training dataset may include images of charactersretrieved from popular search engines and specific databases of characters. Thedataset enforces the pre‐trained system to address a wide range of intraclassvariations, such as lighting, age, and pose, which have a limited number of subjectsbut many images for each subject. In an instance implementation, the datasetsused for training may include 3 to 2469 images of each character / subject for thepurposes of training. The training data / training dataset may include metadata,social media profiles of characters, predefined cast of streaming video content,predefined objects database, standard object shapes, geometry, object labels suchas brands, logos, titles, etc.In another implementation, the character and object detection module
[102] ofthe system via the object recognition module
[112] is also configured to detectand recognize the one or more objects such as including but not limiting to one ormore costumes that some characters are wearing, or one or more places thatare shown in one or more scenes or clips in a video content, or certain otherobject(s) in a frame of a content, for example phones, computers, cars, food,refrigerator, water bottle, etc. these are identified and / or detected and / orrecognized based on various techniques. For example, a comparison of the one ormore objects to a dataset as defined above may help in identifying the one or moreobjects and tagging. Also, the above examples of objects and / or characters areprovided for understanding purposes only and do not restrict or limit the presentdisclosure in any possible manner. Further, the character and object recognitionmodule
[102] implements various techniques for object detection purposes, suchas Region‐Based Convolutional Neural Networks (R‐CNN) models. Also, othertechniques, either known or that may be developed in future, related to imageobject detection or video object detection may be implemented for this purpose.Also, a combination of known techniques may also be developed forimplementation of the object detection purpose. For example, 'optical flow' is atool that calculates and traces a movement of all items present in an image. Ittakes all pixels in an image of a video frame and analyzes their trajectorythroughout the video frame. So, a combination of 'image detection' techniquewith 'object tracking' technique may be implemented for a fast and accurateobject detection in a streaming video.Further, these techniques of character and object recognition are applied by thesystem
[100] before transmitting a content data for distribution on one or morenetworks for a set of viewers. Thus, the character and object recognition may bedone in run‐time, i.e., when the video streaming content is being streamed to auser's device. Further, the present disclosure encompasses that the character andobject recognition is done by the character and object recognition module
[102] in near real‐time for live streaming video content and may be pre‐performed forvideo on demand content.Further, once the character(s) and the object(s) in one or more scenes / shots ofthe media streaming content have been recognized by the character and objectrecognition module
[102] , the processing unit
[104] of the system
[100] isconfigured to index, at least one frame from the at least one selected frame thatencompasses at least one of the one or more characters and the one or moreobjects. In an implementation, all relevant frames of the streaming video contentwith the recognized character(s) and / or the recognized object(s) in each suchframe are indexed. The present disclosure encompasses that the indexing isperformed against a playtime length on a progress bar of the media streamingcontent, and the indexing is displayed on the progress bar by at least one of oneor more markings, one or more colour codes, one or more character names, oneor more emojis, and one or more actual frames in the media streaming content.Further, the processing unit
[104] is configured to index, the at least one framefrom the at least one selected frame by inserting in the at least one frame fromthe at least one selected frame a set of content to be provided on the one or moreuser devices.For instance, in an implementation, indexing of one or more videos may leveragean insertion of a targeted content such as promotion campaigns, advertisements,etc. in a video content of the one or more videos. In general, the targeted contentis considered intrusive to users and the users are not interested in watching thattargeted content or click on the targeted content to visit web pages linked withthat targeted content to utilise the same. The solution to this problem of showingintrusive targeted content to the users, that the present invention encompasses,is by providing in‐feed targeted content that is not intrusive or are less intrusive.Also, in an implementation, the users may be given a choice for viewing targetedcontent where the users can select whether the targeted content should be shownto them or not.In another implementation, to index the at least one frame from the at least oneselected frame, the present disclosure also encompasses insertion of one or morecontent hyperlinks in the at least one frame that are relevant to user interests andcustomised based on user behaviour, past history of user(s), or any other userattributes. Also, in an implementation, the system
[100] is configured to learn theuser characteristics / behaviour / pattern by implementing a machine learningmodel. This machine learning model may be dynamically trained and updatedperiodically.In already existing systems, this can be done through manual intervention whereinan operator first scans a whole content of a media stream and then marks one ormore pointers for embedding in‐feed targeted content in the media stream, or ina semi‐automated manner where a system identifies the one or morepointers / spots for the operator to add / mark the targeted content. However, inthe present invention, this manual intervention of the operator is done away withas the system
[100] can itself automatically detect object(s) or character(s) in aframe of a streaming content and suggests content insertion opportunities in theframe. In an exemplary implementation, for this purpose of content insertion thesystem
[100] communicates with at least one of a campaign server and anadvertisement server and fetches relevant content to be inserted based on thedetected object(s) or character(s). In another exemplary embodiment, theinvention encompasses manual indexing of character(s) and object(s) by a userwatching a content. This indexing done by the user watching the content mayfurther be verified at an operator's end either manually or in an automated orsemi‐automated manner and then made part of the content or rejected asneeded, based on the verification results.For this purpose of indexing the at least one frame from the at least one selectedframe, in an implementation, a target content recommendation server may beconnected to the processing unit
[104] . This target content recommendationserver comprises all contents that can be recommended to a user duringstreaming of a media streaming content on a user device of the user. Wheneverthe system
[100] detects a content recommendation opportunity that is ofrelevance with respect to a user, it is directed to the target contentrecommendation server to search for any recommended content related to anidentified object or an identified character and / or customized to a user interest ora user preference, that can be displayed on a screen or a user interface of a userdevice of the user at a particular instance of a media streaming content beingdisplayed.After indexing and markings are done, a table, i.e., a tabular array comprising, sayfor example, multiple rows and columns may be maintained in a file. Say, thetabular array has all timestamps of the media streaming content where there is acharacter and / or object available in the media streaming content and a list ofcorresponding characters and / or objects detected in the media streamingcontent. It will be appreciated by those skilled in the art that an indexed contentmay be stored in any other format and the present disclosure is not limited tostoring the indexed content in a single tabular array as mentioned above.Further, the processing unit
[104] is configured to generate one or morecustomized artworks based on the indexing of the at least one frame from the atleast one selected frame and a set of user attributes. The one or more customizedartworks comprise at least one of one or more previews, one or more thumbnails,one or more teasers, one or more trailers, one or more images, and one or moreshort videos, related to at least one of the one or more characters and the one ormore objects. In an implementation in the present disclosure the processing unit
[104] may refer to a table of contents as well as one or more user attributes thatare already learned by the system
[100] by implementing at least one of a machinelearning model, user past search history, user watch history and such userinteraction that is obvious to a person skilled in the art and decide whether anyitem in the table, that is, any object or character is of relevance to the userattribute(s). For example, if a user is interested in a cartoon movie, then the userin this case might be interested in graphics, the user in another case might beinterested in toys, etc. The system
[100] after referring to both a timeline or theindexed content and the user attribute(s) generates the one or more customizedartworks and provides such customized artwork(s) to the user. Moreover, in animplementation, the present invention encompasses generating the one or morecustomized artworks for user(s) based on one or more user preferences of theuser(s). This customized artwork(s) includes at least one of: one or more previews,one or more thumbnails, one or more teasers, one or more trailers, one or moreimages, one or more short videos, (such as a *.gif file, etc.) etc. for a datarecommendation related to character(s) or object(s) in the streaming mediacontent for the user(s). For example, a user A is interested in watching comedycontent and a user B is interested in watching a content related to an Artist X. Nowsay, in a same video content C, the artist X performs a comedy act. Thus, the videocontent C may be of interest to the user A as well as the user B. However, thesystem
[100] in this implementation, may generate different artworks for the userA and the user B. For example, a custom artwork shown to the user A maycomprise an image or short video, etc. which suggests that the video content C isa comedy content. And, a custom artwork shown to the user B may comprise animage or short video, etc. which suggests that the video content C includes workof the Artist X. Thus, in this implementation, the system
[100] first searched for aparticular user, that whether a video content contains anything that may be ofrelevance or interest for the particular user, and based on user preferences of saidparticular user, the system
[100] also generates customized artwork for theparticular user, further enhancing user experience. This is possible because thevideo content is indexed and the system
[100] is context‐aware basis indexes inthe video context, and also the system
[100] considers user attributes to generateand provide customized artwork(s) to user(s). So continuing with the aboveexample, say, a video content Z has a short segment featuring the Artist X and thesystem
[100] identifies that the Artist X is relevant to the user B and also identifiesa content C in a target content recommendation server of the system
[100] whichfeatures the Artist X, then the system
[100] automatically generates a customizedartwork of the content C for the user B where the customized artwork suggeststhat the content C features the Artist X, and system
[100] provides the customizedartwork with the video content Z that is being displayed to the user B on the userdevice. Also, in an implementation, following the above example, if more than onecontents similar to the content C for the user B are identified, say the content C,content D, content E and content F are identified as relevant to the user B as allthe contents C, D, E and F feature the Artist X, then all the contents C, D, E, and Fmay be displayed in one or more customized artworks created for the user B. Inone instance, contents C, D, E, and F may be displayed to the user B (where thecustomized artwork may comprise short clips of videos, images, etc. for eachcontent C, D, E, and F, being displayed ) on the user device, or may be scrollableby the user B (where multiple artworks (i.e., customized artworks) are created formultiple contents and the artworks are displayed on a user device of the user Band the user B can scroll through the same) on the user device to access the same.Therefore, the user B may directly watch / play the scenes associated with the ArtistX through a selection of the contents C, D, E, and F displayed to the user B withoutwatching / playing the entire content. Also, this display of content C, D, E, and F,may be made to the user B using a playlist file such as M3U8 file, etc. Also, in animplementation, a standard artwork is taken by the system
[100] from otherdigital platforms, and checked whether the standard artwork is customizable for auser or not. In case the standard artwork is customizable, the standard artwork iscustomised by the system
[100] , and the customised artwork is shown to the user.Alternatively, if the standard artwork is not customizable according to the user,then the standard artwork is shown to the user.Furthermore, the processing unit
[104] is configured to generate the contextaware media streaming content based at least on the one or more customizedartworks. In an implementation to generate the context aware media streamingcontent the processing unit
[104] is further configured to modify the mediastreaming content based on one of a manifest file associated with the one or morecustomized artworks and a hardcode of the one or more customized artworks. Asused herein hardcoding the one or more customizes artworks implies embeddingthe one or more customized artworks directly to the media streaming content.Thereafter, the processing unit
[104] is configured to provide the context awaremedia streaming content on the one or more user devices.For instance, in an implementation, to provide a context aware media streamingcontent on a user device, a media streaming content modified based on acustomised artwork or a link for the customised artwork may be sent to the userdevice using a manifest file. In an embodiment, this customised artwork isgenerated prior to encoding the video content for delivery to the user device. Forinstance, if the user has requested for an on‐demand video like a movie that hasalready been indexed and the system
[100] also knows the user's preferences, insuch a scenario, the customised artwork is generated prior to encoding the videofor transmission to the user's device. In another embodiment, the customisedartwork is generated after the encoding of the video content for transmission tothe user device. For instance, if the user has requested for a live content, theindexing for such content is done in real time, so customised artwork may not begenerated prior to encoding. In another instance, if a new user has requested foran on‐demand video, the user preferences are not known to the system
[100] , insuch a scenario also a customised artwork may not be generated prior to encodinga video for transmission but only after the user has started watching the video andhas provided some inputs to the system
[100] that assist in deriving such user'spreferences. In another embodiment, the system
[100] may generate customizedartwork for a user basis the basic demographics of the user and a pool of similarusers (that is, by considering the preferences of other users with same or similardemographics) or any such correlation between the user and other users for whichthe system already has the data available.The processing unit
[104] is further configured to generate a customized targetedcontent based on the context aware media streaming content. Thereafter, theprocessing unit
[104] , is configured to provide on the one or more user devices,the customized targeted content based on a historical user pattern associatedwith the one or more user devices. In an implementation, the present inventionalso encompasses generating and providing the customized targeted contentbased on the indexing of the video content (i.e., the media streaming content).These customized targeted content may be shown to the viewer based on the pastbehavior of the viewer. For instance, say a video has been indexed with objects,for example, at particular time point in the video, the system
[100] knows throughindexing that characters are talking on a phone, and / or discussing about a mobilephone that is being displayed in the video streamed on a user device. In thisexample, a pop‐up window in parallel occupying a separate area of a screen of theuser device may be displayed wherein a targeted content of said phone may beshown. Alternatively, a link for such targeted content via a button or thumbnail orany other form accessible to the viewer may be shown on the screen or userinterface of the user device. This will not be intrusive to the user as the content isalready being played on the screen, and also as the targeted content is related tothe video being played. For example, there is a communication happeningbetween 2 persons in a video content regarding a product, the video contentbeing, say, watched by 3 users at a time, User A, User B and User C on theirrespective user devices. And the system
[100] finds that the product is relevant tothe User C and not relevant to the User A and the User B. Now, the system
[100] will insert the targeted content only for the User C and not for the User A and theUser B. Second, the targeted content inserted for the User C will be non‐intrusivefor the User C in the sense that it may be just provided, say for example, as ahyperlink in the form of a watermark at a corner of the screen, or say, if theproduct is shown in the frame, then the product may be clickable by the User C onthe user interface on which the User C is watching the video content, or may beprovided in a separate tray accessible to the User C on the user device, or in anyother such non‐intrusive manner. Also, the targeted content shown may bedisplayed on the whole screen or a part of the screen, and the video content beingdisplayed on the screen may be paused or stopped or may be played in parallel asper viewer's desire. Alternatively, an ad content such as a brand name (for in‐feedtargeted content) may be embedded with a streaming video content, such as onan object that is in a frame of the streaming video content. For example, a waterfilter that is in a content being displayed on a user screen may be displayed with abrand name embedded on that water filter. Also, in an implementation, the brandname may be hyperlinked to a respective / relevant web page, and a mark or anyother identifying factor may be presented to the user to access the web page, etc.For instance, the hyperlink could direct the user to an ecommerce website pagefor buying the product for which the targeted content has been inserted.In an alternate embodiment, a content may be paused and details or anyadditional data related to a targeted content or character(s) or object(s) in apaused frame of the content, is displayed to the user while the video is paused.Further, the system
[100] is aware of a context in the media streaming contentbeing displayed to the one or more user devices. For example, there is somecontext of soft‐beverages going on in a content being streamed to a user on a userdevice. Here, the system
[100] identifies an opportunity to insert a targetedcontent of a soft beverage to the user. Further, say in this example, the system
[100] has multiple options of displaying various soft beverages on the user device.Thus, the insertion of the targeted content of the soft beverage in this example,may be based on a pre‐poll conducted by the system
[100] on the user device, orother user patterns from a user data, from which the system
[100] is aware ofwhat option from the multiple options of soft beverages will be most relevant tothe user. Similarly, in the same example, when the system
[100] identifies anopportunity to show a web‐page related to a content of the soft beverages, thesystem
[100] may identify that the user may be interested in a short video contenton how beverages are made, or may be interested in reading some content on theinternet to learn about the chemical composition of soft beverages, etc. Thesystem
[100] thus shows the user the relevant link for the same. Thus, the presentinvention not only enables the system to identify ad opportunities based on theindexed content, i.e., identify where ads can be shown to the user, but alsoprovides customized targeted content to the user based on the user's preferences.For VOD (video on demand), the system
[100] may already have a contentavailable beforehand and therefore the indexing as per the present invention canbe done and presented to the user(s) after processing and insertion of the desiredtargeted content and / or data relevant to the user(s) at required instances of themedia streaming content. For example, there is a pre‐recorded content availableat a server in connection with the system
[100] . Also, in an implementation thesystem
[100] has a repository comprising the one or more user attributes andother data related to user(s). Further, since a video content data may be indexedbeforehand in this case, a data presented to a user may be hardcoded with one ormore relevant insertions related to object and character recognition in the videocontent, i.e., an indexing of a content and user attribute(s). In an implementation,the content available beforehand may not be hardcoded for user(s) and a separatemanifest file, for example, a child manifest, may be inserted for recommendationsto the user(s). In this implementation, an application running in a user device ofthe user(s) may pull the manifest file and show relevantcontent / recommendations to the user(s) basis opportunities detected by thesystem
[100] in the video content being streamed on the user device.In an implementation, the pre‐recorded content is available and encoded by thesystem
[100] following the steps of shortlisting of frames, indexing of content inthe shortlisted frames as explained above, which also comprises videocompression as per a video codec used. This encoded content is now available inall resolutions as desired by an operator who needs to transmit a content to a user.Thus, a media content player on a user device may be able to select, based onvarious parameters, as to which resolution to play on the user device. In animplementation, while indexing is being done in the media streaming content, saythe system
[100] identifies some character or object in one of the frames of themedia streaming content, it immediately contacts a target contentrecommendation server to find some relevant content that can be connected withthe media streaming content at that point of time in the media streaming content.For this purpose, a content title data or any such other content identification data,and a short clip which may comprise a small number of frames, say, 1‐2 frameswhere the system
[100] has identified the object(s) or character(s), are sent to acontent server. Further the target content recommendation server is configuredto compare the objects and / or characters with one or more existing objects in thecontent server. In an implementation, a target content recommendation servermay be contacted for displaying targeted content. Further, the targeted contentis given to the content delivery network (CDN)
[109] for displaying to user(s). In animplementation, a SCTE marker (such as standard markers SCTE 35, SCTE 104, etc.as developed by Society of Cable and Telecommunications Engineers as generallyknown in the art) may be made for suggesting or informing the system
[100] aboutan opportunity to contact target content recommendation server, or to let thesystem
[100] play the media streaming content as it is without displaying arelevant content from the target content recommendation server to the user(s).Thus, while indexing of the media streaming content, the system
[100] keepsthings ready in the background and when the media streaming content is beingplayed on the one or more user devices the system
[100] may connect to thetarget content recommendation server via a URL, etc. at the time when the mediastreaming content needs to be played for the user(s). This may be referred to asdynamic stitching in the pre‐recorded content or VOD (video on demand) content.In an implementation, the users may be divided into various cohorts and a datacontent may be same or similarly indexed for a cohort of users, i.e., the same orsimilar recommendations may be provided to all users in a single cohort, whereinthe cohort of users is defined by the users having same or similarcharacteristics / behavior / pattern. In another implementation, one or morerecommendations made in a video content may be personalized for each userbased on one or more user characteristics / behavior / pattern and the users mayalso not be divided into multiple cohorts.In the live streaming of a data, for example, in case of a live cricket match, one ormore small segments of a content get ingested into a platform and / or to anotherprocessing unit
[104] which processes the content. For example, a live cricketmatch is captured by a camera set up and fed to a processing unit
[104] via acommunication means for distribution to the user(s). The processing unit
[104] may be configured to insert a relevant content for the user(s) in the media contentof the live cricket match received from the camera set up via the communicationmeans. So, in case of the live content being streamed, content in near‐real timehas to be shown to the user(s).Thus, in an implementation, the customization of the video data content may notbe performed in a single system where encoding of content is followed by indexingof the content as explained above in the pre‐recorded content. Instead, in thisimplementation, the content is directly fed to the encoder for encoding thecontent to ingest the encoded content (in various resolutions) in the contentdelivery network (CDN)
[109] . And, separately the same content may be providedto a separate system for object and / or character identification and indexing ofmedia content for providing enhanced context aware media streaming basis thesame. This is done to overcome the issues related to delays which may be incurreddue to object and / or character recognition and indexing of the content. Now, forexample, the indexing in the separate system identifies some points of objectsand / or characters and indexes them to find relevant data to be fetched from thetarget content recommendation server (that is populated for fetching such data)to be displayed at the points of indexing, the information regarding the data to befetched from the target content recommendation server to be displayed to theuser at the time point as suggested by the separate system and played at the userdevice when needed, is saved in a form of a separate file, that is a virtual manifestfile or an updated / revised manifest file in the system that can be interpreted bythe processors to display the required content. The separate file may also containuniform resource locators (URLs) from the target content recommendation server.Based on this updated manifest, the CDN
[109] connected to the target contentrecommendation server fetches relevant content to be displayed to the user. Also,the target content recommendation server and the CDN
[109] may or may not beplaced together at the same location. In this way, the encoded stream that wasdirectly sent to the CDN
[109] and the one that is processed by the separatesystem to index and insert relevant content for the user can be combined and sentby the CDN to be played in synchronization at the user device. For the purpose tocombining this virtual manifest file with the original manifest file, the virtualmanifest file may be interacting with the original manifest file for displayingcontent recommendation to the user. For this purpose, for example, a periodicpolling mechanism may be adopted where the original manifest file polls thisupdated manifest file to be sent. Also, the content available at a specific URL mayalso be configurable by the system
[100] or the operators as needed.In an implementation, the content recommendation data or the targeted contentdata is stored in the form of web links, etc. and presented to the user via contentdelivery networks
[109] , or may be stored in a storage unit
[114] and displayedwhen required by the system
[100] .Further, in an implementation, when the user requests to play a content at theuser device, a server of an operator receives the user request. This server of theoperator may comprise a group of servers such as content management system(CMS), user management system (UMS), H‐play system, and Z‐play system. Thesesystems or servers may communicate with the content delivery networks (CDNs)
[109] to provide URL addresses of the content to be played. This server(s) of theoperator is also configured to authenticate the request of the user where it checkswhether the user is genuine or fake. Once the authentication is done, a short tokenis issued and delivered to the user device in case the user is found genuine. Theshort token comprises the URL. Further, the user device finds the URL in that shorttoken and reaches the CDN
[109] . Further, in an implementation, at the CDN
[109] ,a basic validation may occur. Pursuant to this validation, the CDN
[109] may issueand send a long token to the user device. The long token may comprise of amanifest file that contains a plurality of URLs of all resolutions of all segments ofthe content that needs to be played at the user device. Whenever there is therequirement of an updated manifest, the user device may connect again with theCDN
[109] for the updated manifest file.Additionally, the processing unit
[104] of the system
[100] is also configured toprovide, at least one of a searching capability, a buying capability, an object relateddata accessing capability, a character related data accessing capabilities on theone or more user devices based on the context aware media streaming content.For instance, in an implementation, the present disclosure also encompassesimproving the searching capabilities of platforms by means of the indexing ofscenes, objects and characters. For instance, a user may provide an image, say, ofa character, to search content and since the similar image is present in an index ofsay content X, the user may be directed to a video the content X as one of theoptions, and the system
[100] may also search for other data related to thatcharacter and presented to the user. Also, in another instance, the searchstring / image provided the user maybe mapped to the interests of the user, thepast browsing history and all the interactions of the users may be stored in adatabase or memory unit and from that data of the user, the output of the systemmay be optimized as per the user interests. Additionally, a user can form searchstrategies using keywords and Boolean or proximity operators or semantic search,and the search for these keywords or sentences is done in an indexed metadataso that an appropriate content list can be presented to the user.In an implementation, the invention further encompasses providing a capability todirectly purchase an indexed object / or items relating to one or more indexedscenes / characters in a video content. For example, when a character is wearing aparticular apparel, then that article or similar article can be identified that isavailable at an ecommerce platform, and the user may be given an option via thesystem
[100] to directly purchase said item. Also, here the indexing may help inidentifying scenes that are being displayed to viewer that contain an object suchas a tourist attraction, etc. The invention enables to identify one or more scenesor clips within a content in more detail with respect to context, storyline, genre,cast apparel / costumes, tourist attractions etc. for identifying targeted contentopportunities, character‐based content recommendation, customizedthumbnails, content previews with culled‐out character of interest for users,enhanced character or image based content search, etc. based on the same.In an example, the invention may provide an additional data tray on the userdevice to provide details related to cast / objects in the content based on thesame. This data tray may be accessed by a viewer on his / her device by clicking ordragging on a user interface or by any other means. Alternatively, in animplementation, the data tray may be available in a dedicated part of the userinterface. Also, once the user accesses the data tray on his / her device, the datatray may be available on whole or part of the user interface or screen of the userdevice. Thus, if the data tray is shown partially on the user interface, the user maybe able to view the video content on one part of the user interface and access thedata tray in parallel to the same. Further, this additional data tray may compriseURL links for the viewer to access the relevant data or the data of interest to theuser. The viewer may click to access the additional data tray to access one or moredesired links. For example, in a scene or clip in a video content being displayed onthe user device, an image of monument, say, statue of liberty is being displayed.Here, the system
[100] may identify a targeted content opportunity related to thestatue of liberty, say, a tour and travel package offering a visit to the statue ofliberty. Accordingly, the system
[100] encompasses displaying a targeted contentrelated to the tour and travel package offering a visit to the statue of liberty.Further, the invention also encompasses dynamically changing this targetedcontent or displaying this targeted content for a specific period of time. Therefore,the targeted content may change when a scene changes or after some time whenthe scene changes. For example, a scene or clip displaying the statue of liberty isover and a new scene has another place, say, the Taj Mahal. So, the targetedcontent shown to the user may also change from the "tour and travel packageoffering a visit to the statue of liberty" to the new targeted content related to theTaj Mahal, say "tour and travel package offering a visit to the Taj Mahal". Also, thetargeted content related to the Taj Mahal may be a different targeted contentrelated to Taj Mahal based on a user preference or behavior, and not a targetedcontent of tour and travel package offering a visit to the Taj Mahal. For example,an analysis of past user behavior shows or suggests that a viewer has no interestin travelling or touring but is interested in collecting miniatures or historicalmonuments. Thus, in that case, the viewer may be shown a targeted content forbuying a miniature artefact of Taj Mahal. Thus, in this way, the targeted contentmay be customized for each viewer using the present invention. In anotherexample, when a scene or clip displays Taj Mahal, the user may not be providedwith targeted content feed but a content recommendation in the form of ahyperlink to a web page to read more about Taj Mahal. Also, in thisimplementation of a data tray being visible to the user at the user device, the datatray may contain a combination of targeted content related to the Statue ofLiberty, the Taj Mahal as well as the non‐targeted content recommendationrelated to the Statue of Liberty and / or the Taj Mahal so as to enable user to selectwhich content to access from the available recommendations in the data tray.Also, extending this example, if a video data being streamed contains a video of avery less popular place, and the system identifies that the user may be interestedto know more about that place, then the hyperlink to read and know more aboutthat place may be included in the data tray for the user to access.Further, in an implementation, the content of the data tray may also be editableby the user. For example, if a user is shown content in the data tray to which theuser has more to share, the user can provide inputs on the same, which may beadded to the targeted content based on further validation at the operator's endin a manual, automated or semi‐automated manner.The invention also encompasses short‐form video generation at a character level.Based on the recognition of a particular character in multiple streaming videocontent, the system
[100] may generate a short video for each character based onvarious clips or scenes from one or multiple streaming video content.Figure 2 illustrates an exemplary method flow diagram
[200] for providing acontext aware media streaming content on one or more user devices, shown inaccordance with exemplary embodiments of the present disclosure. The methodis implemented via the system
[100] .The method for providing the context aware media streaming content on the oneor more user devices begins at step 202.The method at step 204 comprises receiving, at a character and object recognitionmodule
[102] , a set of frames of a media streaming content. The media streamingcontent is one of a live video content and a pre‐recorded video content. Thecharacter and object recognition module
[102] comprises at least one of a facedetector module
[106] , a face alignment module
[108] , and a face recognitionmodule
[110] . Also, in an implementation, the character and object recognitionmodule
[102] is a pre‐trained system that is trained on a training dataset, whereinthe training dataset comprises a data associated with at least one of a plurality ofimages of characters and a plurality of objects.Further at step 206, the method encompasses analyzing, by the character andobject recognition module
[102] , at least one selected frame from the set offrames of the media streaming content. The at least one selected frame from theset of frames is selected based on at least one of a logical selection and a randomselection. In an implementation, the present disclosure encompasses logicallyand / or randomly selecting one or more frames in a video content. For example,while logically selecting the one or more frames, the character and objectrecognition module
[102] may analyse a background and / or foreground of theframes, a change in camera angle, a shift of camera to a new scene and / or suchanalysis of one or more consecutive frames in the video content, that is obviousto a person skilled in the art through a quick screening of frames, and pick frame(s)when there is a drastic change in the background and / or foreground of the framesor a change is above a threshold level as specified in the system
[100] .Furthermore, at step 208, the method comprises identifying, by the character andobject recognition module
[102] , at least one of one or more characters and oneor more objects based on the analysis of the at least one selected frame. Asdepicted in figure 3 at step
[302] , the process of identifying, by the character andobject recognition module
[102] , the one or more characters begins at step
[302] .The process
[300] further at step
[304] comprises localizing, by the face detectormodule
[106] , one or more faces in the set of frames of the media streamingcontent.Further, the method
[300] at step
[306] comprises aligning, by the facial alignmentmodule
[108] , the one or more localized faces to normalized canonicalcoordinates,Furthermore, the method
[300] at step
[308] encompasses implementing, by theface recognition module
[110] , a facial recognition method on the one or morealigned localized faces, and next the method
[300] at step
[310] encompassesidentifying, by the face recognition module
[110] , the one or more charactersbased on the implementation of the facial recognition method on the one or morealigned localized faces. The method then terminates at step
[312] .Therefore, in an exemplary implementation, firstly, the face detector module
[106] receives the streaming video content and localizes face(s) in each frame ofthe streaming video content. The received streaming video content may beretrieved from the storage unit
[114] of the system or from an external server.Second, the facial alignment module
[108] aligns the localized / identified faces tonormalized canonical coordinates. Third, the face recognition module
[110] implements the facial recognition method on the aligned faces to identify one ormore characters in the streaming video content. The face recognition module
[110] may implement face matching steps based on threshold matching, neuralnetwork identification, metric learning, etc. In an implementation, the facerecognition module
[110] may include a feature extraction module, and a lossfunction calculator.The character and object detection module
[102] detects and recognizes the oneor more objects via the object recognition module
[112] . In an implementation,detecting and recognizing by the character and object detection module
[102] theone or more objects such as including but not limiting to one or more cars thatsome characters are driving, or one or more places that are shown in one or morescenes or clips in a video content, or certain other object(s) in a frame of a content,for example phones, computers, cars, food, refrigerator, water bottle, etc. theseare identified and / or detected and / or recognized based on various techniques.For example, a comparison of the one or more objects to a dataset as definedabove may help in identifying the one or more objects and tagging. Also, the aboveexamples of objects and / or characters are provided for understanding purposesonly and do not restrict or limit the present disclosure in any possible manner.Further, the character and object recognition module
[102] implements varioustechniques for object detection purposes, such as Region‐Based ConvolutionalNeural Networks (R‐CNN) and Spatial Pyramid Pooling (SPP‐net) models. Also, acombination of known techniques may also be developed for implementation ofthe object detection purpose. So, a combination of 'image detection' techniquewith 'object tracking' technique may be implemented for a fast and accurateobject detection in a streaming video. Further, the present disclosureencompasses that the character and object recognition is done by the characterand object recognition module
[102] in near real‐time for live streaming videocontent and may be pre‐performed for video on demand content.Thereafter at step 210, the method comprises indexing, by a processing unit
[104] ,at least one frame from the at least one selected frame that encompasses at leastone of the one or more characters and the one or more objects. Further, theindexing is performed against a playtime length on a progress bar of the mediastreaming content, and the indexing is displayed on the progress bar by at leastone of one or more markings, one or more colour codes, one or more characternames, one or more emojis, and one or more actual frames in the media streamingcontent. Furthermore, the indexing, by the processing unit
[104] , the at least oneframe from the at least one selected frame comprises inserting in the at least oneframe from the at least one selected frame a set of content to be provided on theone or more user devices. In an implementation, the indexing of videos may alsoleverage an insertion of a targeted content such as promotion campaigns,advertisements, etc. in the video content of the one or more videos. In general,the targeted content is considered intrusive to users and the users are notinterested in watching that targeted content, or click on the targeted content tovisit web pages linked with that targeted content to utilise the same. The solutionto this problem of showing intrusive targeted content to the users, that thepresent invention encompasses, is by providing an in‐feed targeted content thatis not intrusive or are less intrusive. Also, in an implementation, the users may begiven a choice for viewing the in‐feed targeted content where the users can selectwhether the targeted content should be shown to them or not. For this purposeof indexing the at least one frame from the at least one selected frame, in animplementation, a target content recommendation server may be connected tothe processing unit
[104] . This target content recommendation server comprisesall contents that can be recommended to a user during streaming of a mediastreaming content on a user device of the user. Whenever a contentrecommendation is detected by the system
[100] that is of relevance with respectto a user, it is directed to the target content recommendation server to search forany recommended content related to an identified object or an identifiedcharacter and / or customized to a user interest or a user preference, that can bedisplayed on a screen or a user interface of a user device of the user at a particularinstance of a media streaming content being displayed.Next at step 212, the method comprising generating, by the processing unit
[104] ,one or more customized artworks based on the indexing of the at least one framefrom the at least one selected frame and a set of user attributes. The one or morecustomized artworks comprise at least one of one or more previews, one or morethumbnails, one or more teasers, one or more trailers, one or more images, andone or more short videos related to at least one of the one or more characters andthe one or more objects.In an implementation the present disclosure the processing unit
[104] may referto a table of contents as well as one or more user attributes that are alreadylearned by the system
[100] (by implementing a machine learning model, etc.) anddecide whether any item in the table, that is, any object or character is ofrelevance to the user attribute(s). For example, if a user is interested in an actionmovie, then the user in this case might be interested in a powerful antagonist, theuser in another case might be interested in a stimulating fight between aprotagonist and an antagonist, etc. The system
[100] after referring to both atimeline or the indexed content and the user attribute(s) generates the one ormore customized artworks and provides such customized artwork(s) to the user.Also, in another implementation, a standard artwork is taken by the system
[100] from other digital platforms, and checked whether the standard artwork iscustomizable for a user or not. In case the standard artwork is customizable, thestandard artwork is customised by the system
[100] , and the customised artworkis shown to the user. Alternatively, if the standard artwork is not customizableaccording to the user, then the standard artwork is shown to the user.Further at step 214, the method encompasses generating, by the processing unit
[104] , the context aware media streaming content based at least on the one ormore customized artworks. Further, for generating, by the processing unit
[104] ,the context aware media streaming content the method further comprisesmodifying the media streaming content based on one of a manifest file associatedwith the one or more customized artworks and a hardcoding of the one or morecustomized artworks. Next, at step 216, the method comprising providing, by theprocessing unit
[104] , the context aware media streaming content on the one ormore user devices.For instance, in an implementation, to provide a context aware media streamingcontent on a user device, a media streaming content modified based on acustomised artwork or a link for the customised artwork may be sent to the userdevice using a manifest file. In an embodiment, this customised artwork isgenerated prior to encoding the video content for delivery to the user device. Forinstance, if the user has requested for an on‐demand video like a movie that hasalready been indexed and the system
[100] also knows the user's preferences, insuch a scenario, the customised artwork is generated prior to encoding the videofor transmission to the user's device. In another embodiment, the customisedartwork is generated after the encoding of the video content for transmission tothe user device. For instance, if the user has requested for a live content, theindexing for such content is done in real time, so customised artwork may not begenerated prior to encoding. In another instance, if a new user has requested foran on‐demand video, the user preferences are not known to the system
[100] , insuch a scenario also a customised artwork may not be generated prior to encodinga video for transmission but only after the user has started watching the video andhas provided some inputs to the system
[100] that assist in deriving such user'spreferences. In another embodiment, the system
[100] may generate customizedartwork for a user basis the basic demographics of the user and a pool of similarusers (that is, by considering the preferences of other users with same or similardemographics) or any such correlation between the user and other users for whichthe system already has the data available.The method further comprising generating, by the processing unit
[104] , acustomized targeted content based on the context aware media streamingcontent, and providing, by the processing unit
[104] on the one or more userdevices, the customized targeted content based on a historical user patternassociated with the one or more user devices.Further, the method encompassing providing by the processing unit
[104] , at leastone of a searching capability, a buying capability, an object related data accessingcapability, a character related data accessing capabilities on the one or more userdevices based on the context aware media streaming content.The method then terminates at step 218.Various embodiments disclosed herein provide numerous advantages. Morespecifically, the embodiments disclosed herein suggest techniques for providingenhanced context aware media streaming to user(s) wherein content beingstreamed on user devices is indexed to provide for enhanced content search,content recommendation, target content placement, content artworkcustomization, customized targeted content, etc. Since the present system iscontext aware due to the indexing of characters and objects in the media stream,the system generates better recommendations for the users based on indexedcontent and user preferences. Also, providing a limited and relevant content tothe user, requires less data to be ingested on the content delivery networks. Thus,the same leads to saving costs of ingesting more data in the content deliverynetworks and results in better user experience as the user gets to see only therelevant content that the user is interested in. Further, since indexing is done foreach frame of the video content and user behavior / attributes are already learnedby the system, the video content can be linked with user behavior andrecommended content can be shown to the user based on user behavior, forexample, user may like to watch short videos, or may want dubbed content inplace of original content in some particular language, or may want to watchcontent of a particular genre or of a particular artist, etc. Based on this, customizedartwork may also be created for the user which further enhances the userexperience of searching the content.Furthermore, insertion of targeted content related to the object or characterbeing displayed in the streamed video content, and recommended content for theuser or the targeted content being non‐intrusive to the user also improves the userexperience. Considering the increased targeted content tolerance level of the userdue to the above‐mentioned features, and the characteristics of the content fordetermination of number of targeted contents to be inserted in the media stream,ensures an optimal number of targeted contents that may be inserted for differenttypes of users. Further, training of the system using the existing data and the newdata generated by the system also improves characteristics of the system such asthe balance between latency and effectiveness, accuracy of the characterrecognition.Further, in an implementation, 'context‐aware ad opportunities' in a content maybe identified by recognition and indexing system and the content may be encodedalong with the spots / markings for the identified opportunities. Further, thecustomized art, graphics, snippets, frames, shots for the respective items in thecontent where the relevant content, that is, targeted content opportunities areidentified in the content are generated against the relevant content or targetedcontent campaigns available in the libraries, and populated in the CDN server.Further, the system may be trained to identify the targeted content displayopportunities. For example, in a frame, there is a television and a logo on a Productis clearly visible (say the Product belongs to Brand ABC). The system may identifythis as a targeted content opportunity and search for the scene boundary relatedto this scene in the frame. Say, the frame is at XX:30 min. and the scene boundarylies at XX:20 - XX:40 min. In that case, the system may provide to the targetcontent recommendation server the complete segment of XX:20 - XX:40 min. Inthis case, the target content recommendation server, for example, may search forother products of the Brand ABC to be presented to the user. Continuing with theabove example, say the Product is a Television of Brand ABC. And, the targetcontent recommendation server finds a targeted content for a television of BrandDEF, then this television of Brand ABC may be replaced by the target contentrecommendation server with a television of Brand DEF, or only the logo of BrandABC on the television may be replaced with the logo of Brand DEF to be displayedto the user. Further, during the playback at the user device, at the marked spotsfor content relevant content display or targeted content opportunity, the manifestfile may inform the user device to communicate with the CDN for displayingrelevant content or targeted content.Further, in an implementation, recognition of objects and characters and indexingin run time or near real‐time for live‐streaming video content, a local databasemay be maintained that crawls the internet for characters, other items data suchas the items available on the ecommerce websites or other websites such as aneducational website, and the system is trained to compare the characters / objectsin a content with that local database in real‐time. This may also help in handlingissues related to latency and synchronization of content during live streaming ofdata. Consequently, the relevant data may be provided to the user, for example,in a data tray where a user may click on the link to access the item that he desires.Also, the present invention provides an enhanced search option to the user(s)since the system is context aware and understands the context of the contentoffered by the platform. The user is able to perform a more enhanced image ortext based search using the system as the system not only searches in the title anddescription of the media content but also in the indexed content, i.e., thecharacters and objects in the media content.While the invention has been explained with respect to many examples, it will beappreciated by those skilled in the art that the invention is not restricted by theseexamples and many changes can be made to the embodiments disclosed hereinwithout departing from the principles and scope of the present invention.
Claims
1. A method for providing a context aware media streaming content on one or more user devices, the method comprising: ‐ receiving, at a character and object recognition module [102], a set of frames of a media streaming content; ‐ analyzing, by the character and object recognition module [102], at least one selected frame from the set of frames of the media streaming content; ‐ identifying, by the character and object recognition module [102], at least one of one or more characters and one or more objects based on the analysis of the at least one selected frame; ‐ indexing, by a processing unit [104], at least one frame from the at least one selected frame that encompasses at least one of the one or more characters and the one or more objects; ‐ generating, by the processing unit [104], one or more customized artworks based on the indexing of the at least one frame from the at least one selected frame and a set of user attributes; ‐ generating, by the processing unit [104], the context aware media streaming content based at least on the one or more customized artworks; and ‐ providing, by the processing unit [104], the context aware media streaming content on the one or more user devices.
2. The method as claimed in claim 1, wherein the at least one selected frame from the set of frames is selected based on at least one of a logical selection and a random selection.
3. The method as claimed in claim 1, wherein the media streaming content is one of a live video content and a pre‐recorded video content.
4. The method as claimed in claim 1, wherein the character and object recognition module [102] comprises at least one of a face detector module [106], a face alignment module [108], a face recognition module [110] and an object recognition module [112].
5. The method as claimed in claim 4, wherein the identifying, by the character and object recognition module [102], the one or more characters further comprises: ‐ localizing, by the face detector module [106], one or more faces in the set of frames of the media streaming content, ‐ aligning, by the facial alignment module [108], the one or more localized faces to normalized canonical coordinates, ‐ implementing, by the face recognition module [110], a facial recognition method on the one or more aligned localized faces, and ‐ identifying, by the face recognition module [110], the one or more characters based on the implementation of the facial recognition method on the one or more aligned localized faces.
6. The method as claimed in claim 1, wherein the character and object recognition module [102] is a pre‐trained system that is trained on a training dataset, wherein the training dataset comprises a data associated with at least one of a plurality of images of characters and a plurality of objects.
7. The method as claimed in claim 1, wherein the indexing is performed against a playtime length on a progress bar of the media streaming content, and the indexing is displayed on the progress bar by at least one of one or more markings, one or more colour codes, one or more character names, one or more emojis, and one or more actual frames in the media streaming content.
8. The method as claimed in claim 1, wherein the indexing, by the processing unit [104], the at least one frame from the at least one selected frame comprises inserting in the at least one frame from the at least one selected frame a set of content to be provided on the one or more user devices.
9. The method as claimed in claim 1, wherein the one or more customized artworks comprise at least one of one or more previews, one or more thumbnails, one or more teasers, one or more trailers, one or more images, and one or more short videos, related to at least one of the one or more characters and the one or more objects.
10. The method as claimed in claim 1, wherein the generating, by the processing unit [104], the context aware media streaming content further comprises modifying the media streaming content based on one of a manifest file associated with the one or more customized artworks and a hardcoding of the one or more customized artworks.
11. The method as claimed in claim 1, the method further comprising: ‐ generating, by the processing unit [104], a customized targeted content based on the context aware media streaming content, and ‐ providing, by the processing unit [104] on the one or more user devices, the customized targeted content based on a historical user pattern associated with the one or more user devices.
12. The method as claimed in claim 1, the method further comprises providing by the processing unit [104], at least one of a searching capability, a buying capability, an object related data accessing capability, a character related data accessing capabilities, on the one or more user devices, based on the context aware media streaming content.
13. A system [100] for providing a context aware media streaming content on one or more user devices, the system [100] comprising: - a character and object recognition module [102], configured to: ‐ receive, a set of frames of a media streaming content, ‐ analyze, at least one selected frame from the set of frames of the media streaming content, and ‐ identify, at least one of one or more characters and one or more objects based on the analysis of the at least one selected frame; and - a processing unit [104], configured to: ‐ index, at least one frame from the at least one selected frame that encompasses at least one of the one or more characters and the one or more objects, ‐ generate, one or more customized artworks based on the indexing of the at least one frame from the at least one selected frame and a set of user attributes, ‐ generate, the context aware media streaming content based at least on the one or more customized artworks, and ‐ provide, the context aware media streaming content on the one or more user devices.
14. The system [100] as claimed in claim 13, wherein the at least one selected frame from the set of frames is selected based on at least one of a logical selection and a random selection.
15. The system [100] as claimed in claim 13, wherein the media streaming content is one of a live video content and a pre‐recorded video content.
16. The system [100] as claimed in claim 13, wherein the character and object recognition module [102] comprises at least one of a face detector module [106], a face alignment module [108], a face recognition module [110] and an object recognition module [112].
17. The system [100] as claimed in claim 16, wherein to identify, the one or more characters: ‐ the face detector module [106], is configured to localize one or more faces in the set of frames of the media streaming content, ‐ the facial alignment module [108], is configured to align the one or more localized faces to normalized canonical coordinates, ‐ the face recognition module [110], is configured to implement a facial recognition method on the one or more aligned localized faces, and ‐ the face recognition module [110], is configured to identify the one or more characters based on the implementation of the facial recognition method on the one or more aligned localized faces.
18. The system [100] as claimed in claim 13, wherein the character and object recognition module [102] is a pre‐trained system that is trained on a training dataset, wherein the training dataset comprises a data associated with at least one of a plurality of images of characters and a plurality of objects.
19. The system [100] as claimed in claim 13, wherein the indexing is performed against a playtime length on a progress bar of the media streaming content, and the indexing is displayed on the progress bar by at least one of one or more markings, one or more colour codes, one or more character names, one or more emojis, and one or more actual frames in the media streaming content.
20. The system [100] as claimed in claim 13, wherein to index the at least one frame from the at least one selected frame the processing unit [104] is configured to insert in the at least one frame from the at least one selected frame a set of content to be provided on the one or more user devices.
21. The system [100] as claimed in claim 13, wherein the one or more customized artworks comprise at least one of one or more previews, one or more thumbnails, one or more teasers, one or more trailers, one or more images, and one or more short videos, related to at least one of the one or more characters and the one or more objects.
22. The system [100] as claimed in claim 13, wherein to generate the context aware media streaming content the processing unit [104] is further configured to modify the media streaming content based on one of a manifest file associated with the one or more customized artworks and a hardcode of the one or more customized artworks.
23. The system [100] as claimed in claim 13, wherein the processing unit [104] is further configured to: ‐ generate a customized targeted content based on the context aware media streaming content, and ‐ provide on the one or more user devices, the customized targeted content based on a historical user pattern associated with the one or more user devices.
24. The system [100] as claimed in claim 13, wherein the processing unit [104] is further configured to provide, at least one of a searching capability, a buying capability, an object related data accessing capability, a character related data accessing capabilities on the one or more user devices based on the context aware media streaming content.
