Method and system for enriching audiovisual content

Abstract

The invention relates to a method for enriching initial video content, the method comprising: - a step of playing said initial video content, which step is designed to recognise predetermined elements therein; - a step of classifying the recognised elements; - a step of searching for and selecting digital resources and / or services associated with the recognised and classified elements; - a step of creating one or more capsules associated with each of the recognised and classified elements intended to contain the selected digital resources and / or services; - a step of integrating the one or more created capsules into the initial video content so as to deliver enriched video content.

Description

Method and system for enriching audiovisual content Field of invention

[0001] The invention relates to a method for enriching audiovisual content. It also relates to a system implementing this method, in particular a digital platform.

[0002] The field of invention includes digital communications, digital marketing, video and e-commerce. State of the art

[0003] We are currently seeing a strong growth in audiovisual content offerings. A growing number of these audiovisual contents are in fact associated with brands and commercial offers.

[0004] To date, if a person viewing one of these audiovisual contents wishes to purchase a product or service seen in this content, they must then be redirected, via a digital communication network, to a sales site on the Web and use a payment tool. Examples include the "Click and Buy" tools. » or « GoogleAds ".

[0005] Current methods therefore have the disadvantage of not allowing users to access product or service purchasing services in real time.

[0006] Furthermore, video content creators can hardly include in their video content technical means to provide people who view their video creations with simple and direct access to purchases or access to services.

[0007] A main aim of the invention is to remedy this drawback by proposing an innovative method for enriching video content, which is powerful and simple to use.

[0008] This objective is achieved with a method for enriching initial video content, comprising the following steps: a reading of this initial video content, designed to recognize predetermined elements therein, a classification of said elements thus recognized, a search and selection of digital resources and / or services associated with said recognized and classified elements, a creation of one or more capsules associated with each recognized and classified element, intended to contain the selected digital resources and / or services, an integration of the capsule(s) thus created into the initial video content, so as to deliver enriched video content.

[0009] The recognition and classification steps can advantageously implement artificial intelligence techniques.

[0010] The initial video content integration step can be arranged to graphically represent a capsule when viewing the enriched video content.

[0011] The enrichment method according to the invention may further comprise a step for creating in the enriched video content touch selection zones associated with one or more selected elements with which one or more capsules have been associated, so that a touch action on one of these selection zones causes the display of a capsule associated with it.

[0012] The step of creating a capsule can be arranged to include in this capsule an additional video and / or an online store and / or a consultable document such as a press article. Description of figures

[0013] The invention will be better understood in light of the following figures:

[0014] This is a block diagram of an exemplary embodiment of the video content enrichment method according to the invention;

[0015] This is an example of a functional diagram of a prediction and identification algorithm implemented in the video content enrichment method according to the invention;

[0016] Illustrates the first four steps of an example of implementation of the enrichment method according to the invention;

[0017] Illustrates three following steps of the example of realization of the. Definitions

[0018] iFrame: name given to an HTML tag used in computer language to integrate the content of another HTML page into an HTML page.

[0019] API: for ApplicationProgramingInterface: An application programming interface comprising a set of definitions and protocols that facilitates the creation and integration of application software.

[0020] Bounding Box: A rectangle aligned with the axis. This is the simplest type of closed shape in planar form, represented by two points containing the minimum and maximum coordinates of each axis.

[0021] PyTorch: An open-source machine learning framework that accelerates the transition from research prototyping to production deployment.

[0022] SAAS: Software as a Service or "software as a service": an application software solution hosted in the cloud and operated outside the organization or company by a third party, also called a service provider. The SaaS solution is accessible on demand via an Internet connection.

[0023] Confidence Score: Classification confidence scores are designed to measure the model's accuracy when predicting class assignment.

[0024] YOLOv5: A family of compound-scaled object detection models trained on the COCO dataset, including simple features for test time augmentation (TTA), model assembly, hyperparameter evolution, and export to ONNX, CoreML, and TFLite. Detailed description of an example of implementation

[0025] With reference to the, the enrichment method according to the invention can be implemented on a service platform 30 intended for video content creators and available in the form of a SAAS application. On this platform 30, a video content creator can provide as input an initial video creation 31. A video player 32 is provided with an identification / authentication device 33 and connected to a database 34.This video player implements a set of artificial intelligence algorithms 35 which will carry out the following operations: a series 36 of recognition of elements in the initial video content 31, such as products and / or services, people, places, objects, plants or animals, dates and many other elements, implementing known artificial intelligence (AI) techniques, a classification 37 of these recognized elements using known learning techniques and by referring to predetermined categories of capsules, a creation 38 of capsules associated with each element recognized and selected by the user of the platform method 30, an insertion 39 of these capsules thus created in the initial video stream, so as to deliver enriched video content 42 from the initial video content 31 now integrating 43,44 several capsules C1,…,Cn.

[0026] The contents of the capsules thus created are retrieved by accessing, for example in a Cloud (computing cloud) or via digital networks or on the Web, PM1,…PMn marketplaces as well as content platforms, in particular video content, PC1,…PCk.

[0027] We will now describe, with reference to, an example of an artificial intelligence algorithm implemented for the detection of elements (people) in video content.

[0028] This algorithm 100 was carried out with the PyTorch coding tool commonly used in machine learning. In a first step 101, a YOLOv5 object detection model, of the “compound-scaled” type, is implemented on COCO data. From this model, cropped person images are identified using 102 bounding boxes (“Bounding box or B.Box”). The next step 103 is a bounding box prediction from fashion data (“Fashion data”) 107.

[0029] The prediction is then subjected 104 to a confidence score test (“Confidence Score”), a technique well known in the field of machine learning. If the confidence score is less than a predetermined value X, then (108) this bounding box B.Box is ignored. If, on the contrary, the confidence score is greater than X, then we move on to the next step 105 of identifying the class confidence and then to the step 109 of detecting an image containing the desired object.

[0030] We will now describe, with reference to figures 3 and 4, an example of use of enriched video content by a person viewing or consulting this content on electronic communication equipment 1, such as a smartphone, a tablet or a personal computer.

[0031] In a first step I, the person consults the video content and locates an element there, for example a known actor 2. He then selects, in a second step II, by a tactile action with a finger of his hand 3 on the screen 1, this element 2.

[0032] In a third step III, this selection causes the display of a capsule icon 4 in a corner of screen 1, for example the upper right corner.

[0033] The person then selects (step IV) this capsule icon 4, which causes the appearance on screen 1 of a window 5 integrated into the video, this window 5 comprising three zones 50, 51, 52 corresponding respectively to an additional video, to an online store and to a press article, these three resources all being related to the selected element 2.

[0034] Thus, in the case where the user selects (step V) zone 50, a window 20 for viewing an additional video then appears and the person can directly view this additional video within the enriched video content.

[0035] If the user selects (step VI) zone 51, then an online store site 21 appears, integrated within the enriched video content.

[0036] If the user selects (step VII) window 52, ​​a window 22 allowing the reading of a press article appears, integrated within the enriched video content.

[0037] The process of producing enriched audiovisual content can be implemented on a video creation platform. This process allows video content creators to offer their viewers the opportunity to interact within their creation.

[0038] The video player can be available as a SaaS application available to both tech novices and video professionals. It allows you to embed brands that a creator wants to see integrated into your audiovisual content.

[0039] The video player implemented in the method according to the invention can have the following characteristics:product / service recognition: artificial intelligence can recognize products and services integrated in the video,person recognition: artificial intelligence can recognize all people in the video content,location recognition: artificial intelligence can recognize most of the similar locations in the video and give a precise address in the form of a geolocation (GPS),object, plant, animal recognition: artificial intelligence can recognize most of the types of objects, plants, animals in the video,date recognition: artificial intelligence can recognize a specific date (most,analysis of the clothes worn by the protagonists, types of objects etc....Food and drink recognition: Artificial intelligence can recognize a dish and its composition and extract the products that make up the dish. Classification of elements and capsules

[0040] The enrichment process is designed to classify the clips created after the recognition process. Artificial intelligence will classify all recognized elements into a clip embedded in the top right corner of a video.

[0041] It will then deduce three offers on the item:an additional video: this video is extracted from the YouTube catalog or other videos from a public API, the closest to the product to be sold integrated into the capsule. This additional content can be a report, a promotional video, a tutorial video, an offer to purchase a product or service: this product or service can be sourced from millions of products available on international partner marketplaces.an article: a descriptive article of the product is automatically generated by artificial intelligence from information available on information sites.

[0042] The texts describing these offers can be translated into several languages ​​simultaneously. Database

[0043] For product recognition, artificial intelligence selects products available on e-commerce platforms around the world. If the products are not recognized on one of the platforms, the artificial intelligence then sends a "product to be created" message to the product creation service, which will create the product in question: dimensions, colors, materials, etc.

[0044] Artificial intelligence is expected to perform the following operations: "cut" the product in the video content to extract a photo and put it on a white background, extract a rotating 3D video on a white background, assign it a sales price after carrying out an international benchmark of similar products, automatically launch the product's production machine in a production plant, reference the available stocks in the video content, publish the product in the online stores of the videos and make it available to the public.

[0045] The user of the communication device can touch the screen at any time. This action causes a capsule to appear on the screen. The user must first validate the conditions of use of the product prediction algorithm.

[0046] The video player can be equipped with a biometric fingerprint reader, and offer the customer the option of recording their data: banking information, tastes, height, weight, measurements, identity, address, position, etc.

[0047] As soon as the customer validates the item on their screen with their finger, the ordered product is sent directly to their home. Examples of capsules

[0048] There are many types of capsules that can be created. Below is a non-exhaustive list of capsules that can be created following the recognition process in video content.

[0049] 1. PEOPLE

[0050] 2. PLACE

[0051] 3. EVENT

[0052] 4. OBJECTS,

[0053] 5. PLANTS,

[0054] 6. ANIMALS

[0055] 7. COSMETICS

[0056] 8. GAMES

[0057] 9. SPIRITUALITY & RELIGIONS

[0058] 10. PERSONAL DEVELOPMENT

[0059] 11. DATE

[0060] 12. FOOD AND BEVERAGES

[0061] 13. CLOTHING AND ACCESSORIES

[0062] 14. CARS AND ACCESSORIES

[0063] 15. FURNITURE

[0064] 16. COMPETITION

[0065] 17. REAL ESTATE

[0066] 18. BEAUTY

[0067] 19. TRAVEL

[0068] 20. VOTE

[0069] 21. MUSIC

[0070] 22. CARTOON

[0071] 23. CHILDREN

[0072] 24. HIGH TECHNOLOGY

[0073] 25. BOOKS

[0074] 26. BANKING

[0075] 27. SCIENCES

[0076] 28. EDUCATION

[0077] 29. TRAINING

[0078] 30. METHODS

[0079] 31. GOVERNMENT

[0080] As a non-limiting example, we can consider 200 capsules for a 3-minute video, 600 capsules for a 6-minute video and more than 1000 capsules for a video of more than 6 minutes.

[0081] The user of the enrichment method according to the invention integrates video content in the form of a file or a link into the Web platform or into the SAAS application.

[0082] The artificial intelligence then processes the video content for a duration less than or equal to that of the video content, depending on the availability of the queue for this process, referencing all the information in a dashboard assigned to the user.

[0083] The platform for implementing the enrichment method according to the invention can be designed to group together a plurality of services made available to creators and designated under the term “Transmedia Universe”:

[0084] 1. Video consultation

[0085] 2. Account creation

[0086] 3. Upload a video

[0087] 4. Transfer of social network account

[0088] 5. Call (like WhatsApp )

[0089] 6. Chat (like Facebook )

[0090] 7. Buy products and services (like Amazon )

[0091] 8. Monetization of video content (like Youtube )

[0092] 9. Conference calling (like Zoom )

[0093] 10. PAY (like Apple Pay )

[0094] 11. BANK (like N26 )

[0095] 12. REAL ESTATE (like a real estate agency)

[0096] 13. PRODUCTION (like Netflix )

[0097] 14. ENGINE (like Google )

[0098] 15. ACADEMIA (like the school that gives courses)

[0099] 16. VOTES (like SMS+)

[0100] 17. EVENTS (such as Live Nation )

[0101] 18. PRODUCTS (such as Nike )

[0102] 19. GAMES (like Ubisoft )

[0103] 20. And many other features and services that will be added later....Video viewing: Each user can view any video on the platform (music, film, documentary, etc.).Account creation: the user will have an account that will allow him to manage all his operations as on each social network (facebook instagram, twitter).Upload: the user will be able to upload his own videos and photos.Transfer of social network accounts: the enrichment process allows users of any social network to register by creating an account. It also allows users to simply migrate their account, that is, to register with their facebook account or Instagram or other directly. Call (like WhatsApp ): Audio and video calls are possible on the process as on WhatsApp.Chat (like Facebook ): the process has a messenger that allows you to chat with your contacts like Facebook .Purchase of products and services (Amazon type ): The act of purchasing is possible on the process by simply clicking on the selected product on the screen. Monetization of video content (like Youtube ): Every content owner has the opportunity to earn money by publishing videos as soon as people buy their products. All information about their sales is directly recorded in their account in a detailed manner. Conference calling (like Zoom )PAY (like Apple Pay ): the user can pay for your purchases with this payment application.BANK (like N26 ): The User can open a bank account and have a bank statement like any online bank.REAL ESTATE (like a real estate agency): A specialized service allows users to sell or buy their property like in a real estate agency.PRODUCTION (like Netflix ): the process allows you to produce your own videos, series and films or any other program intended to be broadcast directly on the platform.ENGINE (like Google ): the process develops its own search engine and allows certain options that others do not offer. For example, the "Search Bar" option allows you to copy the link of a video to extract all the products and offer them for purchase, subject to obtaining authorization from the owner of the video content to access this content.ACADEMIA (like school learning classes): This option can be developed for all online courses. In this period of COVID-19, teaching is done remotely and allows the student / teacher relationship to be maintained.VOTES (like SMS+): Votes allow the free exchange of messages within a community and to vote for competitions, television shows, etc.EVENTS (like Live Nation ): Live and streaming concerts are possible on the Platform. PRODUCTS (such as Nike ): Creation and manufacturing of different products from various sectors. This allows the platform to significantly increase its margins.GAMES (like Ubisoft ): Online games for people playing on their smartphones and computers. And many other features and services that may be added later.

[0104] Through partnerships with industries across e-commerce, fashion, real estate, and more, it's possible to offer customers the promise of discovering a wide range of products. Video creators now have the opportunity to create short, engaging videos about featured brands.

[0105] The use of artificial intelligence contributes to revenue optimization and increased visibility for brands and creators.

[0106] If video recognition is successful, a pre-set number of capsules will appear in the capsule bar on the right side of your screen. This capsule bar remains fixed for the duration of the viewing. You can touch the screen at any time. This action opens the Capsule functions.

[0107] If video recognition is not enabled, capsules do not automatically appear on the screen. The capsule bar is therefore disabled. However, it is possible to touch the screen at any time. This action brings up the Capsule bar, which becomes active on the screen. It is possible to interact with the Capsule, which animates three options.

[0108] The artificial intelligence implemented in the enrichment method according to the invention is designed to: recognize all the elements that make up the video. Classify them by capsule under categorical references. determine whether the capsules are generated by a prospect, a customer or an advertiser.

[0109] Two bars can be displayed below the video: a “Similar Products” bar: the products / services have been recognized by the video, but are not identified for a Client or Advertiser. a “Specific Products” bar: the products / services have been recognized by the video and are identified as those of the Client or Advertiser. Search bar

[0110] A search bar is also offered to the user of the enrichment method according to the invention to enable him to optimize the relevance of the image or video content which will be associated with the product or service for which one wishes to encourage purchasing acts or user commitments.

[0111] An analytics API is planned to provide a search function: this is to improve the visibility of queries and determine keywords for displaying relevant video or images from the Internet.

[0112] It also involves defining query rules for problematic queries or adjusting search attributes / parameters to address relevant and systemic issues.

[0113] Filters are also provided to establish optimized conversion paths by pre-setting filters on specific keywords based on the most popular filters for searching videos or images.

[0114] An API of an online store, video, website (photo, video, text) can be registered in the algorithm.

[0115] A link is made between the registered element of the API and the duplicated element in the algorithm.

[0116] The element duplicated by Artificial Intelligence is searched in its database.

[0117] When an e-commerce platform makes available products or services whose iframes cannot be directly integrated within video content, it is possible to integrate the products or services with this type of iframes into a dedicated store to allow full playback of the iframes and not exit the video (otherwise it can be considered as click and buy).

[0118] It should be noted that the data integrated, therefore duplicated in the algorithm (for example + 500 million products - images, texts, videos) can require enormous servers and enormous calculation times (recognition, linking, recording). We can then provide an algorithm that allows to link the publication date of the video with the market release date of a product. This has the effect of removing the quantities of research by the algorithm on the products hosted by eCommerce platforms.

[0119] Of course, the invention is not limited to the examples which have just been described and many other embodiments can be envisaged without departing from the scope of this invention. In particular, the number of recognizable elements and capsules which can be integrated into enriched video content are in fact only limited by the capacity and power of the computer servers implemented.

Claims

A method for enriching initial video content, comprising the following steps: reading this initial video content, designed to recognize predetermined elements, classifying said recognized elements, searching for and selecting digital resources and / or services associated with said recognized and classified elements, creating one or more capsules associated with each recognized and classified element, designed to contain the selected digital resources and / or services, and integrating the capsule(s) thus created into the initial video content, so as to deliver enriched video content. A method according to the preceding claim, characterized in that the recognition and classification steps implement artificial intelligence techniques. A method according to the preceding claim, characterized in that the integration step into the initial video content is arranged to graphically represent a capsule when viewing the enriched video content. Method according to the preceding claim, characterized in that it further comprises a step for creating in the enriched video content touch selection areas associated with one or more selected elements to which one or more capsules have been associated, such that a touch action on one of these selection areas causes the viewing of a capsule associated with it. A method according to any one of the preceding claims, characterized in that the step of creating a capsule is arranged to include in that capsule an additional video. A method according to any one of the preceding claims, characterized in that the step of creating a capsule is arranged to include an online store in that capsule. A method according to any one of the preceding claims, characterized in that the step of creating a capsule is arranged to include in that capsule a searchable document.

Images