A system and method for placing, responding to, and monitoring video media embedded advertising

Abstract

The present invention provides a system and method for placing, responding to, and monitoring video media embedded advertising. The system includes: an account system, a placing and responding system, a monitoring system, a settlement system, and a marking system; the method includes: advertisers and creators register accounts through the account system, place advertising targets, the monitoring system learns the visual image of the advertised product, the creator responds to the target on demand, the monitoring system identifies whether there is a visual image of the specified advertising target in the key frame of the video, the marking system marks the position of the advertising target to generate a floating hyperlink, and the settlement system calculates and pays the advertising fee based on the exposure time, number of viewers, exposure impressions, number of clicks, etc. The present invention can efficiently solve the problems of placing, responding to, and monitoring embedded advertising, and realize high-efficiency and large-scale precision marketing.

Description

Technical Field

[0001] The present invention relates to the field of advertisement placement and monitoring, and in particular to a system and method for placing, responding to and monitoring video media embedded advertisements. Background Art

[0002] With the rise of mobile Internet, in addition to traditional video media such as television and movies, live broadcast and short video applications have also become very popular. These video media make full use of users' fragmented time and have also become a new front for advertising. At present, the main forms of advertising based on video media are: patches or inserted ads before, during and after the video, floating ads on the video playback interface, and hard ads or soft ads mixed in the video list for the purpose of promoting advertising products. At the same time, based on the richness of video content, embedded advertising has also begun to rise. The main feature of embedded advertising is that it is not based on advertising promotion, but because the video content itself has the need to expose such products, the only difference is which brand is exposed; or although the video itself does not have such a hard demand, due to the richness of the picture, it is enough to accommodate such redundant information, and such information may attract the attention of users, thereby generating incidental advertising effects.

[0003] A large number of technical solutions for embedding advertisements in videos have been disclosed. For example, patent No. 201810147228.1 disclosed by Huawei Technologies Co., Ltd., patent No. 201910220031.0 disclosed by Li Zongming, patent No. 201811252938.7 disclosed by Beijing Moshanghua Technology Co., Ltd., patent No. 201911206343.2 disclosed by Shenzhen Zhangzhong Information Technology Co., Ltd., and patent No. 201610816210.7 authorized by Shanghai Zhangmen Technology Co., Ltd. These solutions are all trying to integrate the advertisement screen into the original video to achieve the purpose of advertisement exposure.

[0004] Although these methods can insert ads into videos, they modify the original videos, which inevitably interferes with the user's viewing experience and even causes cognitive confusion. From the perspective of advertising placement rules, the exposure of ads in videos can only be more natural, friendly and effective if it is controlled by the creator.

[0005] However, this kind of product placement, which truly integrates the image of the object into the video naturally, can often only be carried out by traditional movies, TV programs, advertisers or agencies in cooperation, and cannot be standardized and popularized, and is not universal. Under the general trend of user-generated content, mobile media has become an advertising media that advertisers rely on, and advertising budgets are constantly focusing on mobile terminals. In particular, a large number of video media and live broadcasts generated by users have gradually become the main content source of mobile video media. At the same time, creators are also eager to use the richness of the images of this video media to appropriately insert advertisements to create more revenue.

[0006] Methods for monitoring video ads have also been disclosed. For example, the "Advertising Placement Monitoring System Based on Image Recognition" filed by China Railway Engineering Technology Co., Ltd. with application number 201911183287.5 uses a technology that compares the original advertisement with the monitoring image of the monitoring video. However, its purpose is to monitor whether there is any misplacement of ads. It is not possible to monitor ads that only appear in a certain area of ​​the dynamic video.

[0007] Although e-commerce has penetrated into various fields, there is currently no platform for the placement, response, production, release and monitoring of product placement advertisements for photographed subjects. Unlike conventional e-commerce platforms, this platform not only needs to provide a standardized product placement bidding module to achieve the transaction of virtual goods with advertising placement as the subject, but also needs to technically monitor the exposure duration and exposure effect of product placement advertisements.

[0008] With the breakthrough of artificial intelligence technology, machine learning and deep learning technologies in the fields of shot and scene conversion, key frame extraction, object detection and recognition have become relatively mature. However, such technologies are generally used in the fields of video content annotation and retrieval, video content detection and recognition, video encoding, information security, etc., and there is no precedent for combining them to monitor embedded advertising in video content.

[0009] "Video scene switching detection is the basis for key frame extraction, video segmentation, video content analysis and retrieval and other applications" (Lu Siye et al., Shanghai University of Engineering Science, Scene Switch Detection Algorithm and Implementation Based on Dual Threshold Grayscale Histogram, Journal of Shanghai University of Engineering Science, March 2018). These technologies often need to process most of the video frames in the video. For example, the lens switching technology is often characterized by frame-by-frame detection, comparing the previous and next frames of the video to identify the sudden or gradual changes of the lens, which often requires a large amount of calculation.

[0010] Key frame extraction technology often extracts key frames by comparing inter-frame information based on shot segmentation. When the video is divided into shots, the key frame representing the segment is selected. The widely used key frame extraction technology requires a lot of calculations on each frame. For example, the frame averaging method needs to calculate the average value of all frames in a shot at a certain pixel point, as well as the difference between the pixels of each frame and this average value, and the one with the smallest difference is used as the key frame; the key frame extraction technology based on visual content matching needs to compare the difference between the first frame of the shot and all frames to set a threshold to determine the key frame; the key frame extraction method based on clustering needs to cluster all frames in the video. (Liu Huayong et al., Key frame extraction algorithm based on improved block color features and secondary extraction, Computer Science, December 2015)

[0011] Most of the above methods are computationally intensive, resource-intensive, and time-consuming. Although they are well-used in other applicable fields, they are not the most efficient methods for monitoring embedded advertising. The key frame extraction method based on sampling is simple and easy to use, but it cannot match the advertising exposure in the video content.

[0012] In the case of a large number of various video resources and high real-time requirements, it is necessary to develop more efficient and applicable advertising exposure monitoring capabilities to meet advertisers' requirements for the frequency and effectiveness of advertising embedded in video media. Summary of the invention

[0013] The present invention provides a system and method for placing, responding to, and monitoring embedded advertisements in video media, so as to solve the problems faced by the prior art in placing and monitoring embedded advertisements in video media, such as advertisement placement management and transaction, large number of video media, strong real-time performance, and advertisement exposure effect monitoring.

[0014] The present invention provides a system and method for placing, responding to, and monitoring video media embedded advertisements. The system includes an account system, a placing and responding system, a monitoring system, and a settlement system.

[0015] Advertisers and creators register accounts through the account system;

[0016] The advertiser releases the advertising target in the delivery and response system and provides the visual image information of the advertising target as the exposure target;

[0017] The creators respond to the target according to the content requirements of the video they are making, and when shooting, they embed the visual image of the advertising target into the video screen to make a responsive video;

[0018] The monitoring system includes a key frame extraction module, an advertising target detection and recognition module, and a timing module; the key frame extraction module extracts the first key frame of the segment from a certain starting time point of the response video, and continues to extract key frames until the response video ends; the advertising target detection and recognition module detects whether there is an exposed target in the key frame screen, and takes the first and last key frames in which the exposed target appears as the start and end time points of the exposure period; the timing module calculates the exposure duration according to the start and end time points of each exposure period; if there are multiple exposure periods, the total cumulative exposure duration is calculated;

[0019] The settlement system calculates and pays the advertising fee based on the exposure duration of the exposure target in the response video and the advertising fee calculation method agreed upon by the advertiser in the delivery and response system.

[0020] Preferably,

[0021] When the same response video responds to multiple advertising targets, each designated start time is used as the first key frame of the advertising target. However, when entering a video period that responds to another advertising target at the same time, the newly entered first key frame is added as the key frame of the previous advertising target; and the stepping time point is set as the minimum stepping of these targets in the video period of the common response. The stepping can be in time intervals or frames.

[0022] For example, a response video responds to three advertising labels A, B, and C at the same time. The minimum exposure duration of A, B, and C is 4 seconds, 6 seconds, and 7 seconds respectively. The start time submitted by the creator indicates that the visual image of A appears at the 1st second, B at the 6th second, and C at the 15th second. Assume that the step duration of A, B, and C is set to 2 seconds, 2.5 seconds, and 3 seconds respectively, and A no longer appears after the 11th second, and B no longer appears after the 17th second. Based on the above assumptions, the key frame extraction module will extract key frames at the following time points respectively: 1 second, 1+2=3 seconds, 3+2=5 seconds; 6 seconds, 6+2=8 seconds, 8+2=10 seconds, 10+2=12 seconds, 12+2.5=14.5 seconds; 15 seconds, 15+2.5=17.5 seconds; 17.5+3=20.5 seconds…; but the key frames for advertisement target A are the key frames of the 1st, 3rd, 5th, 6th, 8th, 10th, and 12th seconds; the key frames for advertisement target B are the key frames of the 6th, 8th, 10th, 12th, 14.5, 15th, and 17.5th seconds; the key frames for advertisement target C are the key frames of the 15th, 17.5, 20.5…and subsequent frames.

[0023] Preferably,

[0024] The monitoring system also includes a counting module. While recording the start and end time of each clip's exposure target, the timing module also records the number of viewers of the response video at that time point. When calculating the advertising fee, the settlement system takes into account the exposure time and the number of viewers.

[0025] Preferably,

[0026] The monitoring system also includes a shot continuity detection module, which uses the method of "first extracting key frames, then identifying shot switching, and then identifying content as needed" to determine the exposure period; after the key frame extraction module extracts key frames step by step at a certain interval, the shot continuity detection module compares the image differences between two adjacent key frames to determine whether the shots between the two adjacent key frames are continuous shots;

[0027] When the previous key frame of the adjacent key frame has an exposed object; if the two frames are continuous shots, it is considered that the object is exposed during the continuous shot period, and the time between the two frames is counted as the exposure time; if the two frames are non-continuous shots, the subsequent key frame is detected for the exposed object; if the subsequent key frame has an exposed object, it is still considered that the object is exposed during the shot period between the two frames, and the time between the two frames is counted as the continuous exposure time; if the subsequent key frame does not have an exposed object, it is considered that the exposed object stops appearing during the shot period between the two frames, and a certain time point between the two frames is recorded as the end time point of the exposure time;

[0028] When there is no exposed target in the previous key frame of adjacent key frames; if the two frames are continuous shots, it is considered that there is no exposed target in the continuous shot period; if the two frames are non-continuous shots, the subsequent key frame is detected for exposed targets; if there is an exposed target in the subsequent key frame, it is considered that the exposed target begins to appear in the shot period between the two frames, and a certain time point between the two frames is recorded as the starting time point of the exposure time; if there is still no exposed target in the subsequent key frame, it is considered that the exposed target has not appeared in the shot period between the two frames.

[0029] Preferably,

[0030] In the process of extracting the key frame of a certain step point step by step, the key frame extraction module not only extracts the video frame Fb of the set step point, but also traces back several frames before Fb to extract the previous candidate video frame Fa or extracts the next candidate video frame Fc several frames after Fb. The shot continuity detection module detects and compares the differences between Fa, Fc and Fb, and determines the key frame based on whether the difference exceeds the set threshold.

[0031] When the previous key frame of the adjacent key frame has an exposed target, only Fa is extracted and the difference between Fa and Fb is detected; if the difference between Fa and Fb exceeds the threshold, the key frame of the step point is determined as Fa; if the difference between Fa and Fb does not exceed the threshold, the key frame of the step point is determined as Fb;

[0032] When there is no exposed target in the previous key frame of the adjacent key frame, only Fc is extracted and the difference between Fc and Fb is detected; if the difference between Fc and Fb exceeds the threshold, the key frame of the step point is determined to be Fc, if the difference between Fc and Fb does not exceed the threshold, the key frame of the step point is determined to be Fb.

[0033] Preferably,

[0034] The stepping of the key frame extraction module to extract the key frame can be the interval time or the number of interval frames; when the advertiser releases the advertising product as the subject in the delivery and response system, the advertiser specifies the minimum duration T of a single exposure. If the stepping interval is the time interval, the time interval Tf≤T / 2;

[0035] When the previous key frame of the adjacent key frame has an exposed object, the two frames are non-continuous shots, and the subsequent key frame does not have an exposed object, the time point Tf / 2 before the subsequent key frame is assumed to be the end time point of the exposure time;

[0036] When there is no exposed object in the previous key frame of adjacent key frames, and there are non-continuous shots between the two frames, and there is an exposed object in the subsequent key frame, the time point Tf / 2 before the subsequent key frame is assumed to be the starting time point of the exposure time.

[0037] The lens continuity detection module of the monitoring system can also be operated in the following ways:

[0038] The shot continuity detection module determines whether the video is a continuous shot by comparing the differences between adjacent video frames;

[0039] If the response video is a finished video file, the shot continuity detection module first divides the response video into a number of shot segments, the key frame extraction module extracts a number of key frames in each shot segment, and the advertising target detection and recognition module detects and recognizes whether there is an exposed target in the key frame;

[0040] In the case where the response video is a live video stream, after the advertising target detection and recognition module identifies the first key frame containing the exposed target, the lens continuity detection module detects in real time whether the video lens has been switched; if the lens has not been switched, the timing continues, and the key frame extraction module intermittently extracts key frames for confirmation until it is detected that the exposed target no longer appears in the key frame, and the time of the previous key frame is used as the end time; if the lens switches, the next round of recognition and timing is restarted.

[0041] In order to make the system more in line with actual usage needs, the following technical improvements can be made.

[0042] The shot continuity detection module can select different detection algorithms based on different efficiency performance; when the response video is a finished video file, the shot continuity detection module adopts an algorithm that prioritizes performance, including deep learning methods such as neural networks; when the response video is a live video stream, the shot continuity detection module adopts an algorithm that prioritizes efficiency, such as a color histogram algorithm.

[0043] The starting time point at which the key frame extraction module extracts the key frame may be one or more of the very beginning time point of the response video, a manually specified time progress point, and a time point in a standard timing system.

[0044] It also includes a labeling system. After the advertising target detection and recognition module detects the exposed target, the labeling system adds a floating hyperlink to the exposed target area in the video. The user can click on the floating hyperlink to obtain more information about the exposed target and perform further operations. The floating hyperlink area has a visual mark, and the video viewer can choose to turn it on or off.

[0045] This system is integrated into the video and live broadcast platform, and the account system is integrated with the account system of the video platform; on the playback page of the video and live broadcast content, the system appropriately displays the visual image of the advertising target and provides a hyperlink for the user to perform the next step.

[0046] When an advertiser places an advertising product as a subject in the delivery and response system, he or she submits not only the visual image information of the advertising subject, but also text information; the system also includes a feedback system, which tracks the comments and questions and answers under the response video. If it is detected that there is text information of the advertising subject in the comments or set answers in the questions and answers, the feedback system will record and submit it to the settlement system as a reference for the settlement of advertising fees.

[0047] The features and advantages of the present invention are:

[0048] 1. It provides a trading platform for advertisers to place advertising bids and creators to respond to bids, transforming the individualized operation of video media advertising implantation into standardized operations, meeting the needs of large-scale advertising placement and monitoring in short videos, live broadcasts and other video media.

[0049] 2. The advertising target delivery and response system provided by this technical solution allows creators to independently respond to advertising targets and determine the exposure of advertisements. The system itself does not modify or integrate the original videos of creators, but only monitors them. This avoids problems that may easily cause confusion in rights and interests and user cognition, and is more in line with the logic and rules of advertising delivery.

[0050] 3. The conventional video content recognition scheme is based on the mode of "first shot switching recognition, then key frame extraction, and then content recognition". Taking into account that the core indicator of embedded advertising monitoring is exposure duration, the present invention proposes a method of "first key frame extraction, then shot switching recognition, and then content recognition on demand", which greatly reduces the video content detection of unnecessary frames, makes the key frames relevant to the video content, avoids excessive parsing and retrieval while meeting the needs of advertising monitoring, greatly improves real-time performance, saves computing power, and adapts to the needs of rapid and large-scale detection of embedded advertising exposure in videos and live broadcast media. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The system framework diagram of the present invention is schematically shown.

[0052] Figure 2 The flowchart of the method of the present invention is schematically shown. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in a variety of different ways as defined and covered by the claims and technical solutions.

[0054] This embodiment is based on the general Internet and mobile Internet hardware and software architecture, including CDN, server, PC client, mobile terminal and other hardware and software and wired and wireless network facilities. In particular, the existing short video, video streaming or live broadcast platform can conveniently integrate this embodiment as an additional function.

[0055] The system of this embodiment includes the following subsystems.

[0056] Account system: Advertisers, creators, etc. register accounts through the account system and log in to perform relevant operations. When this embodiment is implemented by a video or live broadcast platform, the account system is integrated into the platform's system.

[0057] Delivery and response system: Advertisers publish advertising targets, upload visual images, and set the minimum exposure time T for each segment of the visual image on this system, the advertising fee calculation method, such as by display time and number of people, and set the unit price per unit time and number of people. If the creator believes that the visual image is suitable for embedding in the video he or she created, he or she will respond to the advertising target. After the response video is produced, the creator uploads it to the video website, shares the video to the delivery and response system and the monitoring system, and sets the time point when the visual image of the advertising target is first exposed. When the creator is a live broadcaster and wants to integrate the image into his or her live broadcast, after responding to the target, he or she will share the live broadcast content to the delivery and response system and the monitoring system.

[0058] Annotation system: After the monitoring system detects the visual image of the advertising target in the screen, it also records its regional position on the screen, and the annotation system adds a transparent floating layer above the visual image. The transparent floating layer has interactive settings, and users can click to view or interact, such as viewing the enlarged image of the visual image of the advertising target, detailed information, jumping to the purchase page, sharing, quoting to the feedback system, etc.

[0059] Feedback system: The feedback system tracks user comments or user Q&A content at the bottom of the video playback page. When an advertiser places an advertisement product as a subject in the delivery and response system, he or she submits not only the visual image information of the advertisement subject, but also the text information; the feedback system tracks the comments and Q&A under the response video. If it detects that there is text information of the advertisement subject in the comments or a set answer in the Q&A, it will be recorded and submitted to the settlement system as a reference for the settlement of advertising fees.

[0060] Settlement system: After the monitoring system collects statistics on the exposure time, number of viewers, feedback, etc. of the response video, the advertising fee is calculated according to the pricing method set by the delivery and response system. For example, it is calculated by the following formula: (the number of new viewers at the beginning of this exposure period in the billing cycle + the number of new viewers at the end of the billing cycle) × the number of seconds of this exposure period × the price per person per second × coefficient. If users click and interact through the annotation system, or if the feedback system detects that users have an impression of the advertising subject, there will be corresponding rewards. The settlement system calculates and pays the relevant fees to the creator.

[0061] The monitoring system is a system for monitoring the duration of the visual image of the advertising mark appearing in the response video, including a key frame extraction module, a shot continuity detection module, an advertising mark detection and recognition module, a timing module, a counting module, etc.

[0062] After decoding the response video product or buffered content, the key frame extraction module extracts the corresponding video frame according to the relevant algorithm, and determines the key frame in combination with the shot continuity detection module. In this embodiment, the first key frame is specified by the creator at a time point. If it is a live broadcast, the anchor specifies the network time when the visual image of the advertising subject will appear in the live broadcast; the subsequent key frames set the step time or step frame number according to the shortest duration of a single exposure. For example, if the shortest duration of a single exposure of a visual image required by an advertising subject is T seconds, the step time should not be greater than T / 2 seconds, or the step frame number should not be greater than T / 2×frame rate.

[0063] When the same response video responds to multiple advertising targets, each designated start time is used as the first key frame of the advertising target. However, when entering a video period that responds to another advertising target at the same time, the newly entered first key frame is added as the key frame of the previous advertising target; and the stepping time point is set as the minimum stepping of these targets in the video period of the common response. The stepping can be in time intervals or frames.

[0064] For example, a response video responds to three advertising labels A, B, and C at the same time. The minimum exposure duration of a single segment required by labels A, B, and C is 4 seconds, 6 seconds, and 7 seconds respectively; the start time submitted by the creator indicates that the visual image of A appears at the 1st second, B appears at the 6th second, and C appears at the 15th second. Assume that the step durations of A, B, and C are positioned at 2 seconds, 2.5 seconds, and 3 seconds respectively, and A no longer appears after the 11th second, and B no longer appears after the 17th second. Based on the above assumptions, the key frame extraction module will extract video frames at the following time points: 1 second, 1+2=3 seconds, 3+2=5 seconds; 6 seconds, 6+2=8 seconds, 8+2=10 seconds, 10+2=12 seconds, 12+2.5=14.5 seconds; 15 seconds, 15+2.5=17.5 seconds; 17.5+3=20.5 seconds... The key frames for advertisement subject A are video frames at 1, 3, 5, 6, 8, 10, and 12 seconds; the key frames for advertisement subject B are video frames at 6, 8, 10, 12, 14.5, 15, and 17.5 seconds; and the key frames for advertisement subject C are video frames at 15, 17.5, 20.5 seconds, and thereafter.

[0065] In addition to extracting the video frame Fb at the set step point, the key frame extraction module can also trace back several frames before Fb to extract the previous candidate video frame Fa or extract the next candidate video frame Fc several frames after Fb. The lens continuity detection module detects and compares the differences between Fa, Fc and Fb, and determines the key frame based on whether the difference exceeds the set threshold. Whether to extract candidate video frames at the step point to determine the key frame mainly depends on the characteristics of the response video and the requirements of the advertising target. If the response video lens scene switches frequently and the advertising target has strict requirements on the accuracy of the exposure time, the extraction of candidate video frames can be increased.

[0066] The shot continuity detection module determines whether the two frames are the same shot by comparing the differences between adjacent key frames. There are many methods to determine the difference between images, such as pixel comparison method, histogram comparison method, edge-based method, optical flow, motion vector-based method, etc.; as well as derived detection methods based on factors such as phase correlation and texture, and introduces machine learning, neural networks, deep learning and other concepts and algorithms. This module can select different detection algorithms based on different efficiency performance; when the response video is a finished video file, the shot continuity detection module adopts an algorithm that prioritizes performance, including deep learning methods such as neural networks; when the response video is a live video stream, the shot continuity detection module adopts an algorithm that prioritizes efficiency, such as a color histogram algorithm.

[0067] The shot continuity detection module combines the results of the continuity detection of adjacent key frames and whether the visual image of the advertising logo exists in the previous frame to determine whether to submit it to the advertising logo detection and recognition module for key frame content recognition, or to let the key frame extraction module extract the next frame. See the method and process of this embodiment for details.

[0068] Advertising logo detection and recognition module: used to identify the target object of the key frame image that needs to be detected for screen content, so as to determine whether the visual image of the advertising logo exists in the key frame image. The visual image of the advertising logo can be exposed in various forms, and may be deformed, distorted, changed in color and size, etc. in the video, but as long as it can be generally recognized, it can be considered as the advertising logo. This type of algorithm is also quite mature. For example, methods based on target shape, color, and texture features, based on edges, motion, and text recognition, etc., can still effectively identify whether the visual image of the advertising logo exists in the frame to be detected under the interference of factors such as distortion, occlusion, light and shadow changes, and deformation.

[0069] Timing module: When the system determines that the visual image of the advertising target exists in a certain video period, the timing module marks the start and end time of the video and calculates the exposure time. When there are multiple exposed video clips, the cumulative exposure time is calculated.

[0070] Counting module: The counting module records the number of users who viewed and interacted at the beginning and end of the exposure period within a billing cycle, and the user interaction feedback set in the feedback system, such as messages, questions and answers, clicks, purchases, jumps, shares, complaints, etc. The relevant data collected by the counting module is submitted to the settlement system for settlement.

[0071] The method and process of this system for delivering, responding to and detecting embedded advertisements in video media are as follows.

[0072] Step 1: Advertisers and creators register an account and log in.

[0073] Step 2: The advertiser publishes the ad target, uploads the visual image V that he wants to be shown in the video media, and sets the minimum duration T for a single exposure (i.e., each exposure of the visual image V should last at least T seconds or T frames). The advertiser can also set user participation such as interaction, clicks, purchases, and Q&A.

[0074] Step 3: If the creator believes that the visual image V can be shown in the video he created, he will actively respond to the target. After uploading the video he created to the video platform or live broadcasting, he will share the relevant permissions with this system and indicate that the visual image V will start to appear at the starting time point t0 seconds or t0 frame. If the creator does not specify, the time of the first frame of the response video is considered to be t0.

[0075] Step 4: For the response video or its buffered content, the key frame extraction module extracts the first key frame at the starting time point t0. If the creator does not specify the starting time point t0, the first frame of the response video is used as the first key frame.

[0076] Step 5: The advertisement mark detection and recognition module detects and recognizes whether the advertisement mark visual image V exists in the first key frame. If it exists, the process proceeds to step 6; if not, the process proceeds to step 15 to extract the next step entry point t0+T / 2 video frame Fb and the subsequent candidate video frame Fc (Fb is several frames in advance).

[0077] Step 6, set t0 as the start time of the exposure time.

[0078] Step 7: extract the video frame Fb of the next step point and its previous candidate video frame Fa (Fb traces back several frames).

[0079] Step 8, comparing the picture difference between the Fa and Fb video frames to see whether it exceeds the set threshold, if so, the key frame of the step point is set as Fa; if not, the key frame of the step point is set as Fb.

[0080] Step 9, compare the key frame of the step point with the previous key frame to detect whether there is a continuous shot between the two key frames. If it is a continuous shot, go to step 10; if not, go to step 11.

[0081] Step 10, record the time of the key frame of the step point as the continuous exposure time, and restart step 7.

[0082] Step 11, the advertising logo detection and recognition module detects whether the advertising logo visual image V exists in the key frame of the current step point. If yes, proceed to step 10, and record the time of the key frame of the step point as the continuous exposure time; if no, proceed to step 12.

[0083] Step 12, setting the end point of the exposure time of the visual image V of the advertisement mark in the video period as T / 4 before the key frame.

[0084] Step 13, combining the start time and the end time of the exposure period, calculating the exposure duration of the exposure period, the counting module records the number of user views at the end time, and the feedback module records the feedback data.

[0085] Step 14, check whether the response video has ended. If the remaining time of the response video is less than the step time, or the remaining number of frames is less than the step frame, the response video is determined to have ended. If the response video has not ended, proceed to step 15; if the response video has ended, proceed to step 20.

[0086] Step 15: The key frame extraction module extracts the video frame Fb of the next step point and the subsequent candidate video frame (Fb is several frames ahead).

[0087] Step 16, comparing the picture difference between the Fb and Fc video frames to see whether it exceeds a set threshold, if so, the key frame of the step point is set as Fc; if not, the key frame of the step point is set as Fb.

[0088] Step 17, compare the key frame of the step point with the previous key frame to detect whether there is a continuous shot between the two key frames. If it is a continuous shot, go to step 15; if not, go to step 18.

[0089] Step 18, the advertising logo detection and recognition module detects whether there is a visual image V of the advertising logo in the key frame at this step point. If not, go to step 15; if yes, go to step 19.

[0090] Step 19, taking T / 4 seconds before the key frame as the start time point of the exposure of the segment, entering step 7, and starting a loop to detect the exposure duration.

[0091] Step 20, calculate the total cumulative exposure time of each segment.

[0092] Step 21, based on the viewing and feedback conditions at each time point and the advertising fee setting of the advertising target, calculate the advertising fee that the advertiser should pay for the response video and make the payment.

[0093] In the above-mentioned cycle, if the visual image V does not appear in all key frames, and the remaining time or remaining frames of the video are less than the step time or step frames, it is considered that the response video does not effectively respond to the advertising target and no payment is made.

[0094] Through the above technical scheme, the present invention can make the existing video and live broadcast platforms more efficient, low-cost, batch-based, highly friendly and high-conversion in the delivery, response and monitoring of embedded advertisements in video media.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for placing, responding to, and monitoring video media embedded advertising, characterized by: Advertisers and creators register accounts through the account system; The advertiser releases the advertising target in the delivery and response system and provides the visual image information of the advertising target as the exposure target; The creators respond to the target according to the content requirements of the video they are making, and when shooting, they embed the visual image of the advertising target into the video screen to make a responsive video; The key frame extraction module extracts the first key frame of the video clip from a certain starting time point of the response video, and continues to extract key frames until the end of the response video; The advertisement target detection and recognition module detects and recognizes whether there is an exposed target in the key frame screen, and takes the first and last key frames where the exposed target appears as the start and end time points of the exposure period; the timing module calculates the exposure duration according to the start and end time points of each exposure period; if there are multiple exposure periods, the total cumulative exposure duration is calculated; The monitoring system uses the method of "first extracting key frames, then identifying shot switching, and then identifying content as needed" to determine the exposure period; after the key frame extraction module extracts key frames step by step at a certain interval, the shot continuity detection module compares the image differences between two adjacent key frames to determine whether the shots between the two adjacent key frames are continuous shots; When the target is exposed in the previous key frame of the adjacent key frame; if the two frames are continuous shots, it is considered that the target is exposed in the continuous shot period, and the time between the two frames is counted as the continuous exposure time; if the two frames are non-continuous shots, the target is detected in the subsequent key frame; if the target is exposed in the subsequent key frame, it is still considered that the target is exposed in the shot period between the two frames, and the time between the two frames is counted as the exposure time; If there is no exposed target in the subsequent key frame, it is considered that the exposed target stops appearing in the shot period between the two frames, and a certain time point between the two frames is recorded as the end time point of the exposure period; When there is no exposed target in the previous key frame of the adjacent key frame; if the two frames are continuous shots, it is considered that there is no exposed target in the continuous shot period; if the two frames are non-continuous shots, the exposed target detection is performed on the subsequent key frame; if there is an exposed target in the subsequent key frame, it is considered that the exposed target begins to appear in the shot period between the two frames, and a certain time point between the two frames is recorded as the starting time point of the exposure time; if there is still no exposed target in the subsequent key frame, it is considered that the exposed target has not appeared in the shot period between the two frames; The settlement system calculates and pays the advertising fee based on the exposure duration of the exposure target in the response video and the advertising fee calculation method agreed upon by the advertiser in the delivery and response system.

2. A method for placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: When the same response video responds to multiple advertising targets, each designated start time is used as the first key frame of the extracted advertising target. However, when entering a video time period that also responds to another advertising target, the newly entered first key frame is added as the key frame of the previous responding advertising target; and within the common response video time period, the step timing point is set to the minimum step among these targets.

3. The method for placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: The monitoring system also includes a counting module. While recording the start and end time of each clip's exposure target, the timing module also records the number of viewers of the response video. When calculating the advertising fee, the settlement system takes into account the exposure time and the number of viewers.

4. The method for placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: In the process of extracting the key frame of a certain step point step by step, the key frame extraction module extracts the video frame Fb of the set step point, and also extracts the previous candidate video frame Fa by going back several frames before Fb or extracts the next candidate video frame Fc by several frames after Fb. The shot continuity detection module detects and compares the differences between Fa, Fc and Fb, and determines the key frame based on whether the difference exceeds the set threshold. When the previous key frame of the adjacent key frame has an exposed target, only Fa is extracted and the difference between Fa and Fb is detected; if the difference between Fa and Fb exceeds the threshold, the key frame of the step point is determined as Fa; if the difference between Fa and Fb does not exceed the threshold, the key frame of the step point is determined as Fb; When there is no exposed target in the previous key frame of the adjacent key frame, only Fc is extracted and the difference between Fc and Fb is detected; if the difference between Fc and Fb exceeds the threshold, the key frame of the step point is determined to be Fc, if the difference between Fc and Fb does not exceed the threshold, the key frame of the step point is determined to be Fb.

5. The method for placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: The key frame extraction module extracts the stepping of the key frame, which includes the interval time or the number of interval frames; when the advertiser releases the advertising product as the subject in the delivery and response system, the advertiser specifies the minimum duration T of a single exposure. If the stepping interval is the time interval, the time interval Tf≤T / 2; When the previous key frame of the adjacent key frame has an exposed object, and the two frames are non-continuous shots, and the subsequent key frame does not have an exposed object, the time point Tf / 2 before the subsequent key frame is assumed to be the end time point of the exposure time; When there is no exposed object in the previous key frame of adjacent key frames, and there are non-continuous shots between the two frames, and there is an exposed object in the subsequent key frame, the time point Tf / 2 before the subsequent key frame is assumed to be the starting time point of the exposure time.

6. The method for placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: The shot continuity detection module determines whether the video is a continuous shot by comparing the differences between adjacent video frames; If the response video is a finished video file, the shot continuity detection module first divides the response video into a number of shot segments, the key frame extraction module extracts a number of key frames in each shot segment, and the advertising target detection and recognition module detects and recognizes whether there is an exposed target in the key frame; If the response video is a live video stream, after the advertisement target detection and recognition module recognizes the first key frame containing the exposed target, the shot continuity detection module detects in real time whether the video shot has been switched; If the lens does not change, the timing continues. The key frame extraction module intermittently extracts key frames for confirmation until it detects that the exposed target no longer appears in the key frame, and the time of the previous key frame is used as the end time; if the lens switches, the next round of recognition and timing will be restarted.

7. The method of placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: The key frame extraction module extracts the starting time point of the key frame including one or more of the starting time point of the response video, a manually specified time progress point, and a time point in a standard timing system.

8. The method for placing, responding to, and monitoring video media embedded advertising according to claim 1, characterized in that: After the advertising target detection and recognition module detects the exposed target, the annotation system adds a floating hyperlink to the exposed target area in the video. Users can click on the floating hyperlink to obtain more information about the exposed target and perform further operations. The floating hyperlink area has a visual mark and the video viewer can choose to turn it on or off.

9. A system for delivering, responding to, and monitoring video media embedded advertising, used to implement the method for delivering, responding to, and monitoring video media embedded advertising as claimed in any one of claims 1 to 8, characterized in that: It includes an account system, a delivery and response system, a monitoring system, and a settlement system. The system is integrated into the video and live broadcast platform, and the account system is integrated with the account system of the video platform. On the playback page of the video and live broadcast content, the system appropriately displays the visual image of the advertising target and provides a hyperlink for the user to proceed to the next step.

Images