Method, system and device for dynamically implanting advertisement in video based on image recognition

Abstract

The invention discloses a method, a system and a device for dynamically implanting an advertisement in a video based on image recognition, and relates to the technical field of video processing, a trained scene recognition model outputs a recognized and acquired video scene and related information according to a scene type in a current video live broadcast; segmenting the video frame into a plurality of segmented areas, selecting a target area for advertisement implantation in the segmented areas, and sending an implantation instruction to the outside; matching a corresponding target advertisement by using an advertisement recommendation algorithm, implanting the target advertisement into the target area, generating a recommendation coefficient after the advertisement is implanted according to the advertisement recommendation data, and if the recommendation coefficient is lower than expectation, re-matching the target advertisement and implanting the target advertisement; and obtaining advertisement feedback data in each observation period, generating an improvement degree, and constraining the advertisement implantation frequency in the current stage. Through higher matching between the advertisement and the current live broadcast scene and audience interest preference, better feedback can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method, system and device for dynamically inserting advertisements into videos based on image recognition. Background Art

[0002] Dynamic video ad placement is an innovative form of advertising that cleverly embeds advertising content in the form of video into video content, such as movies, TV series, online programs, or user-generated videos. This placement method not only maintains the dynamic and vivid nature of advertising information, but also makes the advertisement more naturally integrated into the audience's viewing experience through deep integration with video content. Dynamic video ad placement can attract the audience's attention, increase the exposure and memorability of the advertisement, and at the same time reduce the audience's resistance to the advertisement, thereby maximizing the advertising effect.

[0003] The Chinese invention patent with application publication number CN105141987A specifically involves an advertisement implantation method and an advertisement implantation system, which obtains an image element tag map from the dynamic video, wherein the image element tag map includes at least one image element; determines whether the image element tag map matches the advertisement graphic element to be implanted; and if the image element tag map matches the advertisement graphic element to be implanted, then the advertisement is implanted. The advertisement implantation method implants advertisement entities without affecting the audience's viewing of the video; allows the audience to select their favorite items and conduct commodity searches; can actively push relevant advertisements in a targeted manner according to the audience's preferences; and can make the implantation effect more realistic.

[0004] Combined with the above application and the contents of the prior art:

[0005] The existing method of inserting advertisements in live video usually pauses the current live broadcast process and continues the live broadcast after the advertisement playing process is completed. However, live video broadcast, especially sports video broadcast, needs to maintain the continuity of the video, so this video insertion method usually cannot obtain effective feedback. Therefore, when the feedback data such as the exposure rate, completion rate and click rate of the advertisement are poor, some pictures in the dynamic video broadcast are usually selected and the corresponding advertisements are inserted in the pictures. This method of inserting advertisements will not affect the continuation of the live video broadcast, and the feedback data obtained is relatively good.

[0006] However, this method of dynamically inserting advertisements into videos usually requires first extracting scene information of the current live video broadcast and information on audience interests and preferences, and then matching the corresponding target advertisements in a pre-prepared advertising program library before recommending them to the audience. Although the feedback effect of this advertising recommendation method is relatively good, after completing the advertising recommendation and implantation process, due to the lack of a corresponding feedback evaluation process, the advertising recommendation and implantation cannot be adjusted in a timely and effective manner, resulting in the fact that the effect of advertising implantation may still be difficult to achieve as expected.

[0007] To this end, the present invention provides a method, system and device for dynamically inserting advertisements into videos based on image recognition. Summary of the invention

[0008] 1. Technical issues to be solved

[0009] In view of the deficiencies of the prior art, the present invention provides a method, system and device for dynamically inserting advertisements in videos based on image recognition, which divides the video frame into several segmented areas, selects the target area for advertisement insertion in the segmented area and issues an insertion instruction to the outside; uses an advertisement recommendation algorithm to match the corresponding target advertisement, and inserts the target advertisement into the target area; generates a recommendation coefficient after the advertisement is implanted from the advertisement recommendation data; if the recommendation coefficient is lower than expected, re-matches the target advertisement and inserts it; obtains advertisement feedback data and generates an improvement degree in each observation cycle, and constrains the advertisement insertion frequency in the current stage. By making the advertisement more compatible with the current live broadcast scene and the audience's interest preferences, better feedback can be obtained, thereby solving the technical problems raised in the background technology.

[0010] (II) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0012] A method for dynamically inserting advertisements in a video based on image recognition, comprising: collecting advertisement feedback data of a live video broadcast to generate an advertisement feedback data set, generating a feedback coefficient Fxs from the advertisement feedback data, and issuing a prompt instruction to an external device if the feedback coefficient Fxs does not exceed a feedback threshold;

[0013] The current live video is collected, and key frames are extracted after preprocessing the video. The extracted key frames are used as input, and the trained scene recognition model identifies the scene type in the current live video and outputs the identified video scene and related information;

[0014] Perform edge detection on video frames in live video broadcast, and divide the video frames into several segmented areas according to the edge detection results, and score the importance of each segmented area. Select the target area for advertisement implantation in the segmented area according to the importance score and issue an implantation instruction to the outside;

[0015] According to the video scene and the characteristic information of the audience, the corresponding target advertisement is matched using the advertisement recommendation algorithm, and the target advertisement is implanted into the target area. The recommendation coefficient Tjs after the implanted advertisement is generated from the advertisement recommendation data. If the recommendation coefficient Tjs is lower than expected, the target advertisement is re-matched and implanted.

[0016] The advertisement feedback data is obtained in each observation period and the improvement degree Imp is generated. If the improvement degree Imp is lower than the improvement threshold, the advertisement insertion frequency in the current stage is restricted.

[0017] Furthermore, when advertisements are inserted in the live video, advertisement feedback data is collected in each sub-period, and the advertisement feedback data is aggregated to construct an advertisement feedback data set; wherein, the exposure rate Bv and the click rate Jv are linearly normalized, and the corresponding data values ​​are mapped to the interval [0,1], according to the following method:

[0018]

[0019] Weight coefficient: 0≤k1≤1, 0≤k2≤1 and k2+k1=1; Bv i is the exposure rate in the ith sub-period, Bv b is the qualified standard value of exposure rate; Jv i is the click rate in the ith sub-period, Jv b The qualified standard value for click rate.

[0020] Furthermore, after receiving the prompt instruction, the current live video is collected and the video is preprocessed to obtain the preprocessed live video; the preprocessed video is used as the target video, the target video is divided into several continuous video frames, and a similarity analysis is performed on the current video frame and other video frames to obtain similarity data.

[0021] Furthermore, the similarity center Xct of each video frame is obtained by calculating a number of similarity data, and each video frame is marked with the similarity center Xct. If the video center Xct exceeds the preset center threshold, it is used as a key frame, and the similarity center Xct is generated in the following manner:

[0022]

[0023] Where n is the number of video frames, Xs(i,j) is the similarity between the i-th video frame and the j-th video frame, and Xs a is the average similarity; weight coefficient: 0≤S1≤1, 0≤S2≤1, and S1+S2=1.

[0024] Furthermore, the video frames in the live video are preprocessed, and the edge detection algorithm is applied to identify the edge information in the video frames, the detected edges are refined, unnecessary edge fragments are removed and broken edges are connected to form a complete edge contour; the region growing algorithm is applied to segment the image into multiple regions, each region contains similar pixel values ​​and texture features, and the image is segmented into foreground and background regions to obtain several segmented regions.

[0025] Furthermore, feature extraction is performed on the acquired segmented regions, and the acquired regional features are summarized; a trained segmented region evaluation model is obtained; after classifying several segmented regions, the regional features within the segmented regions are used as input, and the trained segmented region evaluation model scores the importance of each segmented region.

[0026] Further, after receiving the implantation instruction, the current advertisement data to be implanted is obtained, including at least the keywords, scene tags, target audience information and advertisement duration that should be associated with each advertisement page, and the advertisement program database is obtained after being summarized;

[0027] According to the current video scene and the characteristic information of the audience, the trained advertising recommendation algorithm is used to match the corresponding target advertisement from the advertising program database; after the target advertisement is implanted in the target area, the advertising content is integrated with the video screen by the image fusion algorithm.

[0028] Furthermore, a number of continuous observation cycles are set, and advertisement recommendation data is collected in each observation cycle, including: the correlation between the recommended advertisements and the video scenes, and the non-overlapping time between the recommended advertisements and the corresponding video scenes, and the advertisement recommendation information is aggregated to generate an advertisement recommendation data set;

[0029] The recommendation coefficient Tjs after the implanted advertisement is generated from the advertisement recommendation data set. If the recommendation coefficient Tjs is lower than the recommendation threshold, the current live advertisement recommendation algorithm is optimized. According to the optimized advertisement recommendation algorithm, the corresponding target advertisement is matched from the advertisement program database, and the target advertisement is implanted in the current live video.

[0030] Furthermore, the correlation coefficient Rt and the non-overlapping duration Ct are linearly normalized, and the corresponding data values ​​are mapped to the interval [0,1], and the recommendation coefficient Tjs is generated according to the following formula:

[0031]

[0032] Among them, α is the weight coefficient, Rt i is the correlation coefficient between the i-th recommended advertisement and the video scene, Rt p is the acceptable value of the correlation coefficient, Ct iis the non-overlapping duration between the i-th recommended advertisement and the corresponding video scene, Ct p is an acceptable value for non-overlapping durations.

[0033] Furthermore, after a number of consecutive observation cycles, advertisement feedback data is obtained in each observation cycle, feedback coefficients Fxs in the corresponding observation cycle are generated from the advertisement feedback data, and each observation cycle is marked with the feedback coefficients Fxs;

[0034] The improvement degree Imp is generated by the feedback coefficient Fxs. If the improvement degree mp is lower than the improvement threshold, an implantation frequency constraint instruction is issued to the outside.

[0035] Furthermore, the observation periods before and after the optimization of the advertising recommendation algorithm are sorted according to the time axis and correspond one to one, and the improvement degree Imp is generated by the feedback coefficient Fxs in the following way:

[0036]

[0037] Among them, No i is the median value of improvement in the ith observation period, No a is its mean, Mo p is its qualified standard value; i = 1, 2, ..., k, k is the number of observation cycles, and are the feedback coefficients of the ith observation period before and after maintenance, and is the corresponding mean.

[0038] Furthermore, after receiving the frequency restriction instruction, the frequency of advertisement insertion in the current stage is restricted, and the restriction conditions are as follows:

[0039]

[0040] Where n is the number of nodes where advertisements are implanted, Z ij is the time interval from the i-th advertisement placement to the j-th advertisement placement, Z a is the average value of the time interval, Imp is the improvement;

[0041] The advertisement insertion time node is reselected according to the constrained insertion frequency and used as the insertion node, and the advertisement is inserted into the live video at the selected insertion node.

[0042] A system for dynamically inserting advertisements in a video based on image recognition includes a feedback data collection unit, which collects advertisement feedback data of a live video and generates an advertisement feedback data set, generates a feedback coefficient Fxs from the advertisement feedback data, and issues a prompt instruction to the outside if the feedback coefficient Fxs does not exceed a feedback threshold;

[0043] The video recognition unit collects the current live video and extracts key frames after preprocessing the video. The extracted key frames are used as input, and the trained scene recognition model identifies the scene type in the current live video and outputs the identified video scene and related information.

[0044] A region segmentation unit performs edge detection on a video frame in a live video broadcast, and divides the video frame into a number of segmented regions based on the edge detection result, scores the importance of each segmented region, selects a target region for advertisement implantation within the segmented region based on the importance score, and issues an implantation instruction to the outside;

[0045] The recommendation effect evaluation unit uses the advertising recommendation algorithm to match the corresponding target advertisement according to the video scene and the characteristic information of the audience, and implants the target advertisement into the target area. The recommendation coefficient Tjs after the implanted advertisement is generated from the advertising recommendation data. If the recommendation coefficient Tjs is lower than expected, the target advertisement is re-matched and implanted.

[0046] The improvement analysis unit obtains advertisement feedback data in each observation period and generates an improvement degree Imp. If the improvement degree Imp is lower than an improvement threshold, the advertisement placement frequency in the current stage is restricted.

[0047] A device for dynamically inserting advertisements in a video based on image recognition, comprising at least one processor;

[0048] The memory is used to store executable instructions, where the instructions are executed by at least one of the processors to enable the at least one of the processors to perform the steps of the method for dynamically inserting advertisements in a video based on image recognition.

[0049] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the steps of a method for dynamically inserting advertisements in a video based on image recognition.

[0050] (III) Beneficial effects

[0051] The present invention provides a method, system and device for dynamically inserting advertisements in videos based on image recognition, which has the following beneficial effects:

[0052] 1. Evaluate the current advertising effect based on the feedback coefficient Fxs to verify whether the current advertising playback method can achieve the expected effect. If it is difficult to achieve the expected effect, the current advertising playback method can be adjusted in time to improve the advertising playback effect.

[0053] 2. Evaluate the differences of each video frame based on the similarity center Xct, select the parts with greater differences from several video frames, and extract key frames. By extracting key frames, the recognition effect can be better when performing scene recognition; perform scene recognition on the extracted key frames. After identifying and obtaining the corresponding scenes, more targeted advertising information can be matched based on the recognition data, and better feedback data can be obtained.

[0054] 3. Use the trained segmented region evaluation model to score the importance of each segmented region, so that the area where advertisements can be implanted can be selected within the video frame. Implanting advertisements will not affect the continuation of the live video broadcast, nor does it need to pause the current live video broadcast, so better feedback can be obtained when implanting advertisements.

[0055] 4. By collecting data in advance and building an advertising program database, we can match corresponding embedded advertisements based on the audience's interests and preferences based on the recommendation algorithm. By making the advertisements more closely matched with the current live broadcast scene and the audience's interests and preferences, we can obtain better feedback.

[0056] 5. Generate a recommendation coefficient Tjs from the acquired data, and make a comprehensive evaluation of the pertinence of the currently matched embedded advertisements based on the recommendation coefficient Tjs to determine whether the currently recommended advertisements are appropriate. If the degree of suitability is low, re-determine a new advertisement recommendation strategy to improve the effect of advertisement embedding.

[0057] 6. Generate an improvement degree mp from the feedback data obtained again, and use the improvement degree mp to evaluate the current advertising implantation effect; by quantitatively evaluating the advertising implantation effect, it is possible to confirm whether the current advertising implantation strategy is feasible. If not, it can be adjusted and corrected in a timely manner to ensure that the feedback effect of the advertising implanted in the dynamic video live broadcast can meet expectations.

[0058] 7. Restrict and adjust the current frequency of advertising insertion, reselect the time node for advertising insertion, reduce the audience's aversion to advertising insertion, and provide a certain guarantee for the effect of advertising insertion in dynamic video live broadcasts. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of a method for dynamically inserting advertisements in a video based on image recognition according to the present invention;

[0060] Figure 2 This is a schematic diagram of the system structure for dynamically inserting advertisements into videos based on image recognition according to the present invention;

[0061] Figure 3 The present invention is a schematic diagram of the structure of a device for dynamically inserting advertisements into a video based on image recognition. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] See also Figure 1 The present invention provides a method for dynamically inserting advertisements in a video based on image recognition, comprising:

[0064] Step 1: After collecting the advertisement feedback data of the live video, an advertisement feedback data set is generated, and a feedback coefficient Fxs is generated from the advertisement feedback data. If the feedback coefficient Fxs does not exceed the feedback threshold, a prompt instruction is issued to the outside;

[0065] The step 1 includes the following contents:

[0066] Step 101: When performing a live video broadcast, an observation period including several sub-periods is set. When an advertisement is inserted in the current live video broadcast, advertisement feedback data of the current live video broadcast is collected in each sub-period, including at least exposure rate, click-through rate, etc. The advertisement feedback data is aggregated to form an advertisement feedback data set;

[0067] Step 102: construct the feedback coefficient Fxs after the advertisement is inserted from the feedback data in the advertisement feedback data set, wherein the exposure rate Bv and the click rate Jv are linearly normalized and the corresponding data values ​​are mapped to the interval [0,1] in the following manner:

[0068]

[0069] Weight coefficient: 0≤k1≤1, 0≤k2≤1 and k2+k1=1; the weight coefficient can be obtained by referring to the hierarchical analysis method, Bv i is the exposure rate in the ith sub-period, Bv b is the qualified standard value of exposure rate; Jv i is the click rate in the ith sub-period, Jv b is the qualified standard value of click rate;

[0070] A feedback threshold is pre-set based on historical data and management expectations of the effect of advertising placement. If the feedback coefficient Fxs does not exceed the feedback threshold, it means that the current advertising placement effect is poor and needs to be readjusted. At this time, a prompt instruction is issued to the outside.

[0071] When using, combine the contents in steps 101 and 102:

[0072] When advertisements are inserted or played during the current live video broadcast, the current advertisement feedback data is collected, and the corresponding feedback coefficient Fxs is constructed based on the feedback data. The current advertisement playback effect is evaluated based on the feedback coefficient Fxs to verify whether the current advertisement playback method can achieve the expected effect. If it is difficult to achieve the expected effect, the current advertisement playback method can be adjusted in time to improve the advertisement playback effect.

[0073] The method of dynamically inserting advertisements into videos usually requires first extracting the scene information of the current live video and the audience's interests and preferences, then matching the corresponding target advertisements in the pre-prepared advertisement program library, and then recommending them to the audience. Although this advertisement recommendation method has relatively good feedback effects, after completing the advertisement recommendation and insertion process, due to the lack of a corresponding feedback evaluation process, the advertisement recommendation and insertion cannot be adjusted in a timely and effective manner, resulting in the advertisement insertion effect still being difficult to achieve the expected results.

[0074] Step 2: Capture the current live video, extract key frames after preprocessing the video, use the extracted key frames as input, and use the trained scene recognition model to identify the scene type in the current live video, and output the identified video scene and related information;

[0075] The step 2 includes the following contents:

[0076] Step 201: after receiving the prompt instruction, the current live video is collected and preprocessed, including denoising, color correction, etc., to improve the accuracy of image recognition, and the preprocessed live video is obtained; the preprocessed video is used as the target video, the target video is divided into a number of continuous video frames, and the current video frame is analyzed for similarity with other video frames to obtain similarity data;

[0077] Step 202: Calculate the similarity center Xct of each video frame from a number of similarity data in the following manner:

[0078]

[0079] Where n is the number of video frames, Xs(i,j) is the similarity between the i-th video frame and the j-th video frame, and Xs a is the average similarity; weight coefficient: 0≤S1≤1, 0≤S2≤1, and S1+S2=1;

[0080] Each video frame is marked with similarity center Xct. If the video center Xct exceeds the preset center threshold, it is used as a key frame.

[0081] When in use, after analyzing and obtaining several video frames, the similarity center Xct of each video frame is obtained based on the similarity analysis between different video frames, and the difference of each video frame is evaluated based on the similarity center Xct. The parts with greater differences can be screened out from several video frames to realize the extraction of key frames. By extracting key frames, the recognition effect can be better when performing scene recognition;

[0082] Step 203: After training the recurrent neural network with the sample data, a scene recognition model is obtained. The extracted key frames are used as input, and the trained scene recognition model recognizes the scene type, objects and people in the current live video, and outputs the recognized video scene and related information;

[0083] When using, combine the contents in steps 201 to 203:

[0084] After completing the extraction of key frames, the trained scene recognition model is used to perform scene recognition on the extracted key frames. After identifying and obtaining the corresponding scenes, more targeted advertising information can be matched based on the recognition data, and better feedback data can also be obtained.

[0085] Step 3: Perform edge detection on the video frame in the live video broadcast, and divide the video frame into several segmented areas according to the edge detection result, and score the importance of each segmented area. According to the importance score, select the target area for advertisement implantation in the segmented area and issue an implantation instruction to the outside;

[0086] The step three includes the following contents:

[0087] Step 301: pre-process the video frames in the live video broadcast, apply an edge detection algorithm to identify edge information in the video frames, refine the detected edges, remove unnecessary edge segments, and connect broken edges to form a complete edge contour;

[0088] Applying the region growing algorithm to segment the image into multiple regions, each region contains similar pixel values ​​and texture features, and segmenting the image into foreground and background regions to obtain several segmented regions;

[0089] Step 302: extract features from the acquired segmented regions, including features such as area, shape, and texture, and summarize the acquired regional features; train a convolutional neural network using sample data to obtain a trained segmented region evaluation model;

[0090] After classifying several segmented regions, the regional features in the segmented regions are used as input, and the trained segmented region evaluation model is used to score the importance of each segmented region, evaluate the importance of each region, and select the target region for advertisement placement in the segmented region based on the importance score, so as to avoid selecting the region containing important content, such as people and key objects, and issuing placement instructions to the outside.

[0091] When used, combine the contents in steps 301 and 302:

[0092] After edge detection and classification of the video frame, the video frame is segmented into several regions, and the importance of each segmented region is scored using the trained segmented region evaluation model. The region where advertisements can be implanted can be selected within the video frame. After the advertisements are implanted, the implanted advertisements will not affect the continuation of the live video broadcast, nor will the current live video broadcast need to be paused, so better feedback can be obtained when the advertisements are implanted.

[0093] Step 4: According to the video scene and the characteristic information of the audience, the corresponding target advertisement is matched using the advertisement recommendation algorithm, and the target advertisement is implanted into the target area. The recommendation coefficient Tjs after the implanted advertisement is generated from the advertisement recommendation data. If the recommendation coefficient Tjs is lower than expected, the target advertisement is re-matched and implanted;

[0094] The steps include the following:

[0095] Step 401: After receiving the implantation instruction, the current advertisement data to be implanted is obtained, including at least the keywords, scene tags, target audience information and advertisement duration that should be associated with each advertisement page, and the advertisement program database is obtained after being summarized;

[0096] During live video broadcast, the trained ad recommendation algorithm is used to match the corresponding target ads from the ad solution database based on the current video scene and the audience’s characteristic information, such as the audience’s interest preferences.

[0097] After the target advertisement is implanted in the target area, the image fusion algorithm makes the advertisement content and the video screen seamlessly integrated;

[0098] When in use, after selecting the target area for advertisement placement, data collection and construction of an advertisement program database are performed in advance. When the video is live, the corresponding placement advertisements can be matched according to the audience's interest preferences based on the recommendation algorithm. By making the advertisements more closely matched with the current live scene and the audience's interest preferences, better feedback can be obtained.

[0099] Step 402: set a number of continuous observation periods, collect advertisement recommendation data in each observation period, including: the correlation between the recommended advertisements and the video scenes, and the non-overlapping time between the recommended advertisements and the corresponding video scenes, and aggregate the above advertisement recommendation information to generate an advertisement recommendation data set;

[0100] Step 403: Generate a recommendation coefficient Tjs after the advertisement is embedded from the advertisement recommendation data set, wherein the correlation coefficient Rt and the non-overlapping duration Ct are linearly normalized, and the corresponding data values ​​are mapped to the interval [0,1] according to the following formula:

[0101]

[0102] Among them, α is the weight coefficient, and its value falls within [0.05, 0.95], which can be obtained through simulation software analysis or by referring to the hierarchical analysis method; Rt i is the correlation coefficient between the i-th recommended advertisement and the video scene, Rt p is the acceptable value of the correlation coefficient, Ct i is the non-overlapping duration between the i-th recommended advertisement and the corresponding video scene, Ct p is the acceptable value for non-overlapping duration;

[0103] According to historical data and the expected effect of recommended placement advertisements, a recommendation threshold is pre-set; if the recommendation coefficient Tjs is lower than the recommendation threshold, it means that the current recommendation effect fails to meet expectations, and the current live advertisement recommendation algorithm is optimized;

[0104] According to the optimized advertisement recommendation algorithm, the corresponding target advertisement is matched from the advertisement program database, and the target advertisement is embedded in the current live video;

[0105] When used, combine the contents in steps 401 to 403:

[0106] After completing the advertisement recommendation, the advertisement recommendation and its related data are collected, and the recommendation coefficient Tjs is generated from the acquired data. The targetedness of the currently matched implanted advertisement is comprehensively evaluated based on the recommendation coefficient Tjs to determine whether the currently recommended advertisement is appropriate. If the degree of suitability is low, a new advertisement recommendation strategy is re-determined to improve the effect of advertisement implantation.

[0107] Step 5: Obtain advertising feedback data in each observation period and generate an improvement degree Imp. If the improvement degree Imp is lower than the improvement threshold, restrict the frequency of advertising placement in the current stage.

[0108] The step five includes the following contents:

[0109] Step 501: After a number of consecutive observation cycles, advertisement feedback data is obtained in each observation cycle, feedback coefficients Fxs in the corresponding observation cycle are generated from the advertisement feedback data, and each observation cycle is marked with the feedback coefficients Fxs;

[0110] The observation periods before and after the optimization of the advertising recommendation algorithm are sorted according to the time axis and correspond one to one. The improvement degree Imp is generated by the feedback coefficient Fxs, and the improvement degree Imp is used to evaluate the current advertising implantation effect in the following way:

[0111]

[0112] Among them, No i is the median value of improvement in the ith observation period, No a is its mean, No p is its qualified standard value; i = 1, 2, ..., k, k is the number of observation cycles, and are the feedback coefficients of the ith observation period before and after maintenance, and is the corresponding mean.

[0113] Based on historical data and the expected effect of advertising recommendation, an improvement threshold is pre-set; if the improvement degree mp is lower than the improvement threshold, it means that after re-implanting the advertisement, the implantation effect of the advertisement still fails to achieve the expected effect. At this time, the frequency of advertisement implantation is adjusted and an implantation frequency constraint instruction is issued to the outside;

[0114] When using it, when advertising is implanted in live video, compared with the original advertising implantation method, the improvement degree mp is generated by the feedback data obtained again, and the current advertising implantation effect is evaluated with the improvement degree mp; by quantitatively evaluating the advertising implantation effect, it is possible to confirm whether the current advertising implantation strategy is feasible. If not, it can be adjusted and corrected in time to ensure that the feedback effect of implanting advertising in dynamic video live broadcast can meet expectations.

[0115] Step 502: After receiving the frequency restriction instruction, restrict the advertisement insertion frequency in the current stage. The restriction conditions are as follows:

[0116]

[0117] Where n is the number of nodes where advertisements are implanted, Z ij is the time interval from the i-th advertisement placement to the j-th advertisement placement, Z a is the average value of the time interval, Imp is the improvement;

[0118] Reselecting an advertisement insertion time node according to the constrained insertion frequency and using it as the insertion node, and inserting advertisements into the live video at the selected insertion node;

[0119] When using, combine the contents in steps 501 and 502:

[0120] When the current advertising implantation effect is poor, the current advertising implantation frequency is restricted and adjusted, and the time node for advertising implantation is reselected to reduce the audience's aversion to advertising implantation, thereby forming a certain guarantee for the effect of advertising implantation in dynamic video live broadcast.

[0121] The Analytic Hierarchy Process (AHP) is a decision-making method that decomposes decision-related elements into levels such as goals, criteria, and plans, and conducts qualitative and quantitative analysis on this basis. It is particularly suitable for dealing with target systems with hierarchical and staggered evaluation indicators, and when the target value is difficult to describe quantitatively, the AHP is an effective decision-making tool.

[0122] The core of the hierarchical analysis method is to decompose the decision problem into multiple levels to form a hierarchical structure, which usually includes the target level, the criterion level, the sub-criterion level and the solution level. By solving the eigenvector of the judgment matrix, the priority weight of each element of each level to an element of the previous level is obtained, and the final weight of each alternative solution to the total goal is recursively merged by the weighted sum method, so as to find the optimal solution.

[0123] See also Figure 2 The present invention provides a system for dynamically inserting advertisements in videos based on image recognition, comprising:

[0124] A feedback data collection unit collects advertisement feedback data of the live video and generates an advertisement feedback data set, generates a feedback coefficient Fxs from the advertisement feedback data, and issues a prompt instruction to the outside if the feedback coefficient Fxs does not exceed a feedback threshold;

[0125] The video recognition unit collects the current live video and extracts key frames after preprocessing the video. The extracted key frames are used as input, and the trained scene recognition model identifies the scene type in the current live video and outputs the identified video scene and related information.

[0126] A region segmentation unit performs edge detection on a video frame in a live video broadcast, and divides the video frame into a number of segmented regions based on the edge detection result, scores the importance of each segmented region, selects a target region for advertisement implantation within the segmented region based on the importance score, and issues an implantation instruction to the outside;

[0127] The recommendation effect evaluation unit uses the advertising recommendation algorithm to match the corresponding target advertisement according to the video scene and the characteristic information of the audience, and implants the target advertisement into the target area. The recommendation coefficient Tjs after the implanted advertisement is generated from the advertising recommendation data. If the recommendation coefficient Tjs is lower than expected, the target advertisement is re-matched and implanted.

[0128] The improvement analysis unit obtains advertisement feedback data in each observation period and generates an improvement degree Imp. If the improvement degree Imp is lower than an improvement threshold, the advertisement placement frequency in the current stage is restricted.

[0129] See also Figure 3 Based on the same inventive concept, according to another aspect of the present invention, the present invention also provides a device for dynamically inserting advertisements in a video based on image recognition, comprising at least one processor; a memory for storing executable instructions, wherein the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method for dynamically inserting advertisements in a video based on image recognition.

[0130] See also Figure 3 Based on the same inventive concept, according to another aspect of the present invention, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for dynamically inserting advertisements in a video based on image recognition are performed.

[0131] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for dynamically inserting advertisements in a video based on image recognition, characterized in that: include, After collecting the advertisement feedback data of the live video, an advertisement feedback data set is generated, and a feedback coefficient Fxs is generated from the advertisement feedback data. If the feedback coefficient Fxs does not exceed the feedback threshold, a prompt instruction is issued to the outside; The current live video is collected, and key frames are extracted after preprocessing the video. The extracted key frames are used as input, and the trained scene recognition model identifies the scene type in the current live video and outputs the identified video scene and related information; Perform edge detection on video frames in live video broadcast, and divide the video frames into several segmented areas according to the edge detection results, and score the importance of each segmented area. Select the target area for advertisement implantation in the segmented area according to the importance score and issue an implantation instruction to the outside; According to the video scene and the characteristic information of the audience, the corresponding target advertisement is matched using the advertisement recommendation algorithm, and the target advertisement is implanted into the target area. The recommendation coefficient Tjs after the implanted advertisement is generated from the advertisement recommendation data. If the recommendation coefficient Tjs is lower than expected, the target advertisement is re-matched and implanted. The advertisement feedback data is obtained in each observation period and the improvement degree Imp is generated. If the improvement degree Imp is lower than the improvement threshold, the advertisement insertion frequency in the current stage is restricted.

2. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 1, characterized in that: When advertisements are inserted in the live video, advertisement feedback data is collected in each sub-period, and the advertisement feedback data is aggregated to construct an advertisement feedback data set; among them, linear normalization is performed on the exposure rate Bv and the click rate Jv, and the corresponding data values ​​are mapped to the interval [0,1] in the following manner: Weight coefficient: 0≤k1≤1, 0≤k2≤1 and k2+k1=1; Bv i is the exposure rate in the ith sub-period, Bv b is the qualified standard value of exposure rate; jv i is the click rate in the ith sub-period, Jv b The qualified standard value for click-through rate.

3. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 2, characterized in that: After receiving the prompt instruction, the current live video is collected and pre-processed to obtain the pre-processed live video; The preprocessed video is used as the target video, and the target video is divided into several continuous video frames. The similarity analysis is performed on the current video frame and other video frames to obtain the similarity data.

4. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 3, characterized in that: The similarity center Xct of each video frame is calculated from a number of similarity data, and each video frame is marked with the similarity center Xct. If the video center Xct exceeds the preset center threshold, it is used as a key frame. The similarity center Xct is generated in the following way: Where n is the number of video frames, Xs(i,j) is the similarity between the i-th video frame and the j-th video frame, and Xs a is the average similarity; weight coefficient: 0≤S1≤1, 0≤S2≤1, and S1+S2=1.

5. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 4, characterized in that: The video frames in the live video are preprocessed, and the edge detection algorithm is used to identify the edge information in the video frames. The detected edges are refined, unnecessary edge fragments are removed and broken edges are connected to form a complete edge contour; the region growing algorithm is used to segment the image into multiple regions, each region contains similar pixel values ​​and texture features, and the image is segmented into foreground and background regions to obtain several segmented regions.

6. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 5, characterized in that: The obtained segmented regions are subjected to feature extraction, and the obtained regional features are summarized. After several segmented regions are classified, the regional features within the segmented regions are used as input, and the importance of each segmented region is scored by the trained segmented region evaluation model.

7. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 6, characterized in that: After receiving the implantation instruction, the current advertisement data to be implanted is obtained, including at least the keywords, scene tags, target audience information and advertisement duration that should be associated with each advertisement page, and the advertisement program database is obtained after being summarized; According to the current video scene and the characteristics of the audience, the trained advertising recommendation algorithm is used to match the corresponding target advertisement from the advertising program database; After the target advertisement is implanted in the target area, the image fusion algorithm is used to merge the advertisement content with the video screen.

8. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 7, characterized in that: Setting a number of continuous observation cycles, collecting advertisement recommendation data in each observation cycle, including: the correlation between the recommended advertisements and the video scenes, and the non-overlapping duration between the recommended advertisements and the corresponding video scenes, and aggregating the advertisement recommendation information to generate an advertisement recommendation data set; The recommendation coefficient Tjs after the implanted advertisement is generated from the advertisement recommendation data set. If the recommendation coefficient Tjs is lower than the recommendation threshold, the current live advertisement recommendation algorithm is optimized. According to the optimized advertisement recommendation algorithm, the corresponding target advertisement is matched from the advertisement program database, and the target advertisement is implanted in the current live video.

9. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 8, characterized in that: Perform linear normalization on the correlation coefficient Rt and the non-overlapping duration Ct, map the corresponding data values ​​to the interval [0,1], and generate the recommendation coefficient Tjs according to the following formula: Among them, α is the weight coefficient, Rt i is the correlation coefficient between the i-th recommended advertisement and the video scene, Rt p is the acceptable value of the correlation coefficient, Ct i is the non-overlapping duration between the i-th recommended advertisement and the corresponding video scene, Ct p is an acceptable value for non-overlapping durations.

10. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 9, characterized in that: After several consecutive observation cycles, advertisement feedback data is obtained in each observation cycle, feedback coefficients Fxs in the corresponding observation cycle are generated from the advertisement feedback data, and each observation cycle is marked with the feedback coefficients Fxs; The improvement degree Imp is generated by the feedback coefficient Fxs. If the improvement degree mp is lower than the improvement threshold, an implantation frequency constraint instruction is issued to the outside.

11. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 10, characterized in that: The observation periods before and after the optimization of the advertising recommendation algorithm are sorted according to the time axis and correspond one to one, and the improvement degree Imp is generated by the feedback coefficient Fxs in the following way: Among them, No i is the median value of improvement in the ith observation period, No a is its mean, No p is its qualified standard value; i = 1, 2, ..., k, k is the number of observation cycles, and are the feedback coefficients of the ith observation period before and after maintenance, and is the corresponding mean.

12. The method for dynamically inserting advertisements in a video based on image recognition as claimed in claim 11, characterized in that: After receiving the frequency restriction instruction, the frequency of advertisement insertion in the current stage is restricted. The restriction conditions are as follows: Where n is the number of nodes where advertisements are implanted, Z ij is the time interval from the i-th advertisement placement to the j-th advertisement placement, Z a is the average value of the time interval, Imp is the improvement; The advertisement insertion time node is reselected according to the constrained insertion frequency and used as the insertion node, and the advertisement is inserted into the live video at the selected insertion node.

13. A system for dynamically inserting advertisements in videos based on image recognition, characterized in that: include, A feedback data collection unit collects advertisement feedback data of the live video and generates an advertisement feedback data set, generates a feedback coefficient Fxs from the advertisement feedback data, and issues a prompt instruction to the outside if the feedback coefficient Fxs does not exceed a feedback threshold; The video recognition unit collects the current live video and extracts key frames after preprocessing the video. The extracted key frames are used as input, and the trained scene recognition model identifies the scene type in the current live video and outputs the identified video scene and related information. A region segmentation unit performs edge detection on a video frame in a live video broadcast, and divides the video frame into a number of segmented regions based on the edge detection result, scores the importance of each segmented region, selects a target region for advertisement implantation within the segmented region based on the importance score, and issues an implantation instruction to the outside; The recommendation effect evaluation unit uses the advertising recommendation algorithm to match the corresponding target advertisement according to the video scene and the characteristic information of the audience, and implants the target advertisement into the target area. The recommendation coefficient Tjs after the implanted advertisement is generated from the advertising recommendation data. If the recommendation coefficient Tjs is lower than expected, the target advertisement is re-matched and implanted. The improvement analysis unit obtains advertisement feedback data in each observation period and generates an improvement degree Imp. If the improvement degree Imp is lower than an improvement threshold, the advertisement placement frequency in the current stage is restricted.

14. A device for dynamically inserting advertisements in a video based on image recognition, characterized in that: include, at least one processor; A memory for storing executable instructions, wherein the instructions are executed by at least one of the processors to enable at least one of the processors to perform the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program, characterized in that: The computer program is used by a processor to execute the steps of the method according to any one of claims 1 to 12.

Images