Video advertisement embedding method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for video advertisement implantation, wherein the method for video advertisement implantation comprises: obtaining corresponding video frames according to a video stream to be played; selecting a target area for implanting advertisements from the video frames; implanting the advertisement frames corresponding to the advertisements to be implanted into the target area of ​​the video frames according to the playback order, and obtaining video frames after the advertisements are implanted; and obtaining a video stream containing advertisements according to the video frames after the advertisements are implanted. The above scheme can improve the exposure rate and acceptance of advertisements.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device, electronic device and storage medium for embedding advertisements in a video. Background Art

[0002] As the demand for advertising business continues to expand, inserting ads into videos has gradually become a common way of advertising. Traditional video advertising methods include inserting ads before the video starts, inserting ads on the video screen, inserting ads when the video is paused, etc. These advertising methods that force users' attention will frequently load ads during video playback, which will inevitably affect the user's viewing experience and easily cause users to resist or even feel disgusted. In order to find a good dynamic balance between user experience and the number of video ad exposures, video advertising urgently needs a "soft" ad embedding method to replace these traditional "hard" embedding methods. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a method, device, electronic device and storage medium for video advertising embedding, which can improve the exposure and acceptance of advertising.

[0004] In order to solve the above problems, the first aspect of the present application provides a method for embedding advertisements in a video, comprising: obtaining corresponding video frames according to a video stream to be played; selecting a target area for embedding advertisements from the video frames; embedding advertisement frames corresponding to the advertisements to be embedded into the target area of ​​the video frames according to the playback order to obtain video frames after the advertisements are embedded; and obtaining a video stream containing the advertisements according to the video frames after the advertisements are embedded.

[0005] Among them, the selecting of the target area for embedding advertisements from the video frame includes: classifying all video frames and selecting video frames belonging to a preset category as target video frames; detecting key points of the target area from the target video frames; aligning the key points of the target area of ​​the current target video frame to find the points around the key points of the target area of ​​the current target video frame that best match the key points of the target area of ​​the previous target video frame as the aligned key points; and determining the target area based on all the aligned key points of the current target video frame.

[0006] The step of detecting the key points of the target area from the target video frame includes: detecting a target object having a planar area from the target video frame; and detecting a plurality of key points from the target object as the key points of the target area.

[0007] Among them, detecting a number of key points from the target object as the key points of the target area includes: using a neural network model to perform feature calculations on the target object, obtaining a number of key points of the target object, and outputting them in a preset order to obtain the key points of the target area.

[0008] Among them, the key points of the target area of ​​the current target video frame are registered to find the points that best match the key points of the target area of ​​the previous target video frame around the key points of the target area of ​​the current target video frame as the registered key points, including: affine transforming the first image area surrounded by the key points of the target area of ​​the previous target video frame into a first fixed image area of ​​a preset size according to a first affine transformation matrix; affine transforming the second image area surrounded by the key points of the target area of ​​the current target video frame into a second fixed image area of ​​a preset size according to a second affine transformation matrix; optimizing the second affine transformation matrix using the structural similarity of the first fixed image area and the second fixed image area as an optimization function to obtain an optimized second affine transformation matrix; and obtaining the registered key points of the target area of ​​the current target video frame through matrix inverse operation based on the optimized second affine transformation matrix and the second fixed image area.

[0009] Among them, after obtaining the registered key points of the target area of ​​the current frame target video frame through matrix inverse operation based on the optimized second affine transformation matrix and the second fixed image area, it also includes: smoothing the registered key points of the target area of ​​the current frame target video frame according to the key points of the target area of ​​the previous frame target video frame.

[0010] Among them, the method of implanting the advertisement frames corresponding to the advertisements to be implanted into the target area of ​​the video frame in the playback order to obtain the video frame after the advertisements are implanted includes: projecting each advertisement frame to the target area of ​​the corresponding target video frame through affine transformation to obtain the advertisement frame after affine transformation; performing style fusion on the advertisement frame after affine transformation according to the picture of the target video frame to obtain the advertisement frame after style fusion; replacing the target area of ​​the corresponding target video frame with the advertisement frame after style fusion to obtain the video frame after the advertisement is implanted.

[0011] Among them, the affine transformation is used to project each advertisement frame to the target area of ​​the corresponding target video frame to obtain the advertisement frame after affine transformation, including: calculating the corresponding third affine transformation matrix according to the aligned key points of the target area of ​​the target video frame and the boundary points of the corresponding advertisement frame; according to the third affine transformation matrix, projecting the corresponding advertisement frame to the target area surrounded by the aligned key points of the corresponding target video frame to obtain the advertisement frame after affine transformation.

[0012] The advertisement to be embedded is a dynamic advertisement.

[0013] In order to solve the above-mentioned problem, the second aspect of the present application provides a video advertising implantation device, and the video advertising implantation device comprises: a video decoding module, the video decoding module is used to obtain corresponding video frames according to the video stream to be played; a target area acquisition module, the target area acquisition module is used to select a target area for implanting advertisements from the video frame; an advertising implantation module, the advertising implantation module is used to implant the advertising frames corresponding to the advertisements to be implanted into the target area of ​​the video frame according to the playback order, and obtain the video frame after the advertisements are implanted; a video encoding module, the video encoding module is used to obtain a video stream containing the advertisement according to the video frame after the advertisements are implanted.

[0014] In order to solve the above problem, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the video advertising embedding method of the first aspect.

[0015] In order to solve the above-mentioned problem, the fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the method for embedding advertisements in a video according to the first aspect is implemented.

[0016] The beneficial effects of the present invention are as follows: different from the prior art, the video advertisement embedding method of the present application obtains corresponding video frames according to the video stream to be played, selects the target area for embedding advertisements from the video frames, and then embeds the advertisement frames corresponding to the advertisements to be embedded into the target area of ​​the video frames according to the playback order, thereby obtaining the video frames after the advertisements are embedded; therefore, a video stream containing the advertisements can be obtained according to the video frames after the advertisements are embedded, and each frame content of the advertisement can be stably embedded in a relatively fixed picture area in the video frame in sequence, so that the advertisement becomes a meaningful part of the original video, thereby reducing the resistance in the user experience, and greatly improving the exposure and acceptance of the advertisements without affecting the user's perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a flowchart of an embodiment of a method for embedding advertisements in a video of the present application;

[0018] Figure 2 It is a flowchart of an application scenario of the advertising embedding method of the video of the present application;

[0019] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S12;

[0020] Figure 4 yes Figure 3 A schematic diagram of a flow chart of an embodiment of step S122;

[0021] Figure 5 It is a schematic diagram showing key points of a target object in an application scenario;

[0022] Figure 6 yes Figure 3 A schematic diagram of a flow chart of an embodiment of step S123;

[0023] Figure 7 This is a schematic diagram of an affine transformation of an application scenario;

[0024] Figure 8 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;

[0025] Fig. 9 yes Figure 8 A schematic diagram of a flow chart of an embodiment of step S131;

[0026] Fig.10 It is a schematic diagram of the framework of an embodiment of a device for embedding advertisements in a video of the present application;

[0027] Fig.11 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0028] Fig.12 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0029] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0030] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0031] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0032] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for embedding advertisements in a video of the present application. Specifically, the method for embedding advertisements in a video of the present application may include the following steps:

[0033] Step S11: Obtain corresponding video frames according to the video stream to be played.

[0034] In this application, the video stream to be played can be played in the form of online on-demand, local video playback or live broadcast. The storage location and transmission method of the video stream are not limited. For example, the storage location can be a local hard disk, a mobile storage device, a network server, etc. of the video playback terminal; the transmission method can be network transmission, data line transmission, Bluetooth wireless transmission, etc. The decoding method of the video stream is not limited. By decoding the video stream, the corresponding multiple video frames can be obtained, and the video frames can be played in the player.

[0035] Step S12: selecting a target area for inserting advertisement from the video frame.

[0036] It is understandable that in order to embed advertisements in a video, the advertisement content needs to be embedded in the video frame and displayed as the video frame is played. Therefore, after obtaining the corresponding video frame, a target area for embedding advertisements can be selected from the video frame. The shape of the target area can be determined according to the needs of the advertisement content, such as a rectangle, trapezoid, quadrilateral, irregular trapezoid, circle, sector, arch, etc.

[0037] Step S13: The advertisement frame corresponding to the advertisement to be embedded is embedded into the target area of ​​the video frame according to the playback order, so as to obtain the video frame after the advertisement is embedded.

[0038] Since the video stream corresponds to multiple video frames, all video frames have a certain playback order when playing. Therefore, embedding advertisements into videos actually embeds advertisement frames into the target area of ​​video frames. Specifically, advertisements to be embedded also correspond to advertisement frames, and all advertisement frames also have a certain playback order. Therefore, it is necessary to select a corresponding number of video frames according to the playback order of advertisement frames, and embed each advertisement frame into the target area of ​​video frames. For example, if there are n advertisement frames, the n advertisement frames are advertisement frames P1, advertisement frames P2... advertisement frames Pn-1, advertisement frames Pn in the order of advertisement playback, and a corresponding number of n video frames are selected, and the n video frames are video frames Q1, video frames Q2... video frames Qn-1, video frames Qn in the order of video playback; therefore, advertisement frame P1 is correspondingly embedded into the target area of ​​video frame Q1, advertisement frame P2 is correspondingly embedded into the target area of ​​video frame Q2... advertisement frame Pn is correspondingly embedded into the target area of ​​video frame Qn, thereby obtaining n video frames after the advertisements are embedded.

[0039] In one embodiment, the advertisement to be embedded is a dynamic advertisement, for example, the format of the advertisement to be embedded can be a video format or a gif format, so the multiple advertisement frames contained therein can be extracted in sequence. In other embodiments, the advertisement to be embedded can also be a static advertisement, for example, the format of the advertisement to be embedded is a static image, in which case the static image can be directly used as the advertisement frame, and all advertisement frames are the static image, which is equivalent to repeatedly playing the static image multiple times.

[0040] Step S14: obtaining a video stream containing advertisements according to the video frames after the advertisements are embedded.

[0041] After all the advertising frames corresponding to the advertisements to be embedded are implanted into the target area of ​​the video frame according to the playback order, several video frames with the advertisements embedded can be obtained. The obtained video frames with the advertisements embedded are encoded together with other video frames without the advertisements embedded in the playback order to obtain a video stream containing the advertisements. When the video stream containing the advertisements is played, the advertisement content can be displayed in the original video to be played, thereby realizing the implantation of advertisements.

[0042] Please combine Figure 2 , Figure 2It is a flow chart of an application scenario of the method for embedding advertisements in videos of the present application. In an application scenario, by decoding a video stream, a plurality of corresponding video frames can be obtained; then artificial intelligence can be used to automatically analyze and process the advertisements to be embedded and the video frames, and the advertisement content is encoded into the video stream as part of the content of the video frame. Specifically, artificial intelligence is used to select a target area for embedding advertisements from the video frame, and then the advertisement frame corresponding to the advertisement to be embedded is embedded into the target area of ​​the video frame in the playback order to obtain the video frame after the advertisement is embedded; then the obtained video frame after the advertisement is embedded is encoded together with other video frames without the advertisement embedded in the playback order to obtain a video stream containing the advertisement, which is then broadcast and distributed to the client, and the client data is decoded on the client side, so that the video screen containing the advertisement can be played.

[0043] The above scheme obtains the corresponding video frame according to the video stream to be played, selects the target area for embedding advertisements from the video frame, and then embeds the advertisement frame corresponding to the advertisement to be embedded into the target area of ​​the video frame according to the playback order, so as to obtain the video frame after the advertisement is embedded. Therefore, a video stream containing the advertisement can be obtained according to the video frame after the advertisement is embedded, so that each frame content of the advertisement can be stably embedded in a relatively fixed picture area in the video frame in sequence, so that the advertisement becomes a meaningful part of the original video, thereby reducing the resistance in the user experience and greatly improving the exposure and acceptance of the advertisement without affecting the user's perception.

[0044] Please combine Figure 3 , Figure 3 yes Figure 1 FIG. 1 is a flow chart of an embodiment of step S12 in FIG. 1. In one embodiment, the above step S12 may specifically include:

[0045] Step S121: classify all video frames, and select video frames belonging to a preset category as target video frames.

[0046] It is understandable that in a video, there may be multiple scenes, and for the advertisements to be embedded, it may only be suitable to be embedded in certain specific scenes. Therefore, after obtaining all the video frames corresponding to the video stream, all the video frames need to be classified to select the video frames belonging to the preset category as the target video frames, so as to facilitate the implantation of the advertising content into the target video frames. For example, for live broadcasts of games or competition videos, the entire video is not entirely about the game screen. Therefore, a convolutional neural network can be used to classify the decoded video frames into game screens and non-game screens. The backbone of the convolutional neural network model used is MobileNetV2, and the number of categories is 2, so that the video frames belonging to the game screen can be selected as the target video frames, so that the advertising content can be implanted into the game screen.

[0047] Step S122: Detecting key points of the target area from the target video frame.

[0048] After the target video frame is selected, key points that enclose the target area can be detected from the target video frame, and then the target area can be determined.

[0049] Furthermore, please combine Figure 4 , Figure 4 yes Figure 3 In the flowchart of an embodiment of step S122, in one embodiment, the above step S122 may specifically include:

[0050] Step S1221: Detect a target object having a planar area from the target video frame.

[0051] Step S1222: Detect a plurality of key points from the target object as key points of the target area.

[0052] This application uses a deep learning algorithm to identify the target object in the target video frame, such as the SSD solution based on MobileNetV2, or the Faster-RCCN series, YOLO, etc. The detected target object needs to have a planar area, such as a billboard, vehicle, desktop, floor, wall, or game element. After the target object is detected, several key points can be detected from the target object, and these key points can be used as key points to enclose the target area.

[0053] In one embodiment, the above step S1222 may specifically include: using a neural network model to calculate features of the target object, obtaining a number of key points of the target object, and outputting them in a preset order to obtain the key points of the target area.

[0054] It is understandable that the accuracy and effectiveness of key point detection is an important prerequisite for determining the target area. Specifically, a neural network model can be used to calculate the characteristics of the target object. The neural network model can be a convolutional neural network model to obtain several key points of the target object. Figure 5 As shown, Figure 5 This is a schematic diagram showing the key points of a target object in an application scenario. The neural network model is the HRNet model. In this case, the target object is a flag in the League of Legends game map. After the flag is detected, the four key points of the flag are obtained. For the flag, the four corners of the flag can be used as the key points to be detected, and the key points are output in the order of upper left, upper right, lower right, and lower left according to the position and viewing angle of the plane. For the output of the four key points, the HRNet model can predict four heat maps, and find the maximum corresponding position of each heat map, which respectively represents the order and position of the four points.

[0055] In the above-mentioned scheme for obtaining the key points that enclose the target area, a two-step scheme is adopted, which first detects the target object and then detects the key points of the target object containing the plane area. Compared with the one-step scheme of directly detecting the key points in the video frame, it can reduce excessive background interference, thereby achieving accurate detection of the key points of small targets.

[0056] Step S123: align the key points of the target area of ​​the current target video frame to find the points around the key points of the target area of ​​the current target video frame that best match the key points of the target area of ​​the previous target video frame as the aligned key points.

[0057] Step S124: determining the target area according to all registered key points of the current target video frame.

[0058] It is understandable that due to the jitter in the output of the neural network model, the key points of the target area obtained from the two previous and next video frames will jitter. Therefore, for the process of determining the target area, how to ensure the relative stability of the key points detected between frames is the most core issue in the visual experience of the entire solution. In order to solve the problem of jitter in key points between frames, the present application can adopt a point alignment method, using the key points of the previous frame as a reference, and searching around the key points detected in the current frame to find the position that best matches the point corresponding to the previous frame, to ensure the visual effect of the entire advertising implantation solution.

[0059] Furthermore, please combine Figure 6 , Figure 6 yes Figure 3 In the flowchart of an embodiment of step S123, in one embodiment, the above step S123 may specifically include:

[0060] Step S1231: Affine transform a first image region enclosed by key points of the target region of a previous target video frame into a first fixed image region of a preset size according to a first affine transformation matrix.

[0061] Step S1232: Affine transform the second image area enclosed by the key points of the target area of ​​the current target video frame into a second fixed image area of ​​a preset size according to the second affine transformation matrix.

[0062] Step S1233: Taking the structural similarity between the first fixed image region and the second fixed image region as an optimization function, the second affine transformation matrix is ​​optimized to obtain an optimized second affine transformation matrix.

[0063] Step S1234: Based on the optimized second affine transformation matrix and the second fixed image area, the registered key points of the target area of ​​the current target video frame are obtained through matrix inverse operation.

[0064] It is understandable that if Figure 7 As shown, Figure 7This is a schematic diagram of affine transformation of an application scenario. Taking the number of key points as 4 as an example, since the detected 4 key points do not necessarily form an upright rectangle (in most cases they are irregular quadrilaterals, and the lines are not horizontal or vertical), it is necessary to affine transform the first image area surrounded by the 4 key points of the target area detected in the previous frame of the target video frame into a first fixed image area of ​​a preset size according to the first affine transformation matrix M0. The first fixed image area is an upright rectangular ROI area with a preset size of: length x width = 128x128 pixels. Therefore, the first affine transformation matrix M0 can be calculated according to the target positions (0,0), (127,0), (127,127), (0,127) and the detected key points (X0,Y0), (X1,Y1), (X2,Y2), (X3,Y3). Similarly, according to the second affine transformation matrix M1, the second image area surrounded by the four key points of the target area detected in the current frame target video frame is affine transformed into a second fixed image area of ​​preset size. The second fixed image area is a rectangular ROI area placed upright, and the preset size is: length x width = 128x128 pixels. Therefore, the second affine transformation matrix M1 can also be obtained according to a similar method. Then, the structural similarity (SSIM) between the first fixed image area and the second fixed image area is used as the optimization function to optimize the second affine transformation matrix M1, and the Nelder-Mead numerical iteration method is used for iterative calculation to ensure that the SSIM is maximized, thereby obtaining the optimized second affine transformation matrix M1'. Therefore, based on the optimized second affine transformation matrix M1' and the target positions (0,0), (127,0), (127,127), (0,127) of the second fixed image area, the aligned key points surrounding the target area of ​​the current frame target video frame can be obtained through matrix inverse operation.

[0065] Step S1235: performing smoothing on the registered key points of the target area of ​​the current target video frame according to the key points of the target area of ​​the previous target video frame.

[0066] Specifically, the following formula can be used to perform exponential smoothing on the registered key points of the target area of ​​the current target video frame and the key points of the target area of ​​the previous target video frame to further ensure the stability of the key points. The key points kp_new after smoothing of the current target video frame satisfy the following formula:

[0067] Kp_new=(1-m)*kp_cur+m*kp_pre;

[0068] in, diff=|kp_cur-kp_pre|, kp_cur represents the registered key points of the target area of ​​the current target video frame, kp_pre represents the key points of the target area of ​​the previous target video frame, where α is a constant, for example, can be set to 5.

[0069] Please combine Figure 8 , Figure 8 yes Figure 1 FIG. 1 is a flow chart of an embodiment of step S13 in FIG. 1. In one embodiment, the above step S13 may specifically include:

[0070] Step S131: projecting each advertisement frame to the target area of ​​the corresponding target video frame through affine transformation to obtain an advertisement frame after affine transformation.

[0071] After obtaining the target area of ​​the target video frame, there may be a situation where the size and shape of the content area of ​​the advertisement frame do not match the target area of ​​the target video frame. Therefore, it is necessary to project each advertisement frame to the target area of ​​the corresponding target video frame through affine transformation to obtain the advertisement frame after affine transformation. For example, if the content area of ​​the advertisement frame is a rectangle, it is necessary to project the rectangular advertisement area to the quadrilateral target area enclosed by the key point.

[0072] Furthermore, please combine Fig. 9 , Fig. 9 yes Figure 8 In the flowchart of an embodiment of step S131, in one embodiment, the above step S131 may specifically include:

[0073] Step S1311: Calculate and obtain a corresponding third affine transformation matrix according to the registered key points of the target area of ​​the target video frame and the boundary points of the corresponding advertisement frame.

[0074] Step S1312: According to the third affine transformation matrix, the corresponding advertisement frame is projected to the target area surrounded by the aligned key points of the corresponding target video frame to obtain the advertisement frame after affine transformation.

[0075] Specifically, taking the advertisement frame as a rectangle as an example, the boundary points of the advertisement frame include four vertices. The advertisement frame can be projected to the target area of ​​the target video frame using the third affine transformation matrix. Therefore, the four vertices of the advertisement frame correspond to the four registered key points projected to the target area. Therefore, according to the registered key points of the target area of ​​the target video frame and the corresponding boundary points of the advertisement frame, the corresponding third affine transformation matrix can be calculated. For example, [X, Y, Z] represents the coordinates of the registered key points of the target area of ​​the target video frame, and [x, y, 1] represents the vertex coordinates of the advertisement frame, satisfying the equation:

[0076]

[0077] Among them, the third affine transformation matrix For the two-dimensional case, Z=1, a33=1, so, according to the coordinates of the four registered key points of the target area of ​​the target video frame and the coordinates of the four vertices of the advertisement frame, the eight unknowns in the third affine transformation matrix A can be solved. Therefore, according to the third affine transformation matrix A, based on a mature affine transformation algorithm, such as the warPerspective() function in the third-party tool OpenCV (a cross-platform computer vision and machine learning software library released under the BSD license (open source)), the corresponding advertisement frame can be projected to the target area surrounded by the registered key points of the corresponding target video frame to obtain the advertisement frame after affine transformation.

[0078] Step S132: performing style fusion on the advertisement frame after affine transformation according to the picture of the target video frame to obtain an advertisement frame after style fusion.

[0079] Step S133: replacing the target area of ​​the corresponding target video frame with the advertisement frame after style fusion to obtain a video frame after the advertisement is embedded.

[0080] It is understandable that after obtaining the affine transformed advertising frame, the affine transformed advertising frame can be implanted into the target video frame. However, the color style of the advertising content may be quite different from the color style of the video. If it is directly implanted, it will give the user a sense of disharmony. Therefore, it is necessary to perform style fusion on the affine transformed advertising frame according to the picture of the target video frame. For example, a mature and reliable Poisson Fusion algorithm can be used to obtain a style-fused advertising frame to ensure that the color style of the implanted advertisement is consistent with the video picture. Then, the style-fused advertising frame is used to replace the target area of ​​the target video frame obtained by the previous detection, so as to form a video frame after the implantation of the advertisement. In the obtained video frame after the implantation of the advertisement, the advertisement is integrated with the video content in style and becomes a meaningful part of the original video, thereby reducing the resistance in the user experience.

[0081] The video advertisement embedding method of the present application can realize batch automatic analysis of massive videos, and the video advertisement delivery efficiency is high; and different advertisement contents can be embedded in the same video according to the demand of advertisement delivery, and the reuse rate is high; in addition, the overall solution processing time is no more than 30ms, which can ensure real-time processing; and it supports the embedding of picture advertisements and dynamic video advertisements, and is not limited by the advertisement content, and the delivery content is flexible.

[0082] See also Fig.10 , Fig.10It is a schematic diagram of the framework of an embodiment of the video advertisement implantation device of the present application. The video advertisement implantation device 100 comprises: a video decoding module 1000, the video decoding module 1000 is used to obtain the corresponding video frame according to the video stream to be played; a target area acquisition module 1002, the target area acquisition module 1002 is used to select the target area for implanting advertisement from the video frame; an advertisement implantation module 1004, the advertisement implantation module 1004 is used to implant the advertisement frame corresponding to the advertisement to be implanted into the target area of ​​the video frame according to the playback order, and obtain the video frame after the advertisement is implanted; a video encoding module 1006, the video encoding module 1006 is used to obtain the video stream containing the advertisement according to the video frame after the advertisement is implanted.

[0083] In some embodiments, the target area acquisition module 1002 executes the step of selecting a target area for advertising embedding from the video frame, including: classifying all video frames and selecting video frames belonging to a preset category as target video frames; detecting key points of the target area from the target video frames; aligning the key points of the target area of ​​the current target video frame to find the points around the key points of the target area of ​​the current target video frame that best match the key points of the target area of ​​the previous target video frame as the aligned key points; and determining the target area based on all the aligned key points of the current target video frame.

[0084] In some embodiments, the target area acquisition module 1002 performs detection to obtain key points of the target area from the target video frame, including: detecting a target object with a planar area from the target video frame; detecting a plurality of key points from the target object as key points of the target area.

[0085] In some embodiments, the target area acquisition module 1002 executes detection of a number of key points from the target object as the key points of the target area, including: using a neural network model to perform feature calculations on the target object, obtaining a number of key points of the target object, and outputting them in a preset order to obtain the key points of the target area.

[0086] In some embodiments, the target area acquisition module 1002 performs registration of the key points of the target area of ​​the current frame target video frame to find the points around the key points of the target area of ​​the current frame target video frame that best match the key points of the target area of ​​the previous frame target video frame as the registered key points, including: affine transforming the first image area surrounded by the key points of the target area of ​​the previous frame target video frame to a first fixed image area of ​​a preset size according to a first affine transformation matrix; affine transforming the second image area surrounded by the key points of the target area of ​​the current frame target video frame to a second fixed image area of ​​a preset size according to a second affine transformation matrix; optimizing the second affine transformation matrix using the structural similarity of the first fixed image area and the second fixed image area as an optimization function to obtain an optimized second affine transformation matrix; and obtaining the registered key points of the target area of ​​the current frame target video frame through matrix inverse operation based on the optimized second affine transformation matrix and the second fixed image area.

[0087] In some embodiments, after the target area acquisition module 1002 executes the step of obtaining the registered key points of the target area of ​​the current frame target video frame through matrix inverse operation based on the optimized second affine transformation matrix and the second fixed image area, it also includes: smoothing the registered key points of the target area of ​​the current frame target video frame according to the key points of the target area of ​​the previous frame target video frame.

[0088] In some embodiments, the advertisement implantation module 1004 executes the steps of implanting the advertisement frames corresponding to the advertisements to be implanted into the target area of ​​the video frame in the playback order to obtain the video frames after the advertisements are implanted, including: projecting each advertisement frame to the target area of ​​the corresponding target video frame through affine transformation to obtain the advertisement frames after affine transformation; performing style fusion on the advertisement frames after affine transformation according to the picture of the target video frame to obtain the advertisement frames after style fusion; replacing the target area of ​​the corresponding target video frame with the advertisement frames after style fusion to obtain the video frames after the advertisements are implanted.

[0089] In some embodiments, the advertising implantation module 1004 performs an affine transformation to project each advertising frame onto the target area of ​​the corresponding target video frame to obtain an affine transformed advertising frame, including: calculating a corresponding third affine transformation matrix based on the aligned key points of the target area of ​​the target video frame and the boundary points of the corresponding advertising frame; and projecting the corresponding advertising frame onto the target area surrounded by the aligned key points of the corresponding target video frame according to the third affine transformation matrix to obtain an affine transformed advertising frame.

[0090] See also Fig.11 , Fig.11 1 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 110 includes a memory 111 and a processor 112 coupled to each other, and the processor 112 is used to execute program instructions stored in the memory 111 to implement the steps of any of the above-mentioned video advertisement implantation method embodiments. In a specific implementation scenario, the electronic device 110 may include but is not limited to: a microcomputer, a server.

[0091] Specifically, the processor 112 is used to control itself and the memory 111 to implement the steps of any of the above-mentioned video advertising implantation method embodiments. The processor 112 can also be called a CPU (Central Processing Unit). The processor 112 may be an integrated circuit chip with signal processing capabilities. The processor 112 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 112 can be implemented by an integrated circuit chip.

[0092] See also Fig.12 , Fig.12 The computer-readable storage medium 120 stores program instructions 1200 that can be executed by a processor, and the program instructions 1200 are used to implement the steps of any of the above-mentioned video advertisement embedding method embodiments.

[0093] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for embedding advertisements in a video, characterized in that: include: According to the video stream to be played, obtain the corresponding video frame; Selecting a target area for inserting advertisements from the video frame; specifically comprising: classifying all video frames, selecting video frames belonging to a preset category as target video frames; detecting key points of the target area from the target video frames; registering the key points of the target area of ​​the current target video frame, so as to find the points around the key points of the target area of ​​the current target video frame that best match the key points of the target area of ​​the previous target video frame as the registered key points; determining the target area according to all the registered key points of the current target video frame; The advertisement frame corresponding to the advertisement to be embedded is embedded into the target area of ​​the video frame according to the playback order to obtain the video frame after the advertisement is embedded; A video stream containing advertisements is obtained according to the video frames after the advertisements are embedded.

2. The method for embedding advertisements in a video according to claim 1, characterized in that: The detecting and obtaining the key points of the target area from the target video frame includes: Detecting a target object having a planar area from the target video frame; A plurality of key points are detected from the target object as key points of the target area.

3. The method for embedding advertisements in a video according to claim 2, characterized in that: The detecting of a plurality of key points from the target object as key points of the target area includes: A neural network model is used to calculate the features of the target object, obtain a number of key points of the target object, and output them in a preset order to obtain the key points of the target area.

4. The method for embedding advertisements in a video according to claim 1, characterized in that: The key points of the target area of ​​the current target video frame are registered to find the points around the key points of the target area of ​​the current target video frame that best match the key points of the target area of ​​the previous target video frame as the registered key points, including: Affine transforming a first image area surrounded by key points of the target area of ​​a previous target video frame into a first fixed image area of ​​a preset size according to a first affine transformation matrix; Affine transforming a second image area surrounded by key points of the target area of ​​the current target video frame into a second fixed image area of ​​a preset size according to a second affine transformation matrix; Taking the structural similarity between the first fixed image region and the second fixed image region as an optimization function, optimizing the second affine transformation matrix to obtain an optimized second affine transformation matrix; Based on the optimized second affine transformation matrix and the second fixed image area, the registered key points of the target area of ​​the current target video frame are obtained through matrix inverse operation.

5. The method for embedding advertisements in a video according to claim 4, characterized in that: After obtaining the registered key points of the target area of ​​the current target video frame by matrix inverse operation based on the optimized second affine transformation matrix and the second fixed image area, the method further includes: According to the key points of the target area of ​​the previous target video frame, the registered key points of the target area of ​​the current target video frame are smoothed.

6. The method for embedding advertisements in a video according to claim 1, characterized in that: The step of embedding the advertisement frame corresponding to the advertisement to be embedded into the target area of ​​the video frame according to the playback order to obtain the video frame after the advertisement is embedded includes: Projecting each advertisement frame to the target area of ​​the corresponding target video frame through affine transformation to obtain an advertisement frame after affine transformation; Performing style fusion on the advertisement frame after affine transformation according to the picture of the target video frame to obtain the advertisement frame after style fusion; The target region of the corresponding target video frame is replaced with the advertisement frame after the style fusion to obtain a video frame after the advertisement is embedded.

7. The method for embedding advertisements in a video according to claim 6, characterized in that: The step of projecting each advertisement frame to the target area of ​​the corresponding target video frame through affine transformation to obtain the advertisement frame after affine transformation includes: Calculate a corresponding third affine transformation matrix according to the registered key points of the target area of ​​the target video frame and the boundary points of the corresponding advertisement frame; According to the third affine transformation matrix, the corresponding advertisement frame is projected onto the target area surrounded by the aligned key points of the corresponding target video frame to obtain the advertisement frame after affine transformation.

8. The method for embedding advertisements in a video according to claim 1, characterized in that: The advertisement to be embedded is a dynamic advertisement.

9. A device for embedding advertisements in a video, characterized in that: The video advertisement embedding device comprises: A video decoding module, the video decoding module is used to obtain corresponding video frames according to the video stream to be played; A target area acquisition module, the target area acquisition module is used to select a target area for inserting advertisements from the video frame; specifically includes: classifying all video frames, selecting video frames belonging to a preset category as target video frames; detecting key points of the target area from the target video frames; registering the key points of the target area of ​​the current target video frame, so as to find the points around the key points of the target area of ​​the current target video frame that best match the key points of the target area of ​​the previous target video frame as the registered key points; determining the target area according to all the registered key points of the current target video frame; An advertisement embedding module, the advertisement embedding module is used to embed the advertisement frame corresponding to the advertisement to be embedded into the target area of ​​the video frame according to the playback order, so as to obtain the video frame after the advertisement is embedded; A video encoding module is used to obtain a video stream containing advertisements according to the video frames after the advertisements are embedded.

10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the video advertisement embedding method according to any one of claims 1 to 8.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method for embedding advertisements in a video as described in any one of claims 1 to 8 is implemented.

Images