A video advertisement intelligent insertion method, system and medium based on semantic segmentation and multi-modal fusion

Abstract

The application discloses a video advertisement intelligent insertion method and system based on semantic segmentation and multi-modal context fusion. Through pixel-level semantic mask and multi-frame time sequence analysis, the space gap area available for advertisement embedding in the video is automatically identified, and background stability is judged in combination with structural similarity and brightness change and other indexes. A multi-modal context modeling device is constructed, a context situation vector is generated through a cross attention mechanism by fusing a character emotion vector, a video rhythm vector and a scene structure vector, and an insertion suitability score is output. When the score reaches a set threshold and the gap area meets the conditions, the system performs advertisement generation and visual fusion operation, realizes natural advertisement insertion consistent with the original video content style. The method is suitable for various content platforms such as short videos and live broadcasts, and experiments show that the click rate and the complete play rate can be significantly improved, and the jump-out rate can be reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, computer vision and digital advertising integration, specifically a smart advertising insertion method based on video content understanding, applicable to online content scenarios such as live video and short video streaming. Background Technology

[0002] With the explosive growth of short video and live streaming platforms, traditional advertising formats (such as pre-roll ads and ads inserted before pause) are increasingly perceived by users as disruptive, leading to high skip and block rates and impacting ad conversion rates. Furthermore, current mainstream video ad insertion methods largely rely on fixed times or manual platform settings, failing to dynamically adjust insertion positions based on actual video content and lacking contextual awareness, which can easily cause user attention breaks or emotional aversion.

[0003] In recent years, visual recognition algorithms such as object detection (e.g., YOLO, DETR) and semantic segmentation (e.g., DeepLab, Mask R-CNN) have made significant breakthroughs, enabling frame-level understanding of videos. However, a systematic solution is still lacking for how to effectively combine these visual technologies with ad insertion logic to build a "natural and non-intrusive" ad placement mechanism. Summary of the Invention

[0004] This invention proposes a method for intelligent video content analysis and ad insertion. By performing object detection and semantic segmentation on video frames, it identifies people, objects, background elements, and semantic gaps. Combined with scene rhythm analysis, it automatically identifies the optimal insertion point and dynamically generates ad content that matches the scene, achieving immersive ad insertion. This overcomes at least one technical problem existing in related technologies.

[0005] According to a first aspect of the embodiments of this specification, a method for intelligent insertion of video advertisements based on semantic segmentation and multimodal fusion is provided, characterized by comprising the following steps:

[0006] S1. Perform frame extraction processing on the input video stream to obtain an image frame sequence;

[0007] S2. Use a semantic segmentation model to perform pixel-level recognition on the image frame, generate a semantic mask map, and identify non-critical semantic regions.

[0008] S3. Based on the pixel density of idle regions in consecutive frames, combined with temporal sliding window statistics and structural similarity ( ) and brightness changes ( Analysis to identify advertising gaps that satisfy spatial and temporal continuity;

[0009] S4. For the frames corresponding to the gap region, extract multimodal contextual features, including emotion vectors generated from facial expressions, video rhythm vectors, and image structure vectors.

[0010] S5. Input multimodal features into the fusion network to perform contextual modeling and generate an insertion suitability score;

[0011] S6. When the score exceeds the set threshold and the frame is a legal empty area, perform the ad generation and visual fusion operation to embed the ad into the corresponding frame.

[0012] Preferably, the semantic segmentation model is Mask R-CNN, and its output mask image is labeled with at least three types of semantic labels: people, products, and background, which are used to delineate non-critical regions.

[0013] Preferably, the gap region identification is based on a time sliding window of length T frames, and counts at least M frames that satisfy the interpolable region density threshold; where T=15 and M=10.

[0014] Preferably, the background stability determination requires that the structural similarity (SSIM) between consecutive frames be ≥0.90 and the RGB brightness change (ΔI) be ≤5%.

[0015] Preferably, the emotion vector is a seven-dimensional probability vector, corresponding to the recognition results of seven emotion categories.

[0016] Preferably, the multimodal fusion employs a cross-attention transformer structure, outputting a 25-dimensional contextual semantic vector for inserting score prediction.

[0017] Another aspect of the present invention is to provide a video ad insertion system for performing a video ad intelligent insertion method based on semantic segmentation and multimodal fusion, characterized in that it includes the following modules:

[0018] The video preprocessing and framing module is used to receive the raw video stream and perform frame extraction processing to generate a continuous sequence of image frames.

[0019] The object detection and semantic segmentation module is used to perform object recognition and semantic segmentation on each frame of the image and output a pixel-level mask image.

[0020] The gap point recognition module is used to analyze the pixel ratio of non-critical areas based on the mask image, and combine the time sliding window and background stability to identify the gap areas for advertisement insertion that meet the conditions.

[0021] The context modeling and insertion intent assessment module includes facial expression recognition, rhythm analysis, and structural modeling sub-modules, which are used to extract multimodal contextual features and fuse them to generate an insertion suitability score.

[0022] The dynamic ad generation and fusion module is used to select or generate ad content based on empty areas and context information, and to fuse ad images into video frames;

[0023] The real-time rendering and delivery control module is used to control the frequency of ad insertion, the presentation method, and the final rendering and output operations on the user's device.

[0024] Preferably, the gap point recognition module includes the following sub-modules:

[0025] The video input module is used to receive raw video data and parse metadata such as frame rate and resolution;

[0026] A frame extraction module is used to convert the video data into a sequence of image frames;

[0027] The semantic segmentation module is used to perform pixel-level semantic segmentation on each frame of the image based on the pre-trained Mask R-CNN model, and output a semantic mask map. );

[0028] The gap candidate mask generation module is used to identify non-critical semantic regions from the semantic mask map and generate a binary mask. );

[0029] The region area calculation module is used to calculate the binary mask ( The percentage of pixels in the interpolable region () );

[0030] The continuous frame window determination module is used to determine whether at least M frames satisfy a given time sliding window of length T. This allows for the determination of candidate regions for continuous gaps.

[0031] The background stability analysis module is used to calculate structural similarity in the empty region. and brightness change To determine the stability of background changes;

[0032] The gap area confirmation module is used to perform joint logical judgment on the results from the area calculation module, the continuous frame window determination module, and the background stability analysis module, and output the final advertisement can be inserted mask frame.

[0033] Preferably, the context modeling and insertion intent evaluation module includes:

[0034] The facial expression recognition module extracts facial image features based on a pre-trained model and outputs an emotion vector containing seven basic emotion confidence levels to characterize the current person's emotional state.

[0035] The dynamic rhythm analysis module is used to generate a multi-dimensional rhythm vector that reflects the rhythm characteristics of the picture based on indicators such as the target motion between image frames, the energy change of the audio signal, the speed of the character's movement, and the proportion of silence.

[0036] The scene structure modeling module is used to extract features such as texture complexity, sharpness, scene crowding, color information richness, and left-right brightness symmetry of images to characterize visual structure.

[0037] A multimodal fusion module, which uses a cross-attention mechanism to semantically fuse three types of vectors—emotion, rhythm, and structure—to obtain a comprehensive contextual representation of the current frame;

[0038] An insertion suitability score prediction module is provided. Based on the fused context representation vector, the insertion suitability score prediction module evaluates the contextual fit of the current time when the advertisement is inserted and outputs a score value representing the insertion suitability.

[0039] An insertion trigger control module is used to trigger an advertisement insertion action when the score value reaches a preset threshold and the current frame has been marked as an empty area.

[0040] Another aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when run in a processor, causes the computer to perform all the steps of a video advertising intelligent insertion method based on semantic segmentation and multimodal fusion.

[0041] In the embodiments described in this specification, by introducing an "insertion gap recognition mechanism" based on semantic segmentation, the system can perform pixel-level analysis of the spatial structure in the video frame, and realize the automatic recognition and labeling of "non-interference areas".

[0042] Compared to traditional methods that rely on a timeline or manually set insertion points, this mechanism offers two advantages: First, it enables fine-grained spatial determination. The system utilizes semantic segmentation networks such as Mask R-CNN to divide each frame into subject and background, and combines this with mask analysis algorithms to dynamically evaluate the region's idleness and visual stability, identifying the insertion space without interfering with the main visual focus. Second, it ensures continuous stability across multiple frames. This invention does not rely on instantaneous judgment of blank areas in a single frame, but instead introduces a cross-frame time window statistical mechanism to ensure the insertion area has temporal continuity (e.g., meeting the "non-interference" threshold in more than 10 out of 15 consecutive frames), effectively avoiding misjudgments caused by camera shake or sudden movements. This mechanism upgrades the ad embedding logic from "time-driven" to "spatial semantic-driven," not only improving the visual naturalness of ad fusion but also significantly reducing reliance on manual intervention, laying the foundation for automated and large-scale ad insertion.

[0043] Furthermore, by constructing a "multimodal context modeler," heterogeneous information such as user facial emotions, video rhythm features, and scene structure are fused and modeled to form a global perception of the current video content and an appropriateness judgment for insertion. The key advantage of this fusion modeling is that it overcomes the limitations of a single visual channel and achieves multi-dimensional context perception capabilities. Specifically, the emotion recognition module extracts emotion vectors by analyzing facial expressions to help determine whether the current character's state is suitable for intervention (e.g., avoiding inserting advertisements during tense or sad emotions); the rhythm analysis module uses techniques such as optical flow estimation and sound intensity curve analysis to determine the stability and fluctuation of the scene's rhythm, ensuring that the advertisement does not interrupt the climax of the action or dialogue; and the structure analysis module evaluates the object density and compositional balance in the scene to help select insertion areas outside the visual focus. By inputting the above features into a Transformer structure to achieve cross-modal semantic fusion, and combining this with a lightweight classifier to output an insertion appropriateness score, the system can make a decision on whether to insert an advertisement within milliseconds.

[0044] Experiments show that this modeling mechanism has significant advantages over traditional rule engines in terms of metrics such as ad click-through rate and user bounce rate. Especially in high-paced scenarios such as live streaming and short videos, it can maximize user immersion while balancing content quality and commercial conversion goals. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a functional structure diagram of the overall system units in the embodiment.

[0047] Figure 2 This is a functional structure diagram of the semantic gap recognition unit in Embodiment 1 of the present invention;

[0048] Figure 3 This is a functional structure diagram of the multimodal context modeling and insertion suitability prediction mechanism unit in Example 2. Detailed Implementation

[0049] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0051] Example 1: Figure 1 This paper demonstrates the complete system architecture of the "Intelligent Video Content Analysis and Ad Insertion Method" of this invention. The entire system receives live or short video stream data from the input end and sequentially completes content analysis, gap detection, context modeling, ad generation, final fusion, and delivery control through six main functional modules.

[0052] The system architecture modules are connected via standard data flow channels, and each module has a clearly defined function and independent processing capabilities. The specific structure and functions are as follows:

[0053] Video preprocessing and framing module 101:

[0054] This module receives raw short video or live stream data, performs format conversion, decoding, and frame extraction. The system supports multiple video encoding formats (such as H.264 / H.265) and can perform adaptive frame rate sampling as needed (such as 50fps, 30fps, etc.), outputting a continuous video frame sequence as input for subsequent analysis.

[0055] Module 102: Object Detection and Semantic Segmentation

[0056] This module performs visual recognition operations on each frame of the image, using YOLOv8 to detect the main targets (such as people, products, gestures, etc.) in the image, and calls the pre-trained Mask R-CNN model to perform pixel-level semantic segmentation of the image, outputting a semantic mask map. This mask image is used to subsequently determine insertable areas in the image.

[0057] Neutral point recognition module 103:

[0058] Using the semantic mask image output by segmentation module 102 The gap point recognition module 103 identifies potential ad insertion gaps (insertion positions) in the image by analyzing the pixel ratio of "non-critical semantic regions" in each frame. The system sets pixel thresholds, duration windows, and background stability judgment mechanisms to ensure that the ad insertion position is natural and unobtrusive.

[0059] Context modeling and insertion intent evaluation module 104:

[0060] Upon determining the existence of insertable gaps, this module constructs a "contextual understanding vector" for the video content. This vector includes three types of input:

[0061] The facial expression recognition module outputs the current emotional state of the person.

[0062] The motion rhythm extraction module analyzes the rhythm of the video (such as movement speed, changes in voice energy, etc.).

[0063] The scene structure modeling module outputs the complexity and distribution of attention points of the current scene.

[0064] These three elements are combined through a multimodal fusion network (such as a Transformer structure) to output a contextual vector, and an insertion intent evaluator outputs an ad insertion suitability score. Ad insertion is only triggered when the score is higher than a set threshold (such as 0.7).

[0065] Dynamic Ad Generation and Integration Module 105:

[0066] Based on the insertable positions and contextual vectors output by the gap point recognition module 103 and the context modeling and insertion intent evaluation module 104, the system selects matching materials from the advertising material library or calls generative AI (such as ControlNet, DALL·E, etc.) to generate advertising textures consistent with the current screen style in real time. Then, through technologies such as alpha fusion, affine transformation, and color adjustment, the advertising content is embedded into the image frame to achieve seamless visual transition.

[0067] Real-time rendering and deployment control module 106:

[0068] This module 106 controls the final embedding process of advertisements and the execution of the delivery strategy, and includes three parts: the rendering engine, the image write-back module, and the advertisement delivery controller. It determines the timing and frequency of advertisement insertion based on parameters such as the user platform, the resolution of the viewing device, and the historical click-through rate, and supports interactive advertisements (such as click-to-redirect) or floating advertisements.

[0069] System data flows sequentially from left to right. Each module can be deployed independently or integrated into the same GPU workstation or cloud platform container. Modules communicate with each other through standardized API interfaces, providing high scalability and flexible configuration capabilities.

[0070] Example 2: Semantic Segmentation-Based Insertion Gap Recognition Mechanism

[0071] To illustrate more specifically the first core inventive point of this invention, "semantic segmentation and insertion gap detection mechanism", this second embodiment is a further detailed description of the gap point identification module 103 in the first embodiment. The entire contents of the first embodiment are hereby cited.

[0072] A systematic modeling and implementation verification was conducted from five aspects: sampling data, semantic processing flow, formula for determining insertable regions, stability analysis, and multi-frame strategy. For example... Figure 2 The diagram illustrates the functional structure of the semantic gap recognition unit in this invention, with each process node corresponding to reference numerals 201 to 208. It includes the following processing steps and reference numerals:

[0073] I. Module Structure:

[0074] Video input module 201: This video input module 201 receives the raw video data; it supports multiple formats (such as MP4, FLV, RTMP, etc.) and parses its frame rate, resolution, and other metadata. Its output is a video data handle, which is passed as input to the frame extraction module 202.

[0075] Frame extraction module 202: This module converts video into a continuous frame sequence based on OpenCV. Specifically, it receives the video data handle output by the video input module 201, extracts a continuous frame image sequence using tools such as OpenCV at a specified frame rate (e.g., 50fps), and uniformly scales it to 1080p resolution. The output is an image frame sequence {It}, and each frame It is sent to module 203 (semantic segmentation module) for pixel-level semantic analysis.

[0076] Semantic segmentation module 203: This semantic segmentation module 203 is used to call the pre-trained Mask R-CNN model to output a pixel-level semantic mask Mt; specifically, it receives image frames I from the extraction module 202. t The pre-trained Mask R-CNN model is used to perform semantic segmentation on each frame and output a semantic mask image. , Each pixel in the mask is labeled with a corresponding semantic tag (such as a person, product, background, etc.). It is passed as input to the gap candidate mask generation module 204 (gap candidate mask generation module).

[0077] Gap Candidate Mask Generation Module 204: This module receives the semantic mask Mt output by module 203, and determines which pixels do not belong to F based on the preset "key semantic tag set" F (such as people, products, hands, etc.), marking them as insertable regions. Finally, a binary mask is generated. ; ( Image size is ),in This represents the area where ads can be inserted. Then input module 205 (area calculation module).

[0078] Area calculation module 205: This module receives the mask output by module 204. Count all pixels The total number, and calculate the percentage of that value in the entire frame. Its formula is:

[0079]

[0080] As the "interpolable region density" index of the frame, it is passed to module 206 (continuous frame window determination module) as one of the determination criteria.

[0081] Continuous Frame Window Determination Module 206: Module 206 receives the output from module 205. Value sequence, and construct a sequence of length. A time sliding window, for which... The frames are counted. If there are at least [number] frames within a certain window... If a frame meets this condition, then the time period is determined to be a "continuous gap candidate area". The result is output as a flag bit and passed to module 208 and module 207 for joint judgment.

[0082] Background stability analysis module 207: This module outputs the frame image from module 202. For input, and using from module 204 The interleaved regions are located, and the inter-frame variations in the gap regions are evaluated. Two metrics are used to assess stability: structural similarity-similarity (SSIM) (≥0.90) and brightness variation. (Requirement ≤5%). The stability assessment result is output to module 208 to assist in determining whether the area can be used for ad insertion.

[0083] Empty Area Confirmation Module (208): This module integrates the Rt value from module 205, the time window flag from module 206, and the stability judgment result from module 207, and performs a logical AND operation: only when all three conditions are met simultaneously (i.e., area meets the standard, continuity is sufficient, and background is stable) is the frame marked as an "ad insertion frame," and the final empty area mask is output. This mask can then be used by subsequent ad generation and fusion modules.

[0084] To achieve accurate and natural ad content insertion, this system is designed with a multi-module collaborative video analysis and gap recognition system and process:

[0085] 200 outdoor Vlog videos were selected from public datasets on short video platforms (such as Douyin and Kuaishou). Each video was approximately 8–20 seconds long. Frames were extracted from the videos using FFmpeg (as shown in Figure 202). The frame rate was fixed at 50fps and the resolution was 1920×1080.

[0086] Semantic segmentation: Using Mask R-CNN based on ResNet-101, the semantic mask map Mt is output for each frame. All pixels belonging to "person," "vehicle," and "moving object" are labeled as 1, and all others as 0. Further definitions are provided. The value is set to 0 if the pixel is "background" and its distance from the target boundary is greater than 10px; otherwise, it is 0. This processing is performed as shown in Figure 204, outputting a gap mask. .

[0087] The system calculates the percentage of empty areas in each frame.

[0088]

[0089] Where R t Ratio, representing the percentage of pixels in the "empty area" in frame t, is a normalized real number. B t This represents the binary mask matrix in the t-th frame of the image. This represents the summation of all pixel values ​​in the gap mask matrix, i.e., the number of pixels that are 1; H represents the image height (in pixels); W represents the image width; H×W represents the total number of pixels in the entire frame.

[0090] Count how many frames out of 15 consecutive frames satisfy the condition. .

[0091] Perform sliding statistics on the time window (step size of 1 frame) to determine if there are more than 10 frames that satisfy the condition. .

[0092] To avoid misjudgments caused by lens changes or sudden background shifts, the SSIM evaluation metric and the difference between the color mean and the mean are introduced:

[0093] set up For the current frame, The previous frame;

[0094] SSIM calculation is performed on the candidate regions, requiring ;

[0095] The mean difference of the RGB three channels Δμ ≤ 5.

[0096] Perform the above process on each video segment, extract all candidate frames, determine whether they are insertable gap frames based on three conditions, and count the number of gap regions and the average duration.

[0097] The total number of naturally inserted gap regions identified was, for example, 263; the average number of gap regions identified per video segment was, for example, 1.31; and the average duration of each gap region was, for example, 1.6 seconds (i.e., 80 frames).

[0098] The abruptness rating of the gap area was collected from volunteers' subjective ratings. The average score of the traditional static insertion strategy was 3.0 / 5, while this method can reach 4.1 / 5 (out of 5).

[0099] This statistic is used in Figure 208 to determine the final insertion region output.

[0100] Example 2: Multimodal context modeling and insertion suitability prediction mechanism:

[0101] Figure 3 This embodiment illustrates the functional structure of the "Multimodal Context Modeling and Ad Insertion Suitability Prediction Mechanism" unit, which includes the following processing steps or reference numerals:

[0102] This mechanism is used to determine whether the current video frame is suitable for ad insertion, and is a specific refinement of the context modeling and insertion intent evaluation module 104 in Implementation Example 1. The system achieves precise control over ad insertion scenarios through joint modeling of multiple modalities, combined with emotional state, dynamic rhythm, and scene complexity.

[0103] The system mainly includes the following functional modules, which are connected through intermediate feature tensors to form a clear data flow path:

[0104] Video frame input module 301: This video frame input module 301 is the starting module of the context modeling and insertion intent evaluation module 104, and is responsible for extracting the current video image frame (denoted as ) from the candidate insertion frame sequence. The image frame is then fed in parallel into three subsequent feature extraction modules: facial expression recognition module 302, dynamic rhythm analysis module 303, and scene structure modeling module 304, to support multimodal context modeling. This module supports high-definition resolution (e.g., 1080p) video input, with frame rate synchronized with timestamps, enabling precise temporal positioning.

[0105] The facial expression recognition module 302 is used to analyze the emotional state of the main characters in the current video frame, identify the characters' facial expressions, and quantify them into a seven-dimensional emotion probability vector. This module first locates and encodes the facial regions in the video frames, extracting facial feature tensors. The pre-trained FER+ facial expression recognition network outputs an emotion distribution vector.

[0106] W e f face b e

[0107] In the formula This represents an emotion probability vector with dimension R. 7 This refers to the confidence level corresponding to seven emotion categories: happiness, sadness, surprise, anger, fear, disgust, and neutrality. This emotion vector... It is then transmitted to the multimodal fusion module 305 to participate in the construction of the context vector. This indicates that the input vector is processed. The function is normalized so that its output is a probability distribution, where the sum of all components is 1, and each component value is between 0 and 1. W e This represents the weight matrix in the emotion recognition network that maps facial feature vectors to the emotion category space. Its dimension is 7×d, where d is the facial convolutional feature vector. Dimensions. This represents the feature vector extracted from a facial image using a convolutional neural network, with dimensions of [dimensionality missing]. This is the basic facial expression information input to the model. e This represents the bias vector, with dimension . An offset adjustment parameter is introduced for each emotion category. Therefore, this formula means that the facial feature vector is first... Linear mapping to the score space of 7 emotion categories Then through This is converted into a probability for each type of emotion, forming the final emotion probability vector e.

[0108] The dynamic rhythm analysis module 303 is responsible for modeling the motion of people and changes in audio signals in the video, and extracting the motion rhythm feature vector r. This module first calculates the pixel-level displacement velocity between video frames using the TV-L1 optical flow method, deriving the average optical flow velocity. Next, extract the RMS energy change rate of the audio signal within the t-second time window. And count the movement rate of the center point of the person detection box. Speed ​​standard deviation Ratio of silent zone Finally, the action rhythm vector is constructed. This five-dimensional vector is used to characterize the rhythmic features of the video, and is then fed into the multimodal fusion module 305 for joint modeling.

[0109] The scene structure modeling module 304 is used to extract the visual structure features of the scene in the current video frame and output the structure vector s. This module uses a ResNet-50 backbone network to encode the entire frame image and extracts several indicators from it, including the standard deviation of edge strength. Focus score is a measure of the texture complexity of an image. Reflects the clarity of the subject and the number of targets per unit area. Used to describe crowding level, color channel information entropy To measure image richness, the left-right brightness symmetry of the image. This is used to measure the balance of the composition. The final result is the structural feature vector. And send it to the multimodal fusion module 305.

[0110] The multimodal fusion module 305 is the core feature fusion component of this mechanism. It uses a cross-attention transformer structure to fuse feature vectors from three modalities. (mood), (Rhythm of movement) (Visual structure) is jointly modeled. The system first concatenates the three types of vectors: This constitutes a 25-dimensional (where ) Context vector This vector represents the semantic, emotional, and visual context of the current frame. It is then fed into the rating prediction module 306 as input to determine whether it is appropriate to insert an advertisement.

[0111] The insertion suitability score prediction module 306 receives the context vector c from the fusion module 305, performs score prediction using a two-layer MLP neural network, and outputs the insertion suitability score for the current frame. The specific structure of the model is as follows: ,in for The function is used to normalize the output to the probability domain; , which is a two-layer weight matrix; For activation functions; and This is the bias coefficient. This score is used to quantify the user's acceptance of the ad insertion operation at the current moment and its contextual fit. The score result is then transmitted to the insertion trigger control module 307.

[0112] The insertion trigger control module 307 acts as a decision-making module, jointly determining whether to perform the advertisement insertion operation based on the score prediction result and the marked empty area. When the score value... If the current frame has been determined by the gap area confirmation module 208 in Embodiment 1 to be an insertable frame, the ad insertion process is immediately triggered, and the dynamic ad generation and fusion module described in Figure 105 is called to perform content generation and visual embedding operations. At the same time, this module supports strategy-level parameter configuration (such as insertion thresholds, interval control, frequency limits, etc. for different platforms) to ensure the visual naturalness and commercial effectiveness of the insertion.

[0113] To achieve ad insertion suitability prediction based on multimodal context modeling, we designed a further detailed description of the context modeling and insertion intent evaluation module 104, the entire contents of Embodiment 1 and Embodiment 2 of which are cited here:

[0114] 1. Video frame input module 301:

[0115] Input: Video source file, in H.264 encoded MP4 format;

[0116] Processing: FFmpeg and OpenCV are used to decode the video, extract frames, and separate audio to obtain an image frame sequence. With audio frames Output: Frame image (1920×1080 RGB image) and corresponding audio segment (44100Hz, mono PCM) is transmitted to modules 302–304.

[0117] 2. Facial expression recognition module 302:

[0118] Input: Image frame ;

[0119] Processing: Face detection (based on MTCNN or RetinaFace) is used to crop the face region and feed it into the FER+ network to extract facial features. (Dimension 512), then after The classifier calculates a 7-dimensional sentiment vector:

[0120] 7 ;

[0121] Output: Emotion probability vector Each component corresponds to the emotions "happiness", "sadness", "surprise", "anger", "fear", "disgust", and "neutrality".

[0122] 3. Dynamic Rhythm Analysis Module 303:

[0123] Input: Image frame sequence Audio ;

[0124] Processing: TV-L1 optical flow extraction of target movement velocity in images ;

[0125] Audio RMS energy calculation

[0126] Inter-frame L2 displacement of the character's bounding box midpoint ;

[0127] Standard deviation of optical flow velocity ;

[0128] Silent frame rate When RMS is less than 0.02, it is considered silent;

[0129] Output: Rhythm vector ;

[0130] 4. Scene Structure Modeling Module 304:

[0131] Input: Image frame ;

[0132] Processing: Edge strength Perform a Laplacian transform on the image and calculate the standard deviation; focus score. : Ratio of sharp areas in the image; : Detect the number of objects / image area using YOLOv8; : Calculate the entropy value of the HSV color histogram; : Average brightness difference between the left and right sides of the image;

[0133] Output: Structure vector ;

[0134] 5. Multimodal fusion module 305:

[0135] Input: Emotion vector e, rhythm vector r, structure vector s

[0136] Processing: Initial Fusion Cross-Attention Transformer: 3-layer encoder, hidden size=64, multi-head=4; outputs a contextual encoding vector c ∈R² 5 ;

[0137] Output: c is passed to the score prediction module.

[0138] 6. Insert suitability score prediction module 306:

[0139] enter:

[0140] Processing: A two-layer MLP structure is used for scoring output, with the following formula:

[0141]

[0142] in , , for Function; where the bias term These are parameters automatically learned during model training, used to improve the model's fitting ability and non-linear expression. The bias vectors and weight matrices mentioned above are optimized on the training set using the backpropagation algorithm, and the loss function is binary cross-entropy. Output .

[0143] 7. Insert trigger control module 307:

[0144] enter: The result of determining the gap frame in Figure 208; the judgment logic: when If the current frame is a valid empty area, the ad insertion strategy is triggered (the material generation and fusion module 105 and control module 106 in Example 1 can be called). Output: Insertion execution signal.

[0145] Regarding the data sources and training settings used in the above embodiments.

[0146] 1. Data Acquisition:

[0147] Source: KVideo-3M dataset from a short video open platform;

[0148] Selection criteria: Short videos with ad insertion tags will be retained, totaling 32,000 videos (duration 6–30 seconds).

[0149] Positive samples: Insertion moments where users do not bounce and the video completion rate is high;

[0150] Negative samples: those that generate obvious negative behaviors or cause users to leave the site after insertion;

[0151] Split ratio: training set: validation set = 8:2 (25600 records + 6400 records).

[0152] 2. Model training configuration:

[0153] Optimizer: AdamW

[0154] Initial learning rate: 1e -4 ;

[0155] Batch size: 64;

[0156] Epochs: 20;

[0157] Loss function: Binary cross-entropy (BCE).

[0158] 3. Validation set performance metrics:

[0159] Accuracy: 89.3%

[0160] The harmonic mean of precision and recall (F1-score): 0.87;

[0161] The system's overall classification ability across all thresholds (ROC AUC): 0.91;

[0162] Inference latency: ≈26ms per frame (NVIDIA RTX 3090)

[0163]

[0164] Table 1 Comparison of the technical effects of the present invention and the prior art

[0165] In practical application testing, the advertising insertion system of this invention was deployed on a short video playback platform, and a travel vlog video was selected as the test subject. The video content shows a user walking into a hotel entrance, with the person in the picture smiling. The emotion recognition module identified the "happy" emotion with a probability of 0.68 (i.e., The analysis of the scene structure shows a stable background, high architectural symmetry, and a clear and concise overall image. The rhythm analysis indicates a low optical flow velocity and smooth RMS energy changes in the audio, suggesting a relaxed rhythm without dramatic movements or emotional fluctuations. After fusing these multimodal features, the generated context vector score is 0.83, significantly exceeding the system's insertion threshold (0.7). Therefore, the system decides to insert a hotel recommendation poster ad consistent with the scene's style at that moment, using a semi-transparent overlay to blend into the lower right corner of the image. According to user behavior feedback statistics, the video's ad click-through rate increased by 27.6%, video completion rate increased by 14%, and bounce rate decreased by 18%. Simultaneously, user subjective ratings improved from an average of 3.1 / 5 using traditional methods to 4.4 / 5, validating the effectiveness and user acceptance of the context-aware insertion mechanism constructed in this invention in real-world environments.

[0166] The above embodiments significantly improve the contextual adaptability and user acceptance of ad insertion by performing multimodal fusion analysis through facial emotion, movement rhythm, and visual structure modeling. Compared with traditional single-rule methods, this invention not only improves prediction accuracy but also enhances the naturalness of insertion behavior and commercial conversion efficiency, making it suitable for deployment on short video platforms such as Douyin and Kuaishou.

Claims

1. A method for intelligent insertion of video advertisements based on semantic segmentation and multimodal fusion, characterized in that, Includes the following steps: S1. Perform frame extraction processing on the input video stream to obtain an image frame sequence; S2. Perform pixel-level recognition on the image frame using a semantic segmentation model to generate a semantic mask image. Identify non-critical semantic regions; S3. Based on the pixel density of idle regions in consecutive frames, combined with temporal sliding window statistics and structural similarity and brightness change analysis, identify advertising gap regions that satisfy spatial and temporal continuity; the identification of advertising gap regions that satisfy spatial and temporal continuity specifically includes: Based on the preset key semantic tag set Determine the semantic mask image For pixels that do not belong to the key semantic tag set, generate a binary mask. ,in This represents the area where ads can be inserted; Calculate the binary mask percentage of the whole frame The formula is: ,in Given the entire image frame size; construct a time sliding window of length T, and count the values ​​within the window that satisfy... If at least M=10 frames meet this condition, then the time period corresponding to the time sliding window of length T is determined to be a continuous gap candidate area. Perform background stability analysis on the continuous gap candidate regions and calculate the current frame. Compared to the previous frame Structural similarity (SSIM) and brightness variation within the candidate region Only when And brightness changes At that time, confirm that the area is an advertising gap area; S4. For the frames corresponding to the gap region, extract multimodal contextual features, including emotion vectors generated from facial expressions, video rhythm vectors, and image structure vectors. S5. Input multimodal features into the fusion network to perform contextual modeling and generate an insertion suitability score; S6. When the score exceeds the set threshold and the frame is a legal empty area, perform the ad generation and visual fusion operation to embed the ad into the corresponding frame.

2. The method as described in claim 1, characterized in that: The semantic segmentation model is Its output mask image is marked with at least three semantic tags: people, products, and background, which are used to delineate non-critical areas.

3. The method as described in claim 1, characterized in that: The gap area identification is based on A time sliding window of frame length is used to count at least [number] frames. Frames that satisfy the interpolable region density threshold; where .

4. The method as described in claim 1, characterized in that: The emotion vector is a seven-dimensional probability vector, corresponding to the recognition results of seven emotion categories.

5. The method as described in claim 1, characterized in that: The multimodal fusion employs a cross-attention transformer structure, outputting a 25-dimensional contextual semantic vector for inserting score prediction.

6. A video advertisement insertion system for performing the method according to any one of claims 1-5, characterized in that, Includes the following modules: The video preprocessing and framing module is used to receive the raw video stream and perform frame extraction processing to generate a continuous sequence of image frames. The object detection and semantic segmentation module is used to perform object recognition and semantic segmentation on each frame of the image and output a pixel-level mask image. The gap point recognition module is used to analyze the pixel ratio of non-critical areas based on the mask image, and combine the time sliding window and background stability to identify the gap areas for advertisement insertion that meet the conditions. The context modeling and insertion intent assessment module includes facial expression recognition, rhythm analysis, and structural modeling sub-modules, which are used to extract multimodal contextual features and fuse them to generate an insertion suitability score. The dynamic ad generation and fusion module is used to select or generate ad content based on empty areas and context information, and to fuse ad images into video frames; The real-time rendering and delivery control module is used to control the frequency of ad insertion, the presentation method, and the final rendering and output operations on the user's device.

7. The video advertisement insertion system as described in claim 6, characterized in that: The gap point identification module includes the following sub-modules: The video input module is used to receive raw video data and parse the frame rate and resolution; The frame extraction module is used to convert the video data into a sequence of image frames; The semantic segmentation module performs pixel-level semantic segmentation on each frame of an image based on a pre-trained Mask R-CNN model, and outputs a semantic mask map. ; The gap candidate mask generation module is used to identify non-critical semantic regions from the semantic mask map and generate binary masks. ; The region area calculation module is used to calculate the binary mask. Pixel ratio of interpolable regions ; The continuous frame window determination module is used to determine whether at least one frame exists within a time sliding window of length T. Frame satisfies This allows for the determination of candidate regions for continuous gaps. The background stability analysis module is used to calculate structural similarity in the empty region. and brightness change To determine the stability of background changes; The gap area confirmation module is used to perform joint logical judgment on the results from the area calculation module, the continuous frame window determination module, and the background stability analysis module, and output the final advertisement can be inserted mask frame.

8. The video advertisement insertion system as described in claim 6, characterized in that: The context modeling and insertion intent evaluation module includes: The facial expression recognition module extracts facial image features based on a pre-trained model and outputs an emotion vector containing seven basic emotion confidence levels to characterize the current person's emotional state. The dynamic rhythm analysis module is used to generate a multi-dimensional rhythm vector that reflects the rhythm characteristics of the picture based on the target motion between image frames, the energy change of the audio signal, the speed of the character's movement and the proportion of silence. The scene structure modeling module is used to extract the texture complexity, sharpness, scene crowding, color information richness, and left-right brightness symmetry of the image to characterize the visual structure. A multimodal fusion module, which uses a cross-attention mechanism to semantically fuse three types of vectors—emotion, rhythm, and structure—to obtain a comprehensive contextual representation of the current frame; An insertion suitability score prediction module is provided. Based on the fused context representation vector, the insertion suitability score prediction module evaluates the contextual fit of the current time when the advertisement is inserted and outputs a score value representing the insertion suitability. An insertion trigger control module is used to trigger an advertisement insertion action when the score value reaches a preset threshold and the current frame has been marked as an empty area.

9. A computer-readable storage medium having a computer program stored thereon, which, when run in a processor, causes a computer to perform all the steps of the method as claimed in any one of claims 1-5.

Images