Video advertisement intelligent insertion method and system based on semantic segmentation and multi-modal fusion, and medium

Abstract

The invention discloses a video advertisement intelligent insertion method and system based on semantic segmentation and multi-modal context fusion. Through pixel-level semantic mask and multi-frame time sequence analysis, a space neutral area which can be used for advertisement embedding in a video is automatically identified, and background stability judgment is carried out in combination with indexes such as structural similarity and brightness change; and constructing a multi-modal context modeler, fusing a character emotion vector, a video rhythm vector and a scene structure vector, generating a context situation vector through a cross attention mechanism, and outputting an insertion suitability score. And when the score reaches a set threshold value and the neutral area meets the condition, the system executes advertisement generation and visual fusion operation to realize natural advertisement insertion consistent with the original video content style. The method is suitable for various content platforms such as short video and live broadcast, and experiments show that the click rate and the complete broadcast rate can be remarkably improved, and the jump-out rate is reduced.

Description

Technical Field

[0001] The present invention relates to the field of integration of artificial intelligence, computer vision and digital advertising, and specifically to an intelligent advertising insertion method based on video content understanding, which is applicable to online content scenarios such as live video and short video streaming. Background Art

[0002] With the explosive growth of short video and live streaming platforms, traditional advertising formats (such as pre-roll ads and forced ads before pauses) are increasingly perceived by users as disruptive, leading to high skip and block rates, impacting conversion rates. Furthermore, current mainstream video ad insertion methods still rely on fixed time points or manual platform settings, failing to dynamically adjust the insertion position based on actual content. This lack of contextual awareness can easily cause user attention interruptions or emotional aversion.

[0003] In recent years, significant breakthroughs have been made in visual recognition algorithms such as object detection (e.g., YOLO and DETR) and semantic segmentation (e.g., DeepLab and Mask R-CNN), making frame-level video understanding possible. However, a systematic solution remains to effectively integrate these visual technologies with ad insertion logic to create a "natural and non-intrusive" ad placement mechanism. Summary of the Invention

[0004] This invention proposes a method for intelligent video content analysis and advertisement insertion. By performing object detection and semantic segmentation on video frames, the method identifies people, objects, background elements, and semantically unavailable areas. Combined with scene rhythm analysis, the method automatically identifies the optimal insertion point and dynamically generates advertisement content that matches the scene, achieving immersive advertisement insertion. This method overcomes at least one technical problem existing in the related art.

[0005] According to a first aspect of the embodiments of this specification, a method for intelligent video advertisement insertion based on semantic segmentation and multimodal fusion is provided, characterized by comprising the following steps: S1, extract frames from the input video stream to obtain an image frame sequence; S2. Perform pixel-level recognition on the image frame using a semantic segmentation model to generate a semantic mask map and identify non-critical semantic areas; S3, based on the pixel density of the idle area in consecutive frames, combined with the time sliding window statistics and structural similarity ( ) and brightness changes ( ) Analyze and identify advertising gaps that meet spatial and temporal continuity; S4. Extracting multimodal context features from the frames corresponding to the blank area, including emotion vectors generated by facial expressions, video rhythm vectors, and image structure vectors; S5, inputting the multi-modal features into a fusion network for context modeling to generate an insertion suitability score; S6, when the score exceeds a set threshold and the frame is a legal empty slot region, performing advertisement generation and visual fusion operations to embed the advertisement in the corresponding frame.

[0006] Preferably, the semantic segmentation model is Mask R-CNN, which outputs a mask graph with labels including at least three semantic labels of person, commodity and background, for delimiting non-key regions.

[0007] Preferably, the empty slot region identification is based on a time sliding window with a T-frame length, and not less than M frames satisfying a density threshold of the insertable region are counted; wherein T=15 and M=10.

[0008] Preferably, the background stability determination requires a structural similarity SSIM between consecutive frames to be greater than or equal to 0.90, and a RGB brightness change AI to be less than or equal to 5%.

[0009] Preferably, the emotion vector is a seven-dimensional probability vector corresponding to the identification results of seven emotion categories.

[0010] Preferably, the multi-modal fusion adopts a cross-attention transformer (Cross-Attention Transformer) structure, and outputs a 25-dimensional context semantic vector for insertion score prediction.

[0011] Another aspect of the present application is to provide a video advertisement insertion system for performing a video advertisement intelligent insertion method based on semantic segmentation and multi-modal fusion, characterized by comprising the following modules: A video preprocessing and framing module is configured to receive an original video stream and perform frame extraction processing to generate a sequence of continuous image frames. A target detection and semantic segmentation module is configured to perform target recognition and semantic segmentation on each frame of image to output a pixel-level mask graph. An empty slot recognition module is configured to analyze the pixel ratio of non-key regions based on the mask graph, and identify an advertisement insertion empty slot region meeting the conditions in combination with a time sliding window and background stability. A context modeling and insertion intention evaluation module includes an expression recognition, rhythm analysis and structure modeling submodule, configured to extract multi-modal context features and generate an insertion suitability score by fusion. A dynamic advertisement generation and fusion module is configured to select or generate advertisement content based on the empty slot region and context information, and embed the advertisement image into the video frame by fusion. A real-time rendering and delivery control module is configured to control the advertisement insertion frequency, presentation mode and final rendering and output operations on the user equipment side.

[0012] Preferably, the neutral point recognition module includes the following submodules: Video input module, used to receive raw video data and parse metadata such as frame rate and resolution; A frame extraction module, configured to convert the video data into an image frame sequence; The semantic segmentation module is used to perform pixel-level semantic segmentation on each frame of the image based on the pre-trained Mask R-CNN model and output a semantic mask map ( ); A gap candidate mask generation module is used to identify non-key semantic areas from the semantic mask map and generate a binary mask ( ); The area calculation module is used to calculate the binary mask ( ) in the pixel ratio of the interpolated area ( ); The continuous frame window judgment module is used to judge whether there are at least M frames that meet the requirements in the time sliding window of length T. , thereby determining the continuous gap candidate area; Background stability analysis module, used to calculate the structural similarity in the gap area and brightness changes , to judge the stability of background changes; The blank area confirmation module is used to perform joint logical judgment on the results from the area calculation module, the continuous frame window judgment module and the background stability analysis module, and output the final advertisement insertion mask frame.

[0013] Preferably, the context modeling and insertion intention assessment module includes: A facial expression recognition module, which extracts facial image features based on a pre-trained model and outputs an emotion vector containing seven basic emotion confidence levels to characterize the current person's emotional state; The dynamic rhythm analysis module is used to generate a multi-dimensional rhythm vector reflecting the rhythm characteristics of the picture based on indicators such as the target motion between image frames, the energy change of the audio signal, the speed of the character's movement, and the silence ratio; The scene structure modeling module is used to extract image features such as texture complexity, clarity, scene crowding, color information richness, and left-right brightness symmetry to characterize visual structure; A multimodal fusion module, which semantically fuses emotion, rhythm, and structure vectors through a cross-attention mechanism to obtain a comprehensive contextual representation of the current frame; an insertion suitability score prediction module, wherein the insertion suitability score prediction module evaluates the contextual fit of the current moment of advertisement insertion based on the fused context representation vector and outputs a score value representing the insertion suitability; An insertion trigger control module triggers an advertisement insertion action when the score value reaches a preset threshold and the current frame has been marked as a non-interference region.

[0014] Another aspect of the present application provides a computer readable storage medium having stored thereon a computer program which, when executed in a processor, causes a computer to perform all the steps of the method for intelligent video advertisement insertion based on semantic segmentation and multi-modal fusion.

[0015] In the embodiments of the present application, by introducing an insertion gap identification mechanism based on semantic segmentation, the system can perform pixel-level analysis on the spatial structure in the video image, and realize automatic identification and labeling of the non-interference region.

[0016] Compared with the traditional method of relying on time axis or manually setting insertion points, this mechanism has two advantages: first, it can realize fine-grained spatial judgment. The system uses semantic segmentation networks such as Mask R-CNN to divide the main body and background of each frame, and combines mask analysis algorithms to dynamically evaluate the region idle degree and visual stability, and identify the insertion space without interfering with the main visual focus. Second, it ensures the continuity and stability of multiple frames. Instead of making an immediate judgment based on a blank region in a frame, the present application introduces a cross-frame time window statistical mechanism to ensure that the insertion region is continuous in time (for example, in 15 consecutive frames, more than 10 frames meet the "non-interference" threshold), effectively avoiding false judgments caused by camera shake or sudden actions. This mechanism upgrades the advertisement embedding logic from "time-driven" to "spatial semantic-driven", not only improving the visual naturalness of advertisement fusion, but also greatly reducing the dependence on manual intervention, laying a foundation for the automation and scaling of advertisement insertion.

[0017] In addition, by constructing a "multi-modal context modeler", heterogeneous information such as user facial emotions, video rhythm features, scene structures, etc. are fused and modeled to form a global perception of the current video content and an insertion suitability judgment. The key advantage of this fusion modeling is that it breaks free from the limitations of a single visual channel and realizes multi-dimensional context perception capability. Specifically, the emotion recognition module extracts an emotion vector by analyzing facial expressions to assist in judging whether the current character state is suitable for intervention (for example, to avoid inserting an advertisement in a tense or sad emotion); the rhythm analysis module determines the stability and fluctuation of the picture rhythm through techniques such as optical flow estimation and sound intensity curve analysis to ensure that the advertisement does not interrupt an action climax or a dialogue climax; the structure analysis module evaluates the object density and composition balance in the scene to assist in selecting an insertion region outside the visual focus. By inputting the above features into a Transformer structure to realize cross-modal semantic fusion, and combining a lightweight classifier to output an insertion suitability score, the system can make a decision on whether to insert an advertisement within milliseconds.

[0018] Experiments show that the modeling mechanism has significant advantages over traditional rule engines in terms of ad click rate, user bounce rate and other indicators, especially in high-paced scenarios such as live streaming and short videos, which can maximize user immersion while balancing content quality and business conversion goals. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 The overall system unit function structure diagram of the embodiment is shown in the figure. Figure 2 The semantic gap identification unit function structure diagram of the first embodiment of the present application is shown in the figure. Figure 3 The multi-modal context modeling and insertion suitability prediction mechanism unit function structure diagram of the second embodiment is shown in the figure. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present specification will be described clearly and completely below with reference to the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0022] It should be noted that the terms "include" and "have" and any variations thereof in the embodiments of the present specification and the drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0023] Embodiment I: Figure 1 The complete system architecture of the present application "video content intelligent analysis and advertisement insertion method" is shown. The whole system receives live streaming or short video stream data from the input end, and completes content analysis, gap detection, context modeling, advertisement generation, final fusion and delivery control in turn through six main function modules.

[0024] The system architecture is connected between the modules through standard data flow channels, and each module has clear functional division and independent processing capability. The specific structure and function are as follows: Video preprocessing and framing module 101: This module receives raw short video or live stream data, and performs format conversion, decoding and frame extraction processing. The system supports multiple video encoding formats (such as H.264 / H.265), and can adaptively sample the frame rate as needed (such as 50fps, 30fps, etc.), outputting continuous video frame sequences as subsequent analysis input.

[0025] Target detection and semantic segmentation module 102: This module performs visual recognition operations on each frame of image, uses YOLOv8 to detect the main targets (such as people, goods, gestures, etc.) existing in the image, and calls the pre-trained Mask R-CNN model to perform pixel-level semantic segmentation on the image, outputting a semantic mask image . The mask image is used for subsequent judgment of the insertable area in the picture.

[0026] Blank spot identification module 103: Using the semantic mask image output by the segmentation module 102 , the blank spot identification module 103 identifies the potential advertisement insertion blank (insertion position) in the image by analyzing the pixel ratio of "non-key semantic area" in each frame of image. The system sets pixel threshold, duration window and background stability judgment mechanism to ensure that the advertisement insertion position is natural and not conspicuous.

[0027] Context modeling and insertion intention evaluation module 104: While determining the existence of an insertable blank, this module constructs a "context understanding vector" of the video content. It includes three types of input: The expression recognition module outputs the current emotional state of the person; The action rhythm extraction module analyzes the video rhythm (such as moving speed, speech energy change, etc.); The scene structure modeling module outputs the complexity and focus point distribution of the current picture.

[0028] The three are combined through a multi-modal fusion network (such as a Transformer structure) to output a context vector, and an advertisement insertion suitability score is output by an insertion intention evaluator. Only when the score is higher than the set threshold (such as 0.7), will the advertisement insertion be triggered.

[0029] Dynamic advertisement generation and fusion module 105: According to the empty slot recognition module 103, the context modeling and insertion intention evaluation module 104 outputs the insertable position and context situation vector, the system selects the matching material from the advertisement material library, or calls the generative AI (such as ControlNet, DALL·E, etc.) to generate the advertisement sticker consistent with the current picture style in real time. Then through Alpha fusion, affine transformation and color adjustment technology, the advertisement content is embedded in the image frame, realizing visual seamless connection.

[0030] Real-time rendering and delivery control module 106: This module 106 controls the final embedding process of the advertisement and the execution of the delivery strategy, including the rendering engine, the image writing back module and the advertisement delivery controller. It determines the timing and frequency of inserting advertisements according to user platform, viewing device resolution, historical click rate and other parameters, and supports interactive advertisements (such as click jump) or floating advertisements.

[0031] The system data flow flows from left to right, and each module can be independently deployed or integrated in the same GPU workstation or cloud platform container. The modules communicate through standardized API interfaces, with high scalability and flexible configuration capabilities.

[0032] Embodiment two, insertion empty slot recognition mechanism based on semantic segmentation: To further illustrate the first core point of the present application "semantic segmentation and insertion empty slot detection mechanism", this embodiment two is a further detailed description of the empty slot recognition module 103 in embodiment one, and the entire content of embodiment one is hereby incorporated by reference.

[0033] From the sampling data, semantic processing flow, insertable area judgment formula, stability analysis and multi-frame strategy, the system is modeled and verified. As shown in the figure, the functional structure diagram of the semantic empty slot recognition unit in the present application is shown, and each flow node corresponds to the reference signs 201 to 208. The following processing steps and reference signs are included: Figure 2 I. Module structure: Video input module 201: The video input module 201 receives the input raw video data; supports multiple formats (such as MP4, FLV, RTMP, etc.), and parses its frame rate, resolution and other metadata. Its output is a video data handle, which is passed to the frame extraction module 202 as input ​Frame extraction module 202: This frame extraction module 202 converts the video into a continuous frame sequence based on OpenCV. Specifically, this module receives the video data handle output by the video input module 201, extracts a continuous frame image sequence at a specified frame rate (such as 50 fps) through tools such as OpenCV, and uniformly scales it to 1080p resolution. The output is an image frame sequence {It}, and each frame It is transmitted to module 203 (semantic segmentation module) for pixel-level semantic analysis.

[0034] Semantic segmentation module 203: This semantic segmentation module 203 is used to call a pre-trained Mask R-CNN model to output a pixel-level semantic mask Mt; Specifically, it receives the image frame I t from the extraction module 202, uses a deployed pre-trained Mask R-CNN model to perform semantic segmentation on each frame, and outputs a semantic mask image , Each pixel in the mask image is labeled as a corresponding semantic label (such as a person, a product, a background, etc.). This mask image is passed as input to the empty slot candidate mask generation module 204 (empty slot candidate mask generation module).

[0035] Empty slot candidate mask generation module 204: This module receives the semantic mask Mt output by module 203, and according to a pre-set "key semantic label set" F (such as a person, a product, a hand, etc.), judges which pixels do not belong to F, and marks them as insertable areas. Finally, a binary mask , is generated, where represents the area available for ad insertion. This is then input to module 205 (area calculation module) Area calculation module 205: This module receives the mask output by module 204, counts the total number of in all pixels, and calculates the proportion of this value in the entire frame . Its formula is as the "insertable area density" indicator of this frame, it is passed to module 206 (continuous frame window determination module) as one of the determination criteria.

[0036] Continuous frame window determination module 206: Module 206 receives the value sequence output by module 205, and constructs a time sliding window with a length of , and counts the frames that satisfy in it. If there are at least If the frame meets this condition, the time period is determined to be a "continuous gap candidate area." The result is output as a flag and passed to module 208 and module 207 for joint judgment.

[0037] Background stability analysis module 207: This module uses the frame image output by module 202 as input and uses the Locate the interpolated area and evaluate the inter-frame changes in the blank area. Two indicators are used to judge the stability: structural similarity SSIM (required ≥ 0.90) and brightness change The stability determination result is output to module 208 to assist in determining whether the area can be used for advertisement insertion.

[0038] The blank area confirmation module (208) combines the Rt value from module 205, the time window flag from module 206, and the stability judgment result from module 207, and performs a logical AND operation. Only when all three conditions are met (i.e., the area meets the standard, the continuity is sufficient, and the background is stable) will the frame be marked as an "ad insertion frame" and the final blank mask will be output. This mask can then be used by the subsequent ad generation and fusion modules.

[0039] To achieve accurate and natural advertising content insertion, this system has designed a multi-module collaborative video analysis and gap recognition system and process: 200 outdoor Vlog videos were selected from the public datasets of short video platforms (such as Douyin and Kuaishou), each of which was approximately 8–20 seconds long. Frames were extracted using the FFmpeg tool (as shown in Figure 202), with a fixed frame rate of 50 fps and a resolution of 1920×1080.

[0040] Semantic segmentation: Use Mask R-CNN based on ResNet-101 to output a semantic mask map Mt for each frame. Mark all pixels belonging to "people", "vehicles", and "moving objects" as 1, and the rest as 0. Further define: , when the pixel is "background" and is >10px away from the target boundary; otherwise it is 0. This process is completed in Figure 204, and the output is the gap mask .

[0041] The system calculates the percentage of idle area in each frame where R t The ratio of pixels in the “blank area” in the t-th frame image is a normalized real number. ; B t Represents the empty mask matrix (Binary Mask) in the t-th frame image; represents the sum of all pixel values in the null mask matrix, i.e. the number of pixels counted as 1; H represents the image height (in pixels); W represents the image width; HxW represents the total number of pixels in the entire frame.

[0042] Count how many frames in the last 15 frames satisfy .

[0043] Slide the timing window (step size is 1 frame) to judge whether there are more than 10 frames that satisfy .

[0044] To avoid misjudgment due to lens changes or background mutations, introduce the SSIM evaluation index and color mean difference calculation: Let be the current frame, be the last frame; Calculate the SSIM of the candidate region, requiring ; The mean difference of the RGB three channels Δμ ≤ 5.

[0045] Execute each video according to the above process, extract all candidate frames, and judge whether it is an insertable null frame through the joint judgment of the three conditions, and count the number of null regions and the average duration.

[0046] A total of 263 natural insertion null regions are identified; the average number of null regions identified per video is 1.31; the average duration of each null region is 1.6 seconds (i.e. 80 frames); The null region abruptness score is collected by subjective scoring by volunteers. The average score of the traditional static insertion strategy is 3.0 / 5, and the method can reach 4.1 / 5 (full score 5). The statistics are used in the final insertion region judgment output in Figure 208.

[0047] Example Two, Multi-modal Context Modeling and Insertion Suitability Prediction Mechanism: Figure 3 This embodiment shows the "multi-modal context modeling and advertisement insertion suitability prediction mechanism" unit function structure diagram, which includes the following processing steps or figure marks: This mechanism is used to judge whether the current video picture is suitable for performing an advertisement insertion operation, which is a specific refinement of the context modeling and insertion intention evaluation module 104 in Example One. The system realizes precise control of the advertisement insertion scene through joint modeling of multiple modalities, combined with emotional state, dynamic rhythm and scene complexity: The system mainly includes the following functional modules, and the modules are connected through intermediate feature tensors to form a clear data flow path: Video frame input module 301: This video frame input module 301 is the starting module of the context modeling and insertion intent evaluation module 104, responsible for extracting the video image frame at the current time (denoted as ) from the candidate insertion frame sequence, and sending this image frame into the subsequent three feature extraction modules, namely the facial expression recognition module 302, the dynamic rhythm analysis module 303, and the scene structure modeling module 304 in parallel, to support multi-modal context modeling. This module supports high-definition resolution (such as 1080p) video input, with frame rate and timestamp synchronization, which can be used for precise timing positioning.

[0048] The facial expression recognition module 302 is used to analyze the emotional state of the main character in the current video frame, recognize the facial expression of the character and quantify it into a seven-dimensional emotion probability vector . This module first locates and encodes the face region in the video frame, extracts the facial feature tensor , and outputs the emotion distribution vector through the pre-trained FER+ expression recognition network: W e f face b e In this formula, represents the emotion probability vector (emotion vector), with dimension R 7 , which corresponds to the confidence of the "happy, sad, surprised, angry, fearful, disgusted, neutral" 7 emotion categories respectively. This emotion vector is then transmitted to the multi-modal fusion module 305 to participate in the construction of the context vector. represents the function normalization of the input vector, so that its output is a probability distribution, the sum of all components is 1, and each component value is between 0 and 1. W e represents the weight matrix in the emotion recognition network that maps the facial feature vector to the emotion category space. Its dimension is 7 x d, where d is the dimension of the facial convolution feature vector . represents the feature vector of the facial image after convolutional neural network extraction, with dimension , which is the basic expression information of the model input. b e represents the bias vector (bias vector), with dimension , which introduces an offset adjustment parameter for each emotion category. Therefore, this formula represents that the facial feature vector is first linearly mapped to the score space of 7 emotion categories , and then mapped through Convert it into the probability of each emotion to form the final emotion probability vector e.

[0049] The dynamic rhythm analysis module 303 is responsible for modeling the motion of the characters in the video and the changes in the audio signal, extracting the action rhythm feature vector r. This module first calculates the pixel-level displacement velocity between video frames by the TV-L1 optical flow method, obtaining the average optical flow velocity ; Then extract the audio signal RMS energy change rate within the t second time window , and count the moving speed of the center point of the character detection box , the speed standard deviation and the proportion of silent area , and finally build the action rhythm vector . This five-dimensional vector is used to characterize the video rhythm characteristics, and then transmitted to the multi-modal fusion module 305 for joint modeling.

[0050]

[0051] The multi-modal fusion module 305 is the core feature fusion component of the mechanism, which uses the Cross-Attention Transformer structure to jointly model the three types of modal feature vectors (emotion), (action rhythm), (visual structure). The system first splices the three types of vectors: , forming a 25-dimensional (where ) context vector , representing the semantic, emotional and visual context of the current frame. The vector is then input into the scoring prediction module 306 as input to determine whether it is suitable for ad insertion operation.

[0052] The insertion suitability score prediction module 306 receives the context vector c from the fusion module 305, uses a double-layer MLP neural network to predict the score, and outputs the insertion suitability score of the current frame​​​​​​ The model is specifically structured as: wherein is a function for normalizing the output to the probability domain; is a two-layer weight matrix; is an activation function; and are bias coefficients. The score is used to quantify the user receptivity and context fit for performing the advertisement insertion operation at the current time. The score result is then transmitted to the insertion trigger control module 307.

[0053] The insertion trigger control module 307 serves as a decision module, which jointly determines whether to perform the advertisement insertion operation according to the score prediction result and the marked gap region. When the score value is greater than the threshold value, and the current frame is determined to be an insertable frame by the gap region confirmation module 208 in Embodiment One, the advertisement insertion process is immediately triggered, and the dynamic advertisement generation and fusion module described in FIG. 105 is called to perform content generation and visual embedding operations. At the same time, this module supports policy-level parameter configuration (such as different platform insertion threshold values, interval control, frequency limitation, etc.), to guarantee the visual naturalness and commercial effectiveness of the insertion.

[0054] To achieve the advertisement insertion suitability prediction based on multi-modal context modeling, we designed the context modeling and insertion intention evaluation module 104, and the entire content of Embodiment One and Embodiment Two is hereby incorporated by reference: 1. Video frame input module 301: Input: video source file, format is H.264 encoded MP4 file; Processing: use FFmpeg and OpenCV tools to decode, frame extraction, and audio separation processing on the video, to obtain image frame sequence and audio frame ; Output: frame image (1920x1080 RGB image) and corresponding audio segment (44100 Hz, mono PCM) are transmitted to modules 302-304.

[0055] 2. Facial expression recognition module 302: Input: image frame ; Processing: through face detection (based on MTCNN or RetinaFace), the face region is cropped and sent into the FER+ network to extract facial features (dimension 512), and then a 7-dimensional emotion vector is calculated through the classifier: 7 ;​​ Output: Emotion probability vector where each component corresponds to the emotion "Happy", "Sad", "Surprise", "Angry", "Fear", "Disgust", "Neutral".

[0056] 3. Dynamic Rhythm Analysis Module 303: Input: Image frame sequence , Audio ; Process: TV-L1 optical flow to extract target moving speed in image ; Audio RMS energy calculation Inter-frame L2 displacement of person bounding box center ; Optical flow speed standard deviation ; Silence frame proportion , when RMS is less than 0.02, it is considered as silence; Output: Rhythm vector ; 4. Scene Structure Modeling Module 304: Input: Image frame ; Process: Edge intensity : Laplacian transform of image, standard deviation statistics; focus score : Image clear area ratio; : Object number per image area detected by YOLOv8; : Calculate the entropy value of HSV color histogram; : Average value of left and right brightness difference of image; Output: Structure vector ; 5. Multimodal Fusion Module 305: Input: Emotion vector e, rhythm vector r, structure vector s Process: Initial fusion ; Cross-Attention Transformer: 3-layer Encoder, hidden size=64, multi-head=4; output context encoding vector c ∈R² 5 ; Output: c is passed to the score prediction module.

[0057] 6. Insertion Appropriateness Score Prediction Module 306: Input: Process: use two-layer MLP structure for score output, formula: Where , , is function; where the bias term is the parameter automatically learned in the model training process, used to improve the fitting ability and nonlinear expression effect of the model. The above bias vector and weight matrix are optimized on the training set through the back propagation algorithm, and the loss function is binary cross-entropy (Binary Cross-Entropy). The output .

[0058] 7. Insert trigger control module 307: Input: and the neutral frame determination result in Figure 208; judgment logic: when and the current frame is a legal neutral area, trigger the advertisement insertion strategy (can call the material generation and fusion module 105 and the control module 106 in embodiment one). Output: insertion execution signal.

[0059] About the data source and training settings used in the above embodiments.

[0060] 1. Data collection: Source: KVideo-3M dataset of a short video open platform; Selection rule: keep short videos containing advertisement insertion behavior annotations, a total of 32000 segments (6-30 seconds in length); Positive sample: user does not produce jump-out, high video completion rate at insertion time; Negative sample: produces obvious repulsive behavior or user jumps out after insertion; Segmentation ratio: training set: validation set = 8:2 (25600+6400).

[0061] 2. Model training configuration: Optimizer: AdamW Initial learning rate: 1e -4 ; Batch size: 64; Epochs: 20; Loss function: binary cross-entropy BCE.

[0062] 3. Performance indicators of the validation set: Accuracy: 89.3%; Harmonic mean of accuracy and recall (F1-score): 0.87; The comprehensive classification ability of the system at all thresholds (ROC AUC): 0.91; Inference latency: ≈ 26ms per frame processing (NVIDIA RTX 3090) Table 1 Comparison of technical effects of the present application and prior art In actual application tests, the advertisement insertion system of the present application is deployed in a certain short video platform, and a tourism Vlog video is selected as the test object. The video content is the scene of a user walking into a hotel entrance, with a smiling face in the picture, and the emotion recognition module identifies the "happy" emotion probability as 0.68 (i.e. ), indicating that the user's current emotion is positive; scene structure analysis shows that the background is stable, the architectural symmetry is high, and the overall picture is clear and concise; rhythm analysis results show that the picture flow speed is low and the audio RMS energy change is gentle, indicating that the current picture rhythm is slow and there is no dramatic action or emotional fluctuation. After the fusion of the above multi-modal features, the context vector score generated is 0.83, which is significantly higher than the insertion threshold set by the system (0.7), so the system decides to insert a hotel recommendation poster advertisement consistent with the scene content style at this moment, which is fused into the lower right corner of the picture in a semi-transparent overlay manner. According to user behavior feedback statistics, the advertisement click-through rate of this video has increased by 27.6%, the video completion rate has increased by 14%, the exit rate has decreased by 18%, and the user's subjective score has increased from the traditional method's average of 3.1 / 5 to 4.4 / 5, verifying the effectiveness and user acceptance of the context-aware insertion mechanism constructed by the present application in the actual environment.

[0063] The above embodiments significantly improve the context adaptability and user acceptance of advertisement insertion by modeling facial emotions, action rhythm, and visual structure for multi-modal fusion analysis. Compared with traditional single rule methods, the present application not only improves the prediction accuracy, but also enhances the naturalness of insertion behavior and commercial conversion efficiency, making it suitable for deployment and application in short video platforms such as Douyin, Kuaishou, etc.

Claims

1. A video advertisement intelligent insertion method based on semantic segmentation and multimodal fusion, characterized in that: The following steps are involved: S1, extract frames from the input video stream to obtain an image frame sequence; S2. Perform pixel-level recognition on the image frame using a semantic segmentation model to generate a semantic mask map and identify non-critical semantic areas; S3, based on the pixel density of the idle area in consecutive frames, combined with the time sliding window statistics and structural similarity ( ) and brightness changes ( ) Analyze and identify advertising gaps that meet spatial and temporal continuity; S4. Extracting multimodal context features from the frames corresponding to the blank area, including emotion vectors generated by facial expressions, video rhythm vectors, and image structure vectors; S5. Input the multimodal features into the fusion network for context modeling to generate the insertion suitability score; S6. When the score exceeds the set threshold and the frame is a legal blank area, perform advertisement generation and visual fusion operations to embed the advertisement into the corresponding frame.

2. The method according to claim 1, wherein: The semantic segmentation model is The output mask image is labeled with at least three semantic labels: people, goods, and background, which are used to delineate non-critical areas.

3. The method according to claim 1, wherein: The gap area identification is based on A time sliding window of frame length, counting no less than Frames that meet the insertable region density threshold; .

4. The method according to claim 1, wherein: The background stability determination requires the structural similarity between consecutive frames. , RGB brightness changes .

5. The method according to claim 1, wherein: The emotion vector is a seven-dimensional probability vector, corresponding to the recognition results of seven emotion categories.

6. The method according to claim 1, wherein: The multimodal fusion adopts a Cross-Attention Transformer structure to output a 25-dimensional context semantic vector for insertion into rating prediction.

7. A video advertisement insertion system for executing the method according to any one of claims 1 to 6, characterized in that: Includes the following modules: A video pre-processing and framing module (101) is used to receive an original video stream and perform frame extraction processing to generate a continuous image frame sequence; The target detection and semantic segmentation module (102) is used to perform target recognition and semantic segmentation on each frame of image and output a pixel-level mask image; A gap point recognition module (103) is used to analyze the pixel ratio of the non-critical area based on the mask image, and to identify the gap area that meets the conditions for advertisement insertion by combining the time sliding window and background stability; Context modeling and insertion intention assessment module (104), including expression recognition, rhythm analysis, and structure modeling submodules, is used to extract multimodal context features and fuse them to generate insertion suitability scores; A dynamic advertisement generation and fusion module (105) is used to select or generate advertisement content based on the blank area and context information, and to fuse and embed the advertisement image into the video frame; The real-time rendering and delivery control module (106) is used to control the advertisement insertion frequency, presentation mode and final rendering and output operations on the user device side.

8. The video advertisement insertion system according to claim 7, wherein: The neutral point recognition module (103) includes the following submodules: A video input module (201) is used to receive raw video data and parse metadata such as frame rate and resolution; A frame extraction module (202), configured to convert the video data into an image frame sequence; The semantic segmentation module (203) is used to perform pixel-level semantic segmentation on each frame of image based on the pre-trained Mask R-CNN model and output a semantic mask map ( ); The gap candidate mask generation module (204) is used to identify non-key semantic areas from the semantic mask map and generate a binary mask ( ); The area calculation module (205) is used to calculate the binary mask ( ) in the pixel ratio of the insertable area ; The continuous frame window determination module (206) is used to determine whether there are at least Frame Satisfaction , thereby determining the continuous gap candidate area; Background stability analysis module (207) for calculating structural similarity in the gap area and brightness changes , to judge the stability of background changes; The blank area confirmation module (208) is used to perform a joint logical judgment on the results from the area calculation module (205), the continuous frame window determination module (206) and the background stability analysis module (207), and output the final advertisement insertion mask frame.

9. The video advertisement insertion system according to claim 7, wherein: The context modeling and insertion intention assessment module (104) includes: A facial expression recognition module (302) extracts facial image features based on a pre-trained model and outputs an emotion vector containing seven basic emotion confidences for describing the emotional state of the current person; A dynamic rhythm analysis module (303) is used to generate a multi-dimensional rhythm vector reflecting the rhythm characteristics of the picture based on indicators such as target motion between image frames, energy changes of audio signals, character movement speed and silence ratio; A scene structure modeling module (304) is used to extract image features such as texture complexity, clarity, scene crowding, color information richness, and left-right brightness symmetry, for depicting visual structure; A multimodal fusion module (305), wherein the multimodal fusion module (305) semantically fuses the three types of vectors, namely emotion, rhythm and structure, through a cross-attention mechanism to obtain a comprehensive context representation of the current frame; An insertion suitability score prediction module (306), wherein the insertion suitability score prediction module (306) evaluates the contextual fit of the current moment of advertisement insertion based on the fused context representation vector and outputs a score value representing the insertion suitability; An insertion trigger control module (307) triggers an advertisement insertion action under the premise that the score value reaches a preset threshold and the current frame has been marked as a blank area.

10. A computer-readable storage medium having a computer program stored thereon, which, when the program is executed in a processor, enables the computer to execute all the steps of the method according to any one of claims 1 to 6.

Images