A dynamic advertisement delivery method based on end-side AI real-time content analysis

Abstract

The application relates to the technical field of digital advertising, in particular to a dynamic advertisement putting method based on end-side AI real-time content analysis. The application constructs a lightweight scene monitoring model based on an optical flow method and interframe difference, calculates the motion saliency of a video stream in real time, and triggers key frame extraction; a main analysis model is used to perform instance segmentation on the key frame, to generate a pixel-level mask and a refined boundary of a target subject, and to perform region-of-interest cutting and background elimination based on the mask; a hybrid semantic analysis model is used to extract explicit discrete labels and implicit high-dimensional visual feature vectors in parallel, to capture texture, material and style attributes of the target; end-side knowledge graphs and visual dominance scores are combined to perform semantic consistency verification and logic conflict resolution on the recognition result, to generate visual feature representation; finally, based on histogram analysis and edge detection, a visual interaction floating layer with adaptive transparency and a proper position is rendered on the video layer, and digital advertising is put to users.

Description

Technical Field

[0001] This invention relates to the field of digital advertising technology, specifically to a dynamic advertising delivery method based on real-time content analysis using edge AI. Background Technology

[0002] General visual recognition systems often rely on predefined discrete label systems for coarse-grained full-image classification or simple object detection. They can only identify basic categories like "cars" and "clothing," lacking the ability to perceive the fine-grained visual attributes of the target subject. Specifically, they struggle to effectively extract material textures (e.g., distinguishing the surface features of leather from canvas), design styles (e.g., the combination of retro and modern visual elements), and the overall lighting and atmosphere of the image. This shallow recognition, limited to the category level, prevents the system from differentiating visually similar but semantically distinct targets, frequently resulting in label conflicts or semantic ambiguity, failing to meet the demand for precise retrieval based on deep visual features. Regarding target segmentation and region of interest (ROI) localization, existing anchor point localization techniques typically use rectangular bounding boxes to define the target. This coarse geometric approximation cannot fit the irregular edges of non-rigid objects (e.g., human silhouettes, fluids), resulting in extracted regions containing a large amount of invalid background pixel noise. During dynamic visual component rendering, this localization error can easily cause overlays to obscure key visual information in the video (e.g., subtitle areas, facial expressions, or gestures), disrupting the compositional integrity of the image and severely interfering with the user's immersive visual experience.

[0003] To address the technical challenge of achieving semantic and logical consistency verification of target objects in video streams under the constraints of mobile terminal devices (limited CPU, memory, and power consumption), enabling visual analysis tasks to be completed based on a lightweight edge-side model, a dynamic advertising delivery method based on edge-side AI real-time content analysis is proposed. Summary of the Invention

[0004] This invention aims to provide a dynamic advertising delivery method based on real-time content analysis using edge AI. It utilizes an edge computing architecture and a hybrid semantic analysis model to achieve real-time, pixel-level fine-grained semantic understanding and logical consistency verification of video streams. The method is executed on the terminal device side and includes the following steps:

[0005] Based on inter-frame differences, a lightweight scene monitoring model is constructed using optical flow to continuously monitor the currently playing video stream. When the video stream meets the preset trigger conditions, key frames are extracted from the video stream, and the main analysis model is used to perform subject detection and segmentation on the key frames to determine the boundaries of the subjects in the key frames.

[0006] A close-up image of the main subject is cropped from the keyframe according to the boundary; a hybrid semantic analysis model is used to perform forward reasoning on the cropped close-up image of the main subject and output semantic representation data in parallel; based on the lightweight knowledge graph on the edge, semantic logic verification and optimization are performed on the semantic representation data to generate visual feature representation;

[0007] Based on the visual feature representation, a privacy-preserving product request is generated and sent to the advertising server. The returned recommended product data is received, and an intelligent interactive overlay is rendered on the video playback interface of the terminal device.

[0008] Preferably, the determination of the preset triggering condition includes the following steps:

[0009] The optical flow field amplitude and direction distribution of inter-frame pixel displacement are used as input features to calculate the optical flow statistical histogram of the current frame and the previous N frames. An optical flow amplitude threshold and a direction entropy threshold are set to jointly judge the optical flow statistical histogram to detect in real time whether the video screen is in a stable state and whether there is low motion blur. A lightweight focus tracking algorithm is used to record the number of screen dwell frames of the same target. When the number of dwell frames exceeds a preset threshold, it is determined that there is a stable focus object.

[0010] Preferably, the lightweight scene detection model is a 3-layer convolutional neural network (CNN), and the training and usage steps include:

[0011] A video frame dataset labeled with analyzable binary tags is used to supervise the training of a convolutional neural network, which outputs a continuous analyzability score between 0 and 1. The score comprehensively evaluates image stability, blur level, subject salience, and lighting suitability. When the real-time calculated analyzability score is greater than a preset threshold, keyframe extraction is triggered.

[0012] Preferably, the main analysis model is YOLOv8n-seg, which has been optimized for mobile devices. The step of determining the boundary of the subject in the keyframe specifically includes:

[0013] Using the master analysis model, a single instance segmentation inference task is performed on the extracted keyframes; the pixel-level masks and bounding box coordinates of all identified subjects in the image are output, along with the subject category confidence; based on the subject category confidence, the core subjects with the highest confidence and belonging to the category of goods or wearable items are selected; based on the pixel-level masks of the core subjects, the pixel-level boundaries of individual target subjects are determined.

[0014] Preferably, after determining the boundaries of the subject, the method further includes the following steps:

[0015] For all independent subjects in the same keyframe whose confidence exceeds a preset threshold, a minimum tight bounding box is calculated based on their independent pixel-level mask. According to the minimum tight bounding box, the corresponding multi-subject close-up regions are cropped from the keyframe. Morphological operations and edge feathering are used to optimize the cropped close-up regions, remove background pixel interference caused by bounding box expansion, and generate a set of subject close-up images without background interference.

[0016] Preferably, the hybrid semantic analysis model adopts a hybrid semantic feature extraction architecture, and the specific construction and execution steps include:

[0017] The pre-trained MobileCLIP is used as the backbone network, and fine-tuned through transfer learning and contrastive learning using a multi-label dataset from a product vertical domain. The fine-tuned model is then used to perform forward inference on the cropped close-up image of the subject, outputting two types of semantic representation data in parallel: the first type is an explicit structured label group, which includes discrete-dimensional labels such as subject category, brand, color, style, material, and scene, for accurate keyword matching; the second type is an implicit dense semantic vector, which maps image features to a high-dimensional continuous vector space, capturing unstructured semantic information including retro style, texture, and atmosphere features, supporting fuzzy semantic matching based on vector similarity.

[0018] Preferably, the step of performing semantic logic verification and optimization based on the edge-side lightweight knowledge graph specifically includes:

[0019] A lightweight knowledge graph and hierarchical classification tree are constructed on the client side. The hierarchical classification tree defines the hierarchical inclusion relationship between tags to standardize the semantic granularity of tags. The knowledge graph defines the partial overall relationship and co-occurrence probability matrix between tag entities. The co-occurrence probability matrix quantifies the semantic association strength and logical compatibility of different tags in the same spatiotemporal scenario.

[0020] Preferably, the semantic verification and query optimization steps further include performing multi-dimensional tag cleaning and logical conflict resolution:

[0021] First, a confidence-based initial screening is performed. Based on the confidence values ​​output by the hybrid semantic analysis model, an adaptive threshold truncation algorithm is applied to select a set of candidate labels. Second, a graph-based semantic logic verification is performed. The label pairs in the candidate label set are traversed, and the co-occurrence probability matrix is ​​queried. When the co-occurrence probability of two labels is detected to be lower than a preset logical mutual exclusion threshold, a semantic conflict is determined to exist.

[0022] Finally, an arbitration based on visual dominance is performed. For label pairs with semantic conflicts, their visual dominance scores in keyframes are calculated. The scores are calculated by weighting the pixel area ratio, subject center distance, and texture sharpness. The pair with the higher visual dominance score is retained, and logically paradoxical labels are removed to generate visual feature representations.

[0023] Preferably, the step of generating a privacy-protected product request includes:

[0024] The generated visual feature representation is hashed and / or anonymized using the SHA-256 algorithm, retaining only the tag ID and not uploading any original images or user profile data; a lightweight JSON request is constructed with the processed tag as the query parameter and sent to the ad server via HTTPS.

[0025] Preferably, the step of rendering the intelligent interactive overlay on the video playback interface of the terminal device includes:

[0026] The real-time HSV / brightness histogram of video frames is dynamically fused with the background layer of the advertising component using the Alpha channel, and the component transparency is automatically adjusted according to the average brightness of the current screen.

[0027] Edge detection is used to avoid the face and subtitle areas, and the floating position of the AI ​​assistant is intelligently adjusted and activated. The AI ​​assistant, combined with the edge-side small language model (SLM), provides a conversational intelligent agent to the user.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1. By leveraging the synergistic effect of a hybrid semantic analysis model and knowledge graph conflict resolution, under the strict computational resource constraints of mobile devices (CPU utilization <15%, memory <200MB), the accuracy of recommended tags increased from 65% to 92%, and the conflict rate decreased from 12% to <1%. Through the search space compression (80% dimensionality reduction) caused by knowledge graph conflict resolution, this invention achieved a 20% reduction in power consumption.

[0030] 2. Compared to traditional solutions that rely on cloud communication, this invention completely offloads core computing to the edge for execution. This reduces network data transmission by over 99.8%, and the recommended end-to-end latency is reduced from 500-1000ms to <100ms.

[0031] 3. Unlike traditional cloud-based analytics solutions, all core computational steps of this invention—from scene monitoring and subject segmentation to semantic extraction and conflict arbitration—are completed in a closed loop on the terminal device. The system only sends hash-encrypted or anonymized visual feature representations (such as tag IDs or embedding vectors) to external servers, without uploading any original video images or user profile data. This "data-on-the-end" architecture fundamentally cuts off the path for the outflow of sensitive user information, fully complying with current stringent data security regulations. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a dynamic advertising delivery method based on real-time content analysis using edge AI according to the present invention.

[0033] Figure 2 This is a flowchart illustrating the trigger condition determination logic of Embodiment 1 of the present invention;

[0034] Figure 3 This is a schematic diagram of the intelligent floating layer rendering decision-making process in Embodiment 1 of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please see Figures 1 to 3 This invention provides a dynamic advertising delivery method based on real-time content analysis using edge AI. The technical solution is as follows, please refer to... Figure 1 The flowchart includes:

[0037] Based on inter-frame differences, a lightweight scene monitoring model is constructed using optical flow to continuously monitor the currently playing video stream. When the video stream meets the preset trigger conditions, key frames are extracted from the video stream, and the main analysis model is used to perform subject detection and segmentation on the key frames to determine the boundaries of the subjects in the key frames.

[0038] A close-up image of the main subject is cropped from the keyframe according to the boundary; a hybrid semantic analysis model is used to perform forward reasoning on the cropped close-up image of the main subject and output semantic representation data in parallel; based on the lightweight knowledge graph on the edge, semantic logic verification and optimization are performed on the semantic representation data to generate visual feature representation;

[0039] Based on the visual feature representation, a privacy-preserving product request is generated and sent to the advertising server. The returned recommended product data is received, and an intelligent interactive overlay is rendered on the video playback interface of the terminal device.

[0040] Example 1

[0041] This embodiment applies to a "dynamic streaming media digital advertising insertion system" running on a smart mobile terminal (such as a smartphone or tablet). When a user watches a "cooking tutorial" or "travel scenery" video stream at 30 frames per second and a resolution of 1920×1080 through a video app, the system needs to determine in real time on the device whether the current scene is suitable for inserting a virtual advertisement (e.g., placing a virtual bottle of beverage on the desktop). In order to accurately capture the best insertion time without affecting the viewing experience and without consuming too much computing power, the system adopts the lightweight scene monitoring method described in this embodiment.

[0042] Reference Figure 2 The flowchart illustrates the trigger condition determination logic. First, the lightweight scene monitoring model constructed using optical flow based on inter-frame differences includes: calculating the two-dimensional displacement vector of each pixel between adjacent frames frame-by-frame using a dense optical flow algorithm (such as the Farneback algorithm or the TV-L1 algorithm); downsampling the video frame resolution to 320×180 pixels to reduce computational load; traversing each pixel point on the output optical flow field matrix to calculate the magnitude (amplitude) and direction angle of its optical flow vector; dividing the optical flow amplitude into multiple statistical intervals (e.g., 10 intervals) and the direction angle into multiple direction intervals (e.g., 8 directions), statistically analyzing the pixel proportion of each interval, and generating an optical flow amplitude histogram and a direction histogram as input features for subsequent trigger determination. Continuously monitoring the currently playing video stream, specifically, the system first establishes a first-in-first-out frame buffer queue with a capacity of 5 frames, used to store the current time t and past... to For each moment of the video grayscale image, the processor converts the acquired RGB video stream into a single-channel grayscale image in real time and downsamples the resolution to 320×180 pixels to reduce the computational load. Then, it calls the Gunnar Farneback dense optical flow algorithm, using the current frame and the previous frame as input pairs, to calculate the optical flow vector pixel by pixel, and outputs a two-dimensional optical flow field matrix containing horizontal displacement component dx and vertical displacement component dy. The size of this matrix is ​​consistent with that of the downsampled image.

[0043] Furthermore, using the optical flow field amplitude and direction distribution of inter-frame pixel displacement as input features, the system calculates the optical flow statistical histogram of the current frame and the previous N frames. Specifically, for the two-dimensional optical flow field matrix output by the previous step, the system traverses each pixel and calculates the magnitude (i.e., amplitude) and arctangent (i.e., direction angle) of its optical flow vector. The system divides the amplitude range into 10 statistical intervals (e.g., 0 to 2 pixels, 2 to 4 pixels, etc.) and the direction angle range from 0 to 360 degrees into 8 direction intervals (each interval is 45 degrees). The processor counts the number of pixels falling into each interval and normalizes the number of pixels in each interval by dividing it by the total number of pixels, generating two normalized one-dimensional arrays: an optical flow amplitude histogram vector of length 10 and an optical flow direction histogram vector of length 8. The system performs the above operation on the most recent 5 frames of data in the buffer queue and calculates the arithmetic mean of the histogram vectors of these 5 frames, which is used as the smoothed optical flow statistical feature of the current time window.

[0044] Furthermore, an optical flow amplitude threshold and a directional entropy threshold are set to jointly judge the optical flow statistical histogram. Specifically, the system presets an amplitude static threshold of 0.15 (representing that the overall average displacement of the image is extremely small) and a directional entropy blur threshold of 1.5. The processor first calculates the cumulative proportion of the low amplitude range of "0 to 2 pixels" in the smoothed optical flow amplitude histogram vector. If this proportion exceeds 85%, the image is determined to be in a physically stable state. Subsequently, the system calculates the information entropy of the optical flow direction histogram vector. Specifically, the probability value of each direction interval is multiplied by its natural logarithm, and the sum is taken as the absolute value. If the calculated information entropy value is less than the directional entropy blur threshold, it indicates that the motion direction in the image is consistent (such as a panning shot) and there is no messy low motion blur. Only when the above two conditions are met simultaneously will the system generate a "candidate stable frame" mark and proceed to the next level of judgment.

[0045] The physical meaning of the optical flow amplitude stationary threshold of 0.15 is as follows: In an image coordinate system downsampled to 320×180 resolution, the average pixel displacement between adjacent frames does not exceed 0.15 pixels, corresponding to an average displacement of approximately 0.9 pixels at the original 1920×1080 resolution. This threshold is based on the following understanding: For a 30fps video stream, a 0.9-pixel inter-frame displacement corresponds to a physical motion speed of approximately 27 pixels per second. For typical display devices, motion changes at this speed are almost imperceptible to the human eye, reaching the physiological threshold for visual stability. This threshold can be linearly adjusted according to the frame rate. The information entropy of the optical flow direction histogram is calculated based on the Shannon entropy principle; its specific value is equal to the negative sum of the products of the probabilities of all direction intervals and their corresponding logarithmic probabilities. For the case with eight direction intervals, the theoretical maximum information entropy value is approximately 2.079.

[0046] The physical meaning of the directional entropy blur threshold of 1.5 is: a mapping relationship between information orderliness and image sharpness based on an eight-directional interval division. Theoretically, when the image motion directions are completely randomly distributed (i.e., each direction accounts for 12.5%), the system is in a state of maximum disorder. At this point, due to the irregular spatial dispersion of pixels, severe motion blur inevitably occurs, leading to the loss of edge details. Setting the threshold to 1.5 essentially defines a "orderliness" boundary: it requires that the motion vector of the current frame must be highly concentrated in a few specific directional dimensions. This high degree of directional consistency typically corresponds to stable panning shots or uniform subject movement. In this case, the photosensitive path of the pixels has strong directional selectivity, effectively suppressing visual interference caused by random shaking and ensuring that the object contours in the keyframes have the physical conditions for high-precision segmentation by the edge AI.

[0047] When the calculated information entropy value is less than the set threshold (1.5), it indicates that the optical flow direction is highly concentrated in a specific direction, such as in a panning shot scene; at this time, it indicates that the motion pattern is clear and the background extraction is relatively reliable. Combining this information entropy condition with the optical flow amplitude condition, it can be determined that the video image is in a stable state with low motion blur.

[0048] Furthermore, a lightweight focus tracking algorithm is used to record the number of screen dwell frames for the same target. Specifically, the system defines the central area of ​​the screen (e.g., 30% to 70% of the width and height) as the core area of ​​interest. The CamShift tracking algorithm based on color histograms is used to initialize the dominant color feature within the core area of ​​interest. In the continuous video stream, the system calculates the centroid coordinates of the dominant color feature area frame by frame. If the cumulative Euclidean distance of the centroid position within 15 consecutive frames (i.e., 0.5 seconds) is less than 20 pixels, the counter is incremented. When the value of the counter exceeds the preset dwell frame threshold (set to 15 frames in this embodiment; in a 30fps video stream, 15 frames correspond to a visual dwell time of 0.5 seconds, which is the minimum closed loop for determining visual focus, filtering out interference from objects that flash by instantly, and ensuring that the target object has formed a stable visual dwell in the core area of ​​the screen), the system determines that there is an object with a stable visual focus in the current image. At this time, combined with the "candidate stable frame" marker from the previous step, the keyframe extraction instruction is formally triggered, and the RGB frame of the current original resolution is sent into the deep learning model.

[0049] Furthermore, a three-layer convolutional neural network (CNN) is used as a lightweight scene detection model to score the analyzability of extracted keyframes. Specifically, the input layer of this model receives a three-channel RGB image with a size adjusted to 128×128 pixels, and the data is normalized to between 0 and 1. The first layer is a convolutional layer with 32 convolutional kernels of size 3×3, a stride of 2, and the activation function ReLU, followed by a 2×2 max pooling layer, and the output feature map size is 32×32×32. The second layer is a convolutional layer with 64 kernels of size 3×3. The first layer consists of 3×3 convolutional kernels with a stride of 1 and the ReLU activation function, connected to a 2×2 max-pooling layer, resulting in an output feature map of size 16×16×64. The second layer is a convolutional layer with 128 3×3 convolutional kernels, a stride of 1, and the ReLU activation function, connected to a global average pooling layer, compressing the spatial features into a one-dimensional feature vector of length 128. The output layer is a fully connected layer containing one neuron, using the Sigmoid activation function to map the output value to a continuous floating-point number between 0 and 1, which is the "analyzability score".

[0050] Furthermore, the model is trained under supervised training based on labeled data, and subsequent processes are triggered based on the score. Specifically, during the model training phase, a dataset containing 10,000 video screenshots is constructed. Each image is manually labeled with an analyzability tag ("1" represents a clear image, prominent subject, uniform lighting, and no severe blur; "0" represents a jittery image, too dark, or no subject). The binary cross-entropy loss function and Adam optimizer are used to iteratively update the network weights. The initial learning rate is 0.001, and the batch size is 32, until the validation set accuracy reaches over 95%. During the real-time inference phase, when the analyzability score of the current frame calculated by the system is greater than the preset threshold of 0.85 (representing a comprehensive weight balance between image quality and recognition reliability; in the 0 to 1 scoring system, 0.85 requires the image to reach a high confidence level in four dimensions: stability, clarity, subject prominence, and lighting conditions), the frame is determined to fully meet the visual requirements for ad insertion. The system then locks the keyframe and transmits it to the subsequent ad rendering module via the memory bus for digital content overlay.

[0051] By combining the optical flow statistical histogram with the output analyzability score, a refined joint screening of video image stability, blurriness, and lighting conditions is achieved, effectively filtering out invalid frames such as high-speed motion or meaningless transition shots. This hierarchical triggering mechanism based on image quality and visual focus dwell time ensures that high-computing analysis is only initiated when there is a stable focus and the image quality meets the standards, thus reducing the invalid computing load and power consumption of the terminal device while ensuring the accuracy of subsequent recognition.

[0052] Furthermore, the master analysis model is used to perform subject detection and segmentation on the keyframes to determine the boundaries of the subjects in the keyframes. Specifically, the system calls the YOLOv8n-seg model, which has been quantized and accelerated for mobile GPUs (such as Adreno or Mali series), from the video memory. This model is a single-stage instance segmentation network. The input receives RGB keyframe images with a resolution of 640×640 pixels selected in the previous steps. Multi-scale features are extracted through a backbone network containing 225 layers, and then feature fusion is performed through a feature pyramid network. Finally, the prediction results are output on three detection heads at different scales. Each prediction result contains two parts of information: one is the bounding box coordinates (X center, Y center, width, height) and class confidence used to locate the object, and the other is the prototype vector coefficients used to generate pixel-level masks. The model can output the segmentation results of all potential targets in the image in parallel during a single forward inference process (which takes about 30 milliseconds).

[0053] Furthermore, the output image contains pixel-level masks and bounding box coordinates for all identified subjects, along with subject category confidence scores. Specifically, the model output layer undergoes non-maximum suppression post-processing to filter out redundant boxes with an overlap greater than 0.45, ultimately generating a target list. Assuming the current keyframe is an "outdoor sports" scene image, the list may include: "person" (confidence 0.92), "backpack" (confidence 0.88), "sports shoes" (confidence 0.85), etc. For each target in the list, the model decodes a binary mask image with the same width and height as the input image. In this mask image, pixels belonging to the target have a value of 1, and pixels belonging to the background have a value of 0. The model also outputs the bounding box coordinates of the target, for example, the bounding box of the backpack is [top left corner (200, 350), bottom right corner (400, 550)].

[0054] Furthermore, the system filters out the core subjects with the highest confidence level and belonging to the category of goods or wearable items. Specifically, the system pre-sets an "advertising value category whitelist," which includes category IDs with commercial embedding value such as "bottles," "handbags," "electronic products," and "shoes and clothing." The processor traverses the target list, first removing categories not belonging to the whitelist (such as "sky" and "grass"). Then, among the remaining targets, they are sorted in descending order according to their confidence level. The target with the highest confidence level (e.g., "backpack" with 0.88) and whose value is greater than a preset basic threshold (e.g., 0.6, specifically set based on the probability distribution characteristics of the target detection model's output results. In deep learning inference, 0.5 is usually considered the critical point between random classification and confident classification. Setting the threshold to 0.6 adds redundancy and requires the recognition result to have clear statistical significance) is selected as the "core subject." If multiple high-confidence targets exist, the system marks them as a "multi-subject set" and retains them for subsequent processing.

[0055] Furthermore, based on the pixel-level mask of the core subject, the pixel-level boundary of a single target subject is determined. Specifically, for a selected "core subject" (such as a backpack), the processor extracts its corresponding binary mask image and uses the Canny edge detection operator or contour finding algorithm (such as findContours) to trace along the boundary line where the pixel value in the mask image jumps from 0 to 1, thereby obtaining the precise pixel-level contour path of the subject. This path consists of a series of two-dimensional coordinate points that closely fit the edge of the backpack, eliminating background pixels in the gaps of the backpack handles and ensuring the accuracy of the boundary.

[0056] Furthermore, based on the minimum tight bounding box, corresponding multi-subject close-up regions are cropped from the keyframes. Specifically, for all independent subjects (e.g., backpacks and sneakers) with a confidence level exceeding 0.6 in the same keyframe, the processor calculates the minimum bounding rectangle (i.e., minimum tight bounding box) of their pixel-level masks. To prevent the loss of edge information, the system adds a 10-pixel buffer margin on each of the calculated rectangle's top, bottom, left, and right sides. Then, using the coordinates of this expanded rectangle, the system calls the ROI (Region of Interest) cropping function of an image processing library (such as OpenCV) to cut out backpack close-up patches of 220×220 pixels and sneaker close-up patches of 150×100 pixels from the original RGB keyframe with a resolution of 1920×1080.

[0057] Furthermore, morphological operations and edge feathering are employed to optimize the cropped close-up areas. Specifically, since cropping is based on a rectangular frame, the close-up image inevitably contains some background pixels that are not the main subject (such as the grass next to the backpack). To generate clean material, the processor performs an AND operation between the close-up image and the corresponding binary mask, forcing the background area with a mask value of 0 to be filled with a transparent color (Alpha=0). Subsequently, to address any jagged edges that may exist at the mask edges, the system applies a 3×3 Gaussian blur kernel for edge feathering, or performs a morphological opening operation (erosion followed by dilation) to smooth the edge jaggedness. Finally, a set of PNG format close-up images of the main subject with completely transparent backgrounds and soft edges without background interference is generated. This set is then transmitted to the subsequent image retrieval module for product matching.

[0058] By using a mobile-optimized instance segmentation model to obtain pixel-level masks of the core subject, and combining them with minimum tight bounding boxes and edge feathering, the irregular edges of non-rigid objects are accurately stripped and background pixel noise introduced by selection box expansion is removed. This refined processing effectively solves the background interference problem caused by traditional rectangle detection, generating high-quality close-up images with smooth edges and prominent subjects, providing a clean input source with high signal-to-noise ratio for subsequent feature extraction.

[0059] Furthermore, a pre-trained MobileCLIP is used as the backbone network, and transfer learning and contrastive learning are performed to fine-tune it using a multi-label dataset from a vertical product domain. Specifically, the system constructs a lightweight hybrid semantic analysis model, whose core architecture is based on MobileCLIP-S2, a mobile-optimized version of the OpenAICLIP model. This model includes an image encoder and a text encoder. The image encoder uses a modified ViT-B / 16 (VisionTransformer) structure, with the number of layers reduced to 12, and the input resolution adapted to 224×224 pixels. The text encoder uses Transformer... The Sformer architecture has a maximum sequence length of 77 tokens. During training, the system loads pre-trained weight parameters on the LAION-400M dataset and then fine-tunes them using a vertical domain dataset containing 500,000 pairs of "product images - detailed attribute description text". The fine-tuning process uses a contrastive loss function, which maximizes the cosine similarity between matched product images and description texts in the same batch (BatchSize=128) while minimizing the similarity between unmatched pairs. The batch size is 128, the initial learning rate is 0.0001, the temperature parameter is 0.07, and a cosine annealing learning rate decay strategy is used.

[0060] Furthermore, the fine-tuned model is used to perform forward inference on the cropped close-up image of the subject, and output two types of semantic representation data in parallel. Specifically, the processor receives a close-up image of the subject without background (e.g., a red backpack) with a size adjusted to 224×224 pixels generated in the previous step, and inputs it into the fine-tuned image encoder. After processing through layers of self-attention mechanisms, the model extracts the high-level semantic features of the image. At the output end, the system designs two parallel branch structures: one for generating a classification head for explicit labels and the other for generating a projection head for implicit vectors.

[0061] Following the cropping step, a non-modal completion module based on GAN (Generative Adversarial Network) is introduced to upgrade the "close-up subject image" into a semantically complete reconstructed subject image; specifically, this is achieved through a lightweight DeepFill or EdgeConnect variant network. This network, based on prior knowledge of the visible texture features and object structure, performs pixel-level "illusion" reconstruction of missing areas occluded by foreground objects (such as tables or arms). For example, when recognizing only the upper body of a "trench coat," the network automatically completes the outline and texture of the hem, generating an unoccluded, complete object view, which is then fed into MobileCLIP for feature extraction. This effectively solves the technical challenge of significantly reduced recognition rates in scenarios where objects are partially occluded. By restoring incomplete visual input to a complete semantic object, the accuracy and robustness of subsequent vector generation are greatly improved, ensuring that even products located at the edge of the image or those that are occluded can be accurately identified and recommended.

[0062] Furthermore, the first output category is an explicit structured label group, containing discrete-dimensional labels including the main product category, brand, color, style, material, and scene. Specifically, the classification head branch consists of a fully connected layer that maps image features to a 2000-dimensional label space, corresponding to 2000 predefined product attributes (such as "backpack," "Nike," "red," "retro," "canvas," "outdoor," etc.). The system executes a Sigmoid activation function on the output of this fully connected layer, mapping the value of each dimension to a probability value between 0 and 1. The processor sets a confidence threshold (e.g., 0.5, which is based on the equilibrium point of the probability distribution in binary logistic regression, as defined in MobileCLIP). In the output layer of the model, 0.5 serves as the theoretical neutral value for determining whether a label is valid. It represents the equiprobability threshold between "predicted existence" and "predicted non-existence." Setting the threshold to 0.5 aims to ensure that each selected dimension index has a statistically positive attribution certainty, thereby preserving as rich a fine-grained semantic feature as possible while initially filtering out irrelevant label noise with extremely low probability. The output vector is traversed to select all dimension indices with probability values ​​greater than the threshold and convert them into corresponding text labels. Finally, a set of structured JSON data is output, such as {"Category": "Backpack", "Color": "Red", "Material": "Canvas", "Style": "Vintage"}. This set of labels is directly used for SQL precise keyword queries in the advertising database.

[0063] Furthermore, the second type of output is an implicit dense semantic vector, which maps image features to a high-dimensional continuous vector space. Specifically, the projection head branch consists of a multilayer perceptron (MLP) containing two fully connected layers and a GELU activation layer. This branch projects the original feature vector output by the image encoder into a joint embedding space of dimension 512. The system performs L2 normalization on the final output 512-dimensional vector (i.e., divides the vector by its magnitude) to make it a unit vector. This unit vector is the implicit dense semantic vector. It not only encodes the fact that there is a "red backpack", but also implicitly encodes unstructured information that is difficult to fully express in words, such as the "roughness of the texture", "smoothness of the design lines" and "overall nostalgic atmosphere". This vector is transmitted to a vector search engine (such as Faiss) to retrieve advertising materials with highly similar appearance styles (such as another retro backpack that is blue but has a very similar design style) through cosine similarity calculation.

[0064] A hybrid semantic feature extraction architecture is employed to output explicit structured labels and implicit dense semantic vectors in parallel. This approach retains the determinism of keyword matching while capturing deep, unstructured information, including retro style and texture. This dual representation mechanism effectively bridges the semantic gap in traditional classification algorithms, supports fuzzy matching based on vector similarity, and can accurately associate products that are similar in appearance but have different styles, greatly improving the diversity and accuracy of recommendations.

[0065] Furthermore, a lightweight knowledge graph and hierarchical classification tree are constructed on the device side. Specifically, the system deploys a simplified knowledge graph in the local storage space of the mobile terminal (such as an SQLite database or NoSQL key-value pair store). This graph contains two core data structures: the first part is a hierarchical classification tree, with the root node being "product," branching downwards to first-level categories such as "clothing," "electronics," and "home furnishings," further subdivided into second-level categories such as "tops" and "skirts," down to specific leaf nodes such as "pleated skirts" and "A-line skirts." This tree structure is linked through parent and child pointers. The first part establishes the hierarchical inclusion relationship of "dresses belong to skirts, and skirts belong to clothing," which is used to replace overly broad labels (such as "clothes") with more specific labels (such as "dress"). The second part is the co-occurrence probability matrix, which is a sparse matrix with a dimension of 2000×2000. The rows and columns correspond to all possible label IDs. The element values ​​in the matrix (between 0 and 1) represent the probability of two labels appearing simultaneously in a large number of real-world scenarios. For example, the co-occurrence probability of "beach" and "sunglasses" is 0.85, while the co-occurrence probability of "swimsuit" and "down jacket" is 0.001.

[0066] The process of constructing the co-occurrence probability matrix involves: collecting a historical dataset containing at least 500,000 real purchase or browsing records from e-commerce scenarios; extracting all appearing product attribute tags; and forming a 2000-dimensional tag vector space after deduplication.

[0067] For each pair of tags (i,j), count the number of times they appear together in the same product or shopping session, c(i,j); calculate the co-occurrence probability as p(i,j) = c(i,j) / max(ni,nj), where max(ni,nj) refers to the maximum value between the two. Through this calculation, all probability values ​​are normalized to the [0,1] interval by min-max.

[0068] For label pairs that have not appeared in historical data, Laplace smoothing is applied, and a minimum positive probability value (e.g., 0.001) is assigned. The logical mutual exclusion threshold (e.g., 0.05, where 0.05 in the co-occurrence probability matrix represents that the two labels have almost no statistical correlation in the massive historical data and are on the edge of the long-tail distribution) can be dynamically adjusted according to the label mutual exclusion characteristics of different application fields.

[0069] The proposed logical conflict resolution method based on the co-occurrence probability matrix is ​​based on the following understanding: when the co-occurrence probability of two labels is below a threshold, it indicates that they rarely appear together in historical data, but this does not necessarily mean they are logically incompatible. To reduce the risk of false positives, this method incorporates an arbitration mechanism based on visual dominance, prioritizing the retention of labels that have stronger salience in the current visual scene.

[0070] Further, a confidence-based initial screening is performed to select a candidate label set. Specifically, the processor receives the original explicit label list output from the previous steps and its corresponding confidence scores (e.g., "dress": 0.92, "red": 0.88, "summer": 0.75, "snow": 0.65, "clothes": 0.95). The system adopts an adaptive threshold truncation algorithm. First, it calculates the average confidence score of all labels (e.g., 0.83). This average is multiplied by a leniency factor of 0.8 as a dynamic truncation threshold (0.664). The system traverses the original list, retains all labels with confidence scores higher than the threshold, and removes those ambiguous low-scoring labels (e.g., removing "snow" with a confidence score of only 0.65), forming a candidate label set.

[0071] Furthermore, a graph-based semantic logic check is performed to determine whether there is a semantic conflict. Specifically, the processor combines the tags in the candidate tag set in pairs to generate several tag pairs (e.g., "summer - dress" and "summer - red"). For each tag pair, the system queries the local co-occurrence probability matrix and reads the corresponding probability value. Suppose the system detects a tag pair consisting of the "summer" tag and a misidentified "skiing" tag (assuming it passed the initial screening but is logically invalid), and its co-occurrence probability value is 0.02, which is lower than the preset logical mutual exclusion threshold of 0.05. The system then determines that these two tags have a serious semantic logic conflict and marks them as a "mutually exclusive conflict pair", which needs to enter the arbitration process.

[0072] Furthermore, the system performs arbitration based on visual dominance, eliminates logically paradoxical labels, and generates visual feature representations. Specifically, for label pairs marked as mutually exclusive and conflicting (e.g., "summer" vs. "skiing"), the processor backtracks to the visual detection stage and calculates the dominance score of their corresponding visual regions. The calculation formula is: Score = (pixel area of ​​the object corresponding to the label / total area of ​​the screen) × 0.5 + (1 / normalized distance from the center of the object to the center of the screen) × 0.3 + (Laplacian sharpness score of the object texture) × 0.2. Assuming that the dominance score of "summer" (corresponding to the sunny beach area in the background) is 0.8, while the dominance score of "skiing" (corresponding to the white reflective point misidentified in the corner of the screen) is 0.1, the system will decisively retain the "summer" label and remove the "skiing" label. At the same time, using a hierarchical classification tree, the higher-level label "clothes" is removed, leaving only the more specific "dress". Finally, the system outputs a cleaned and verified pure label group (e.g., "dress", "red", "summer") as a visual feature representation and sends it to the advertising server.

[0073] By introducing federated learning, the knowledge graph is upgraded to a dynamically evolving graph based on collective intelligence. The edge knowledge graph is no longer a static, read-only file, but a writable database with local update capabilities. When multiple users' edge models frequently detect a new co-occurrence pattern within a specific time period (e.g., the co-occurrence of "outerwear" and "business scenario," which violates the old logic), the terminal generates an encrypted gradient update package. When the device is idle and charging, these logical changes are aggregated in the cloud through a federated averaging algorithm, the global graph structure is updated, and then distributed to all terminals. The entire process only transmits logical weights and does not transmit any user behavior data. This mechanism empowers the advertising system to capture "instantaneous trends" and "subcultural trends," solving the technical pain points of traditional static knowledge bases being slow to update and unable to adapt to rapid market changes. At the same time, the federated learning architecture achieves real-time synchronization and intelligent iteration of the entire network's business logic while ensuring that user privacy data never leaves the edge.

[0074] A lightweight knowledge graph containing a tag co-occurrence probability matrix was constructed on the edge. Through multi-dimensional confidence filtering, graph logic verification, and visually dominant arbitration based on pixel area and texture clarity, rigorous conflict resolution was performed on the recognition results. This mechanism effectively eliminates tags with multiple solutions or logical paradoxes that violate common sense, ensuring that the final generated query parameters possess both semantic consistency and visual saliency, greatly improving the accuracy and relevance of ad recommendations.

[0075] Furthermore, the generated visual feature representation and user identifier are hash-de-identified using the SHA-256 algorithm. The steps for generating privacy-protected product requests specifically include: First, performing a SHA-256 hash operation on the device's unique hardware identifier (IMEI or IDFA) to generate an anonymous session ID; second, converting each discrete tag in the visual feature representation into a numeric tag ID; third, packaging the numeric tag ID array, the anonymous session ID, and the 512-dimensional implicit semantic vector into a JSON structure, with the total data packet size controlled within 2KB; finally, using the TLS 1.3 protocol to perform full end-to-end encryption on the JSON request and sending it to the advertising server via an HTTPS channel. Specifically, the processor receives the visual feature representation (i.e., the cleaned tag list) output from the preceding steps and the device's unique hardware identifier (such as IMEI or IDFA). To sever the association between the advertising request and a specific user, the system first performs a SHA-256 hash operation on the hardware identifier string. This operation first pads the input string with binary bits so that its length modulo 512 leaves a remainder of 448, and then adds 64 bits of length information. Subsequently, it initializes eight 32-bit hash initial values, performs bitwise operations (such as XOR and circular right shift) and non-linear logic function operations through 64 rounds of cyclic compression functions, and finally outputs a 256-bit digest string consisting of 64 hexadecimal characters, which serves as the "anonymous session ID" for this session. At the same time, for structured tags (such as "dress"), the system queries a locally preset mapping table and converts them into semantically unrelated numeric tag IDs (such as "10086"), completely removing the original text information, thereby ensuring that the uploaded data only contains content indexes and no user profile data.

[0076] Furthermore, a lightweight JSON request is constructed using the processed tags as query parameters. Specifically, the system allocates a character buffer in memory and constructs a JSON object conforming to the RESTful API specification. This object contains three core fields: "session_hash" (i.e., the anonymous session ID generated above), "target_ids" (i.e., the converted numeric tag ID array, such as [10086,2045,3011]), and "embedding_vec" (i.e., the 512-dimensional implicit semantic vector generated in the previous step). To reduce network transmission load, the system performs half-precision compression on the floating-point semantic vector, truncating the 32-bit floating-point number of each dimension to 16 bits, keeping the overall size of the data packet within 2KB. Finally, the JSON object is converted into a binary byte stream through a serialization function, ready for transmission.

[0077] Furthermore, the request is securely sent to the advertising server via the HTTPS protocol. Specifically, the system calls the Socket interface of the network protocol stack to establish a TCP connection with the cloud advertising server and immediately initiates the TLS 1.3 handshake process. During the handshake phase, the client and server exchange random numbers and negotiate to generate a temporary symmetric encryption session key (such as an AES-256-GCM key) using the Elliptic Curve Diffie-Hellman (ECDH) algorithm. The system uses this session key to fully encrypt the JSON binary byte stream generated in the previous steps, generating a ciphertext data packet, and sends it to the specified API endpoint via the HTTP POST method. This ensures that even if the data is intercepted during transmission over a public network, it cannot be decrypted and restored, thus completing the sending of the advertising request under privacy protection.

[0078] By employing SHA-256 or homomorphic encryption algorithms to anonymize visual feature representations and transmitting only lightweight feature data instead of the original image, the path to leakage of users' sensitive privacy information is fundamentally cut off. This minimized data interaction strategy not only meets stringent data compliance requirements but also significantly reduces network transmission load, ensuring millisecond-level request responses even in weak network environments.

[0079] The generation of privacy-preserving product requests and the receipt of recommendation data adopt a local black-box matching architecture based on a Trusted Execution Environment (TEE). The specific steps include:

[0080] Pre-loading encrypted index: Under idle or high-bandwidth network conditions on the terminal device, an encrypted advertising feature index library is pre-downloaded from the advertising platform to the secure storage area of ​​the terminal device's Trusted Execution Environment (TEE); Hardware isolated computation: The semantic representation data generated by the hybrid semantic analysis model is directly input into the Trusted Execution Environment. In a hardware-isolated area inaccessible to the operating system, the advertising feature index library is decrypted and plaintext matching computation is performed with the semantic representation data; Zero-knowledge proof reporting: Only the matching advertising ID is output as recommended product data, and an advertising display certificate is generated for the advertising server through zero-knowledge proof (ZKP), completing advertising delivery and billing verification without uploading any user profiles or semantic data.

[0081] In this invention, the system introduces a Trusted Execution Environment (TEE), such as ARMTrustZone or AndroidKeystore. Under Wi-Fi conditions, the terminal device periodically synchronizes an encrypted index of popular advertisements from the cloud. This data falls directly into the secure memory area of ​​the TEE, inaccessible to the external operating system (RichOS). When video analysis yields the semantic vector of the subject, this vector bypasses the network and is directly "sunk" into the TEE area. The TEE, acting as a "hardware black box," internally decrypts the advertisement index and performs high-dimensional vector similarity matching with the semantic vector. Because the TEE is isolated from the main operating system at the hardware level, even if the user device is rooted or infected with malware, attackers cannot access any data during the matching process.

[0082] After matching is complete, the TEE only returns the matched ad ID (for rendering) to the application layer and generates a zero-knowledge proof (ZKP) credential to send to the server. This credential mathematically proves that "a specific ad has been shown to a qualified user," but it contains no specific information about "who the user is" or "what the user is watching." This architecture eliminates the latency bottleneck of real-time network communication, ensuring millisecond-level accurate ad delivery based on real-time video content, even in weak network or offline environments.

[0083] Reference Figure 3 The flowchart illustrates the intelligent overlay rendering decision-making process. Further, it employs real-time HSV / luminance histograms of video frames and dynamic Alpha channel fusion with the background layer of the advertising component. Specifically, the graphics rendering engine captures a frame currently being rendered by the video player, converts it from RGB color space to HSV color space, and the processor extracts pixel data from the V channel (luminance channel) to construct a luminance statistical histogram containing 256 intervals. The system iterates through this histogram to calculate the average luminance value of the current frame (range 0 to 255). To ensure the readability of the advertising overlay text while maintaining background transparency, the system uses inverse linear mapping logic to calculate the Alpha transparency value of the advertising background layer: if the current average luminance value is 200 (high brightness scene), the system sets the Alpha value of the advertising background to 230 (high opacity) to increase contrast; if the current average luminance value is 50 (low light scene), the system sets the Alpha value to 100 (semi-transparent) to prevent the advertisement from appearing too abrupt and glaring. Finally, the calculated Alpha value is written into the material properties of the UI rendering pipeline, completing the frame-level dynamic fusion.

[0084] The step of dynamically fusing the Alpha channel using real-time HSV / luminance histogram of video frames and advertising components:

[0085] 1. Image color space conversion: Convert the current video playback frame from RGB to HSV and extract the V channel (luminance) pixel data;

[0086] 2. Brightness histogram generation: Perform 256 interval histogram statistics on the V channel to calculate the average brightness value Bavg of the screen (range 0-255).

[0087] 3. Alpha value calculation: A piecewise linear mapping function is used: when Bavg > 200, Alpha = 240 - (Bavg - 200) × 0.2; when Bavg ∈ [100, 200], Alpha = 180 + (Bavg - 100) × 0.2; when Bavg < 100, Alpha = max(80, 100 - 100 - Bavg × 0.1).

[0088] 4. Real-time fusion: The luminance histogram is recalculated and the alpha value is updated every 10 frames.

[0089] Furthermore, edge detection is employed to avoid the face and subtitle areas, and the floating position of the AI ​​assistant is intelligently adjusted and activated. Specifically, the system first calls a lightweight face detection interface to obtain the bounding box coordinates of the person's face in the current image, and at the same time, the bottom 20% of the screen area is rigidly divided into a "subtitle no-go zone". Subsequently, the processor executes the Sobel edge detection operator on the current video frame, calculates the gradient magnitude of the entire image, and generates a grayscale edge energy map. The system presets four candidate floating coordinate points in the four quadrants of the screen: upper left, lower left (avoiding the subtitle area), upper right, and lower right. For each candidate point, the cumulative edge energy value of the surrounding 150×150 pixel area is calculated, and it is checked whether the area intersects with the face bounding box. The system selects the candidate point with the smallest cumulative edge energy value and that does not overlap with the face as the best display position through comparison calculation, and anchors the AI ​​assistant floating icon to this coordinate (e.g., screen coordinates x=1600, y=200), thereby ensuring that the advertisement does not obscure the key textures or key information of the video subject.

[0090] The specific steps for using edge detection to avoid face and caption regions are as follows: Every 3 frames, a lightweight face detector is called to obtain a set of face bounding boxes with a confidence score > 0.8. The restricted areas are defined as follows: face restricted area: the face bounding box extends outward by 20 pixels; caption restricted area: the area at the bottom 15% of the screen; boundary restricted area: 10 pixels reserved at each of the four edges of the screen. Edge detection uses the Sobel operator to calculate the gradient of the entire frame and generate an edge energy map. Finally, candidate points that meet the following conditions are selected: they do not intersect with any face restricted areas, they are not in the caption restricted areas, and the cumulative edge energy value is the smallest. If all candidate points conflict with faces, the candidate point with the smallest face overlap is selected, and the transparency of the advertisement background is increased.

[0091] Furthermore, the AI ​​assistant, combined with the edge-side Small Language Model (SLM), provides users with a conversational agent for the displayed advertisements. Specifically, when a user clicks the floating icon to trigger an interaction, the system activates the edge-side Small Language Model residing in the terminal's NPU (Neural Processing Unit). This model adopts a TransformerDecoder-Only architecture, with the number of parameters compressed to 1.2 billion (1.2B). It undergoes 4-bit weight quantization to adapt to mobile device memory. The model contains 24 stacked self-attention layers, each with a hidden layer dimension of 2048, and a multi-head attention mechanism with 32 heads. The system converts the structured product information of the current advertisement (such as "Brand: XX, Function: Electrolyte supplement, Price: 5 yuan") into natural language prompts as the system's preset context input model. When it receives the user's voice-to-text input (such as "Does this contain sugar?"), the model performs autoregressive inference, calculates the probability distribution of the next token through matrix multiplication and Softmax normalization, generates reply text that matches the product facts word by word (such as "This beverage is a 0-sugar formula..."), and renders the generated dialogue content directly in the floating window to achieve local intelligent shopping guide without delay.

[0092] TinyLLaMA-1.1B was used as the base model, with 4-bit weights quantized to 280MB. LoRA fine-tuning was performed based on 50,000 triples (user question, product attribute, response text). The mapping from product information to the prompt uses a predefined template, such as brand: Nike, category: athletic shoes, features: breathable, cushioning, price: 399 yuan. A search-enhanced generation (RAG) architecture was adopted: the terminal maintains a local product attribute ontology library. When a user asks a question, a match is first performed in the library, and SLM is invoked only if no match is found. Post-processing filtering is performed on the responses generated by SLM to detect and replace over-promises or illegal statements.

[0093] By calculating the HSV histogram and edge distribution of video frames in real time, the transparency and floating position of advertising components are dynamically adjusted, intelligently avoiding key areas such as faces and subtitles, achieving visually seamless and non-intrusive integration. Simultaneously, combined with the accompanying dialogue service provided by the on-device small language model, the rigid advertising display is transformed into an intelligent aid with interactive value, effectively improving user acceptance and immersive viewing experience.

[0094] This invention achieves real-time end-to-end analysis, from video scene monitoring and subject segmentation to semantic verification, by constructing a closed-loop processing flow on the terminal device side. This effectively avoids the privacy leakage risks and transmission delays associated with uploading raw images to the cloud. Simultaneously, by combining hybrid semantic analysis and knowledge graph verification, it significantly improves the semantic matching and logical consistency between advertising content and video footage, achieving a precise and non-intrusive native-like delivery experience.

[0095] Example 2

[0096] The aforementioned embodiments primarily addressed relatively stable shots or simple overlay displays. However, in more complex scenarios, such as handheld "vlogs" or intense "action movie" clips, the camera experiences significant displacement, rotation, and shaking. Directly pasting images would render the advertising objects as floating on the screen like "paper cutouts," severely compromising realism. This embodiment addresses how to stably "nail" virtual advertising objects (such as roadside billboards) to a real plane in the video while the camera is moving rapidly, and how to dynamically change their lighting and shadows with the environment.

[0097] Specific implementation steps:

[0098] Furthermore, based on sparse SLAM, a local 3D map is constructed and the camera pose is calculated. Specifically, the system acquires a video stream at a rate of 30 frames per second, extracts FAST feature points from each frame, and uses optical flow to track the pixel displacement of feature points between consecutive frames. The processor selects key frames with sufficient parallax, calculates the essential matrix through epipolar geometric constraints, and then decomposes the rotation matrix R and translation vector t of the camera relative to the world coordinate system. At the same time, the 3D spatial coordinates of the feature points are recovered using triangulation to construct a local sparse point cloud map. To address the blurring caused by rapid motion, the system introduces inertial measurement unit (IMU) data for pre-integration and corrects the drift of pure visual calculation through vision-inertial tight coupling optimization (VIO). Finally, the system outputs the precise 6-DOF pose (6-DoFPose) of the camera in 3D space for each frame.

[0099] Furthermore, a plane fitting algorithm is used to determine the optimal planar area for ad placement. Specifically, the processor executes the RANSAC (Random Sample Consensus) plane fitting algorithm on the sparse point cloud map constructed above. The system randomly selects three points in the map to calculate the plane equation parameters (normal vector n and distance d), counts the distances from all other points to the plane, sets a distance threshold of 5 centimeters, and if the proportion of interior points that meet the threshold exceeds 60%, the plane is deemed valid (e.g., identifying a wall or ground on the roadside). The system then calculates the angle between the normal vector and the gravity vector of the plane, selects planes perpendicular to the ground as candidate ad locations, and establishes the coordinates of the four vertices of the plane in three-dimensional space to construct the anchoring reference for the virtual billboard.

[0100] Furthermore, the system performs virtual object rendering and compositing based on spherical harmonic lighting estimation. Specifically, to make the embedded virtual billboard's lighting and shadows realistic, the system captures the current video frame and uses a lightweight neural network (such as EnvNet) to regress the ambient lighting parameters of the current scene from a single RGB image. These parameters are represented as a vector of nine coefficients of a second-order spherical harmonic function, corresponding to the dominant hue, direction, and intensity distribution of the ambient light, respectively. The graphics rendering engine (such as OpenGLES) loads the 3D model of the virtual billboard and its PBR (physically based rendering) material, sets the camera pose matrix calculated above as the observation matrix, injects the spherical harmonic coefficients into the lighting calculation module in the shader, and calculates the diffuse color and specular reflection components of the virtual billboard surface in real time, ensuring that the light and dark distribution of its illuminated and shadowed surfaces are completely consistent with the real objects in the video. Finally, the rendered virtual image with correct perspective and lighting is superimposed onto the original video frame to achieve a "deceptively realistic" dynamic implantation.

[0101] This embodiment discloses a method for dynamic augmented reality advertising insertion in complex motion scenes. Addressing the image shake and perspective changes caused by drastic camera movement, the method first utilizes visual-inertial tightly coupled SLAM technology to calculate the high-precision six-DOF camera pose in real time and construct a local sparse 3D map. Second, it uses the RANSAC algorithm to fit the planar structure in the scene to determine the optimal anchoring position of the virtual advertisement. Third, it combines a single-image spherical harmonic illumination estimation network to invert ambient lighting parameters in real time and drive the PBR rendering of the virtual object. Finally, while maintaining strict consistency in geometric perspective and lighting logic, the virtual advertising material is seamlessly synthesized into the dynamic video stream, achieving highly immersive and shake-resistant native in-video advertising insertion.

[0102] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dynamic advertising delivery method based on real-time content analysis using edge AI, characterized in that, The method is executed on the terminal device side and includes the following steps: Based on inter-frame differences, a lightweight scene monitoring model is constructed using optical flow to continuously monitor the currently playing video stream. When the video stream meets the preset trigger conditions, key frames are extracted from the video stream, and the main analysis model is used to perform subject detection and segmentation on the key frames to determine the boundaries of the subjects in the key frames. A close-up image of the main subject is cropped from the keyframe according to the boundary; a hybrid semantic analysis model is used to perform forward reasoning on the cropped close-up image of the main subject and output semantic representation data in parallel; based on the lightweight knowledge graph on the edge, semantic logic verification and optimization are performed on the semantic representation data to generate visual feature representation; Based on the visual feature representation, a privacy-preserving product request is generated and sent to the advertising server. The returned recommended product data is received, and an intelligent interactive overlay is rendered on the video playback interface of the terminal device.

2. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The determination of the preset triggering condition includes the following steps: The optical flow field amplitude and direction distribution of inter-frame pixel displacement are used as input features to calculate the optical flow statistical histogram of the current frame and the previous N frames. An optical flow amplitude threshold and a direction entropy threshold are set to jointly judge the optical flow statistical histogram to detect in real time whether the video screen is in a stable state and whether there is low motion blur. A lightweight focus tracking algorithm is used to record the number of screen dwell frames of the same target. When the number of dwell frames exceeds a preset threshold, it is determined that there is a stable focus object.

3. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The lightweight scene detection model is a 3-layer convolutional neural network (CNN), and the training and usage steps include: A video frame dataset labeled with analyzable binary tags is used to supervise the training of a convolutional neural network, which outputs a continuous analyzability score between 0 and 1. The score comprehensively evaluates image stability, blur level, subject salience, and lighting suitability. When the real-time calculated analyzability score is greater than a preset threshold, keyframe extraction is triggered.

4. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The main analysis model selected is YOLOv8n-seg, which has been optimized for mobile devices. The step of determining the boundary of the subject in the keyframe specifically includes: Using the master analysis model, a single instance segmentation inference task is performed on the extracted keyframes; the pixel-level masks and bounding box coordinates of all identified subjects in the image are output, along with the subject category confidence; based on the subject category confidence, the core subjects with the highest confidence and belonging to the category of goods are selected; based on the pixel-level masks of the core subjects, the pixel-level boundaries of a single target subject are determined.

5. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 4, characterized in that, After determining the boundaries of the subject, the method further includes the following steps: For all independent subjects in the same keyframe whose confidence exceeds a preset threshold, a minimum tight bounding box is calculated based on their independent pixel-level mask. According to the minimum tight bounding box, the corresponding multi-subject close-up regions are cropped from the keyframe. Morphological operations and edge feathering are used to optimize the cropped close-up regions, remove background pixel interference caused by bounding box expansion, and generate a set of subject close-up images without background interference.

6. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The hybrid semantic analysis model adopts a hybrid semantic feature extraction architecture, and its specific construction and execution steps include: The pre-trained MobileCLIP is used as the backbone network, and fine-tuned through transfer learning and contrastive learning using a multi-label dataset from a product vertical domain. The fine-tuned model is then used to perform forward inference on the cropped close-up image of the subject, outputting two types of semantic representation data in parallel: the first type is an explicit structured label group, which includes discrete-dimensional labels such as subject category, brand, color, style, material, and scene, for accurate keyword matching; the second type is an implicit dense semantic vector, which maps image features to a high-dimensional continuous vector space, capturing unstructured semantic information including retro style, texture, and atmosphere features, supporting fuzzy semantic matching based on vector similarity.

7. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The semantic logic verification and optimization steps based on the edge-side lightweight knowledge graph specifically include: A lightweight knowledge graph and hierarchical classification tree are constructed on the client side. The hierarchical classification tree defines the hierarchical inclusion relationship between tags to standardize the semantic granularity of tags. The knowledge graph defines the partial overall relationship and co-occurrence probability matrix between tag entities. The co-occurrence probability matrix quantifies the semantic association strength and logical compatibility of different tags in the same spatiotemporal scenario.

8. The dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 7, characterized in that, The semantic verification and query optimization steps further include performing multi-dimensional tag cleaning and logical conflict resolution: First, a confidence-based initial screening is performed. Based on the confidence values ​​output by the hybrid semantic analysis model, an adaptive threshold truncation algorithm is applied to select a set of candidate labels. Second, a graph-based semantic logic verification is performed. The label pairs in the candidate label set are traversed, and the co-occurrence probability matrix is ​​queried. When the co-occurrence probability of two tags is detected to be lower than a preset logical mutual exclusion threshold, a semantic conflict is determined to exist; Finally, an arbitration based on visual dominance is performed. For tag pairs with semantic conflicts, their visual dominance scores in keyframes are calculated. The scores are calculated based on a weighted average of pixel area ratio, subject center distance, and texture sharpness. The side with the higher visual dominance score is retained, and the logical paradox label is removed to generate a visual feature representation.

9. A dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The steps for generating a privacy-protected product request include: The generated visual feature representation is hashed and / or anonymized using the SHA-256 algorithm, retaining only the tag ID and not uploading any original images or user profile data; a lightweight JSON request is constructed with the processed tag as the query parameter and sent to the ad server via HTTPS.

10. A dynamic advertising delivery method based on real-time content analysis using edge AI according to claim 1, characterized in that, The step of rendering the intelligent interactive overlay on the video playback interface of the terminal device includes: The real-time HSV / brightness histogram of video frames is dynamically fused with the background layer of the advertising component using the Alpha channel, and the component transparency is automatically adjusted according to the average brightness of the current screen. Edge detection is used to avoid the face and subtitle areas, and the floating position of the AI ​​assistant is intelligently adjusted and activated. The AI ​​assistant, combined with the edge-side small language model (SLM), provides users with a conversational agent for the displayed advertisements.

Images