A Multimodal Agent-Driven OTT Visual Feature Extraction System and Method

Abstract

This invention relates to the field of Internet television service technology, specifically to an OTT visual feature extraction system and method driven by a multimodal intelligent agent. The system includes: capturing a real-time video stream from a device's screen; processing the target advertisement image and the real-time video stream, extracting heterogeneous visual features from the dual streams using a multimodal visual perception model, including global content-aware features and local geometric structure features; executing a hierarchical matching algorithm, using the global content-aware features to calculate and filter candidate frames, establishing an initial keypoint correspondence set containing all matches within the candidate frames using local geometric structure features, spatially clustering the set to separate advertisement instances, and calculating the bounding boxes of the instances through geometric transformations; and highlighting the area where the target advertisement is located on the original video frame based on the bounding boxes to generate a visual monitoring result. This invention achieves OTT advertisement visual feature extraction and monitoring through multimodal intelligent agent-driven processing.

Description

Technical Field

[0001] This invention relates to the field of Internet television service technology, specifically to an OTT visual feature extraction system and method driven by a multimodal intelligent agent. Background Technology

[0002] With the development of OTT services, automated monitoring of advertisements on OTT platforms to verify the compliance and effectiveness of ad placement has become one of the technological development directions in this field. Existing technologies typically employ automated scripts to control equipment and combine them with computer vision algorithms to identify screen content.

[0003] Among existing visual recognition solutions, one type is template-matching, which has a relatively straightforward computational process, but its recognition results are highly sensitive to changes in the size, angle, and appearance of advertising materials. Another type is local feature point-based matching, designed to address these variations. While offering higher robustness, this method also increases computational complexity; furthermore, when applied to OTT interfaces with complex content elements, similar visual elements in the background may interfere with the uniqueness and accuracy of the matching results.

[0004] When existing technologies are applied to complex OTT real-time monitoring scenarios, there is room for further improvement in processing efficiency.

[0005] To address this, a multimodal intelligent agent-driven OTT visual feature extraction system and method are proposed. Summary of the Invention

[0006] The purpose of this invention is to provide an OTT visual feature extraction system and method based on multimodal intelligent agents. Through multimodal intelligent agent-driven operation, it enables the extraction and monitoring of visual features of OTT advertisements. The system captures the real-time video stream of the device screen; processes the target advertisement image and the real-time video stream, extracting heterogeneous visual features from the dual streams using a multimodal visual perception model, including global content-aware features and local geometric structure features; executes a hierarchical matching algorithm, using the global content-aware features to calculate and filter candidate frames, and within the candidate frames, using local geometric structure features to match and establish an initial keypoint correspondence set containing all matches; spatially clusters the set to separate advertisement instances, and calculates the bounding boxes of the instances through geometric transformations; based on the bounding boxes, the area where the target advertisement is located is highlighted on the original video frame to generate a visual monitoring result.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An OTT visual feature extraction system driven by a multimodal intelligent agent includes:

[0009] The monitoring task parsing module receives monitoring tasks containing target advertising images and navigation instruction sequences, sends navigation instructions to OTT devices through the Internet of Things interface to execute interface control, and captures the device's real-time screen video stream;

[0010] The dual-stream heterogeneous feature extraction module processes the target advertisement image and real-time video stream, and extracts dual-stream heterogeneous visual features through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features.

[0011] The multi-stage matching module executes a hierarchical matching algorithm, uses global content-aware features for calculation, filters out candidate frames, uses local geometric structure features for matching to establish an initial key point correspondence set containing all matches, performs spatial clustering on the set to separate the ad instances, and calculates the bounding box of the ad instances through geometric transformation.

[0012] The results generation module highlights and adds visual markers to the area where the target advertisement is located on the original video frame based on the bounding box, generating visual monitoring results.

[0013] Preferably, the process of capturing the real-time video stream from the screen of the capturing device includes:

[0014] The system receives a structured monitoring task via an application programming interface (API) containing a unique task ID, a target advertising image, and a sequence of navigation instructions consisting of multiple atomic remote control commands. It verifies the communication connection status with the IoT human-machine interaction device. After a successful connection, the system sends the navigation instruction sequence sequentially to the OTT large screen device for navigation execution via the API. Simultaneously with sending the first instruction, the system starts an image acquisition device to capture the real-time video stream of the screen using a standard video encoding format. Each frame of data is then timestamped before being output.

[0015] Preferably, the multimodal visual perception model includes:

[0016] The visual spatial encoder receives the current frame static image from the real-time video stream as input, processes it using a first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, which serves as a local geometric structure feature.

[0017] The dynamic change encoder receives the difference image between the current frame and the previous frame as input, processes it using a second convolutional neural network, captures the changing regions, directions and patterns on the screen, and generates dynamic change feature vectors.

[0018] The instruction context encoder receives navigation instructions that cause the current interface state as input, and processes them using a pre-set embedding lookup table and a multilayer perceptron to map the instruction text into a low-dimensional instruction context feature vector.

[0019] The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network to generate an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual spatial encoder, performs dynamic recalibration through channel attention mechanism and spatial attention mechanism, weights the feature map, and outputs global content-aware features.

[0020] Preferably, the hierarchical matching algorithm is executed in the following ways:

[0021] In the fast candidate frame screening step, a cosine similarity algorithm is used to iteratively calculate between the global content-aware features of the target advertisement and the global content-aware features of each frame of the real-time video stream. When the calculated similarity score is higher than a preset high similarity threshold, the current video frame is determined as a candidate frame. Within the candidate frames, the k-nearest neighbor search algorithm is used to match between local geometric structure feature descriptors. A ratio test with a preset ratio threshold is applied for preliminary screening to generate the initial key point correspondence set containing all matches.

[0022] Preferably, the process of spatially clustering the set to separate the advertising instances includes:

[0023] Extract the spatial coordinates of the key points of each matching pair in the real-time video frame from the initial key point correspondence set, form a point set from all coordinates, and input the point set into a spatial density-based clustering algorithm. The clustering algorithm automatically divides these key points into independent clusters based on the spatial proximity between points. The membership of each cluster defines the advertising instance affiliation of the corresponding key point matching pair.

[0024] Preferably, the geometric transformation calculation includes:

[0025] For the divided independent clusters, a random sampling consensus algorithm with configured maximum number of iterations and reprojection error threshold is used to process the key point correspondences within the clusters. In each iteration, random sampling is used to estimate the minimum set of key point matching pairs for the transformation model, and the number of interior points in all correspondences that conform to the estimated transformation model is calculated. The homography matrix with the maximum number of interior points is selected and output as the optimal homography matrix for the current cluster. The coordinates of the four original vertices of the target advertisement image are multiplied with the optimal homography matrix to obtain the bounding box of the advertisement instance.

[0026] OTT visual feature extraction methods driven by multimodal agents include:

[0027] Receive monitoring tasks containing target advertising images and navigation instruction sequences, send navigation instructions to OTT devices through IoT interfaces to perform interface control, and capture real-time video streams of the device screen;

[0028] The target advertisement image and real-time video stream are processed, and dual-stream heterogeneous visual features are extracted through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features.

[0029] The hierarchical matching algorithm is executed, and global content-aware features are used for calculation to filter out candidate frames. Within the candidate frames, local geometric structure features are used for matching to establish an initial key point correspondence set containing all matches. The set is spatially clustered to separate the advertising instances, and the bounding box of the advertising instance is obtained through geometric transformation.

[0030] Based on the bounding box, the area where the target advertisement is located is highlighted and visually marked on the original video frame to generate a visual monitoring result.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] 1. By adopting a phased feature extraction and matching process, the invention first uses global features with low computational overhead for preliminary screening, limiting the number of video frames that need to be precisely analyzed to a small range. While maintaining high recognition accuracy, the invention effectively reduces the overall computational requirements of the system and improves processing efficiency to meet real-time requirements.

[0033] 2. By applying a multi-instance spatial clustering algorithm to group the spatial coordinates of the key points in the initial matching, this invention can effectively distinguish matching point pairs belonging to different advertising instances, suppress false matching caused by similar elements in the background, and achieve independent identification and positioning of multiple target instances, thereby improving the accuracy of identification results in complex scenarios.

[0034] 3. By integrating device navigation control and visual content verification into a unified, task-driven automated process, this invention reduces the need for manual intervention, improves the execution efficiency of large-scale monitoring tasks, and ensures the synchronization of visual capture and interface status through programmatic control, which helps to improve the consistency and reliability of the entire monitoring process. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of an OTT visual feature extraction system driven by a multimodal intelligent agent, provided in an embodiment of the present invention.

[0036] Figure 2 A flowchart illustrating the multimodal visual perception model provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the OTT visual feature extraction method based on multimodal intelligent agent driving provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1:

[0040] An OTT visual feature extraction system driven by a multimodal intelligent agent, specifically as follows: Figure 1 As shown, it includes:

[0041] The monitoring task parsing module receives monitoring tasks containing target advertising images and navigation instruction sequences, sends navigation instructions to OTT devices through the Internet of Things interface to execute interface control, and captures the device's real-time screen video stream;

[0042] The dual-stream heterogeneous feature extraction module processes the target advertisement image and real-time video stream, and extracts dual-stream heterogeneous visual features through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features.

[0043] The multi-stage matching module executes a hierarchical matching algorithm, uses global content-aware features for calculation, filters out candidate frames, uses local geometric structure features for matching to establish an initial key point correspondence set containing all matches, performs spatial clustering on the set to separate the ad instances, and calculates the bounding box of the ad instances through geometric transformation.

[0044] The results generation module highlights and adds visual markers to the area where the target advertisement is located on the original video frame based on the bounding box, generating visual monitoring results.

[0045] Furthermore, the process of capturing the real-time video stream from the screen of the capturing device includes:

[0046] The system receives a structured monitoring task via an application programming interface (API) containing a unique task ID, a target advertising image, and a sequence of navigation instructions consisting of multiple atomic remote control commands. It verifies the communication connection status with the IoT human-machine interaction device. After a successful connection, the system sends the navigation instruction sequence sequentially to the OTT large screen device for navigation execution via the API. Simultaneously with sending the first instruction, the system starts an image acquisition device to capture the real-time video stream of the screen using a standard video encoding format. Each frame of data is then timestamped before being output.

[0047] Specifically, the process of capturing the device's real-time screen video stream begins with receiving a structured monitoring task. This task is received through a RESTful application interface based on the HTTP protocol and encapsulated in JSON format. Its data structure includes taskId (a unique task ID), targetImageUrl (the URL of the target advertisement image), and an array of instructions named commands. This commands array consists of multiple atomic remote control instructions, such as {action: keypress, key: KEYCODE_DPAD_RIGHT} to execute key presses, and {action: wait, duration_ms: 2000} to control the execution rhythm.

[0048] Before executing navigation commands, the system first verifies the communication connection status with the IoT human-machine interface device. Specifically, it sends a GET request to a specific heartbeat detection API endpoint of the target device, and only proceeds after receiving a response with an HTTP status code of 200 OK.

[0049] After a successful connection, the system simultaneously starts the image acquisition device while sending the first instruction of the instruction sequence to the OTT large-screen device through the interface. In this embodiment, the device is an external HDMI video capture card, which connects to the output port of the OTT device via an HDMI cable to acquire the screen's digital signal. The capture card uses H.264 as the video encoding format for capture and sets a constant rate factor of 18 to control the data bitrate while ensuring visual information fidelity.

[0050] To ensure the timing traceability required for event analysis and auditing, the system appends a timestamp to each frame of data in the captured video stream. In this embodiment, the timestamp is formatted with time zone information and an ISO 8601 format string, and then the timestamped video stream is output.

[0051] By including steps such as structured task reception, connection status verification, synchronous acquisition startup, and data frame timestamp appending in the task process, the consistency and reliability of automated monitoring tasks are improved. Furthermore, the timestamped video data output by this process provides time-traceable information for subsequent event analysis and auditing.

[0052] Furthermore, the specific process of the multimodal visual perception model is as follows: Figure 2 As shown, it includes:

[0053] The visual spatial encoder receives the current frame static image from the real-time video stream as input, processes it using a first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, which serves as a local geometric structure feature.

[0054] The dynamic change encoder is configured to receive the difference image between the current frame and the previous frame as input, process it using a second convolutional neural network, capture the changing regions, directions and patterns on the screen, and generate dynamic change feature vectors.

[0055] The instruction context encoder receives navigation instructions that cause the current interface state as input, and processes them using a pre-set embedding lookup table and a multilayer perceptron to map the instruction text into a low-dimensional instruction context feature vector.

[0056] The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network to generate an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual spatial encoder, performs dynamic recalibration through a channel attention mechanism, weights the feature map, and outputs global content-aware features.

[0057] Specifically, the visual spatial encoder focuses on extracting rich spatial details and geometric information from a single frame of static image, and its design aims to provide high-precision and robust local features for subsequent feature matching. In practice, the encoder receives the current frame from a real-time video stream from the screen as input. To ensure processing consistency and optimal model performance, the input image is first preprocessed and normalized, and pixel values ​​are standardized.

[0058] The encoder uses the EfficientNet-B0 model as the core of the first convolutional neural network. In order to obtain features of geometric texture rather than highly abstract semantics, the model does not use its final classification layer. Specifically, the feature map after the fourth MBConv module in the network structure is taken as the output. The feature map at this stage is relatively large in size, retaining sufficient spatial resolution, and its receptive field is sufficient to capture key local structures. The high-resolution spatial content feature map with rich spatial details generated in this process is defined as local geometric structure features, which provides a solid foundation for subsequent accurate keypoint matching.

[0059] To capture changes over time and understand user intent, the model employs two independent encoders in parallel. The dynamic change encoder handles visual temporal changes, while the instruction context encoder focuses on transforming human-readable, discrete navigation instructions into mathematical representations with continuous semantics that the neural network can understand and process.

[0060] The system receives the current frame and the previous frame from a real-time video stream as input. To focus on the intensity changes in content and structure rather than color noise, the system first converts these two frames into single-channel grayscale images. Then, by calculating the absolute difference between corresponding pixels in the two grayscale images, a raw difference image is generated. This difference image visually reflects all the areas on the screen that have changed. To further enhance the robustness of the signal and filter out minor acquisition noise, the system applies a binarization threshold to this difference image, generating a binary mask image that clearly indicates areas of significant change. This processed binary image serves as the final input to the dynamic change encoder.

[0061] In terms of network structure, the lightweight MobileNetV2 model is used as the core of the second convolutional neural network. The second convolutional network receives a binary mask image and processes it through depthwise separable convolutional layers to effectively capture changing regions, orientations, and patterns.

[0062] In the final stage of processing, the two-dimensional feature map output by the network passes through a global average pooling layer, which averages the feature map in the spatial dimension, compresses it, and generates a fixed-dimensional dynamically changing feature vector. This dynamically changing feature vector represents the key information that changes on the screen, stably representing the interface dynamics and providing crucial temporal context information for subsequent multimodal attention fusion.

[0063] The command context encoder is a collaborative operation of a pre-defined embedded lookup table and a multilayer perceptron. First, the lookup table is constructed, and the system defines a complete command vocabulary. In this embodiment, navigation commands consist of a series of atomic remote control commands. The system parses all possible atomic commands, such as {action: keypress, key: KEYCODE_DPAD_RIGHT}, and breaks them down into the smallest semantic units, such as keypress, wait, KEYCODE_DPAD_RIGHT, and KEYCODE_DPAD_LEFT. All these unique semantic units together constitute the system's command vocabulary.

[0064] After the vocabulary is constructed, the system assigns a fixed integer index to each unique instruction unit. For example, keypress corresponds to index 1, wait to index 2, KEYCODE_DPAD_RIGHT to index 3, and so on. A pre-defined embedding lookup table is created as a two-dimensional matrix. The number of rows in this matrix equals the total number of unique instruction units in the vocabulary, and the number of columns is a pre-defined, low-dimensional vector. Each row of the matrix is ​​a randomly initialized low-dimensional vector representing the corresponding instruction unit.

[0065] When the instruction context encoder receives a specific navigation instruction text (such as KEYCODE_DPAD_RIGHT), it first looks up the corresponding integer index (e.g., 3) in the vocabulary. Then, it uses this index to find and extract the 128-dimensional vector corresponding to the third row in the embedding lookup table matrix. This vector is the initial numerical representation of the instruction. This process successfully maps discrete text symbols into a low-dimensional continuous vector.

[0066] Finally, the low-dimensional vector extracted from the lookup table is fed into a multilayer perceptron (MLP) for nonlinear transformation and deep feature extraction, ultimately outputting an instruction context feature vector that can comprehensively represent the current operational intent. Specifically, the MLP consists of two fully connected layers: the first layer receives a 128-dimensional embedding vector, outputs 256 neurons, and uses the ReLU activation function; the second layer receives a 256-dimensional vector and outputs the final 128-dimensional instruction context feature vector.

[0067] Furthermore, the multimodal attention fusion unit receives and integrates contextual information from two different modalities. Specifically, it concatenates the dynamic change feature vector from the dynamic change encoder with the instruction context feature vector from the instruction context encoder in terms of dimension, forming a higher-dimensional composite context vector. This composite vector is a comprehensive, multimodal mathematical description of the current monitoring state.

[0068] Next, the composite context vector is input to the attention generation network, which consists of two parallel, structurally independent multilayer perceptrons (MLPs). These MLPs are used to generate channel attention and spatial attention guidance signals, respectively. The first MLP receives the composite context vector and outputs a channel attention guidance vector with the same dimension as the number of channels in the visual spatial encoder's output feature map. The second MLP also receives the composite context vector and outputs a spatial attention guidance vector for generating spatial attention. The attention generation network consists of two parallel, structurally independent MLPs. The first MLP generates the channel attention guidance vector, and its structure is an input layer – an activation function (ReLU) – an output layer. The second MLP generates the spatial attention guidance vector, and its structure is the same as the first MLP, with the number of input layers equal to the dimension of the concatenated composite vector.

[0069] Then, the system performs the first dynamic recalibration using a channel attention mechanism. It passes the channel attention guide vectors through a sigmoid activation function, normalizing their values ​​to between 0 and 1, forming a set of explicit channel weights. These weights are applied to the original spatial content feature map output by the visual spatial encoder, weighting each channel of the feature map through channel-by-channel multiplication. This step intelligently enhances or suppresses the importance of different feature channels based on the current dynamic changes and instruction context.

[0070] After channel recalibration, the system continues to execute the spatial attention mechanism for a second fine-tuning. To integrate contextual information into the spatial attention generation process, the system first spatially expands the spatial attention guidance vector, copying it into a contextual feature map with the same spatial dimension as the channel-weighted feature map. Subsequently, the system concatenates this contextual feature map with the channel-weighted visual feature map along the channel dimension to form a fused feature map containing the original visual information and multimodal contextual information. Finally, the system uses this fused feature map as input and passes it through a spatial convolutional layer configured with a large-sized convolutional kernel (7x7 in this embodiment) to analyze the response of the fused features at different spatial locations and generate a single-channel heatmap of the same size as the original feature map. Then, it is converted into a spatial attention map through the Sigmoid activation function, where the value of each pixel represents the importance of the corresponding spatial location. This spatial attention map is multiplied element-wise on the channel-weighted feature map. This step allows the model to focus computational resources on specific key regions in the image based on the context.

[0071] Finally, the feature map after dynamic recalibration through the aforementioned channel-space dual attention mechanism will pass through a global average pooling layer. This pooling layer compresses its spatial dimension, ultimately generating a fixed-dimensional global content-aware feature vector that contains the core content of the image and deeply integrates temporal and instructional context information. This feature vector has high generalization and context awareness capabilities, providing high-quality input for subsequent efficient matching and filtering processes.

[0072] By employing a phased feature extraction and matching method, high-speed preliminary screening is performed using global features with low computational overhead. The high-computational-complexity local feature extraction steps are limited to a small number of relevant candidate frames. While maintaining high recognition accuracy, the overall computational load of the system is effectively reduced and the processing efficiency is improved, thus enabling the monitoring solution to better meet the application requirements of real-time performance and accuracy.

[0073] Furthermore, the detailed execution method of the hierarchical matching algorithm includes:

[0074] In the fast candidate frame screening step, a cosine similarity algorithm is used to iteratively calculate between the global content-aware features of the target advertisement and the global content-aware features of each frame of the real-time video stream. When the calculated similarity score is higher than a preset high similarity threshold, the current video frame is determined as a candidate frame. Within the candidate frames, the k-nearest neighbor search algorithm is used to match between local geometric structure feature descriptors. A ratio test with a preset ratio threshold is applied for preliminary screening to generate the initial key point correspondence set containing all matches.

[0075] Specifically, the system iteratively employs the cosine similarity algorithm to calculate the similarity score between the global content-aware feature vector of the target advertisement image and the global content-aware feature vector of each frame in the real-time video stream. Only when the calculated score is higher than a preset similarity threshold of 0.9 is the video frame identified as a candidate frame and enter the subsequent processing flow.

[0076] In the precise matching step, the system's processing objects are limited to the candidate frames selected in the previous stage. For each frame, the system uses a FLANN-based k-nearest neighbor search algorithm to match the set of local geometric feature descriptors of the target advertisement image with the descriptor set of the current candidate frame, setting k to 2 to find the nearest and second nearest neighbors of each descriptor. Subsequently, the system applies a ratio test with a preset ratio threshold of 0.75 to perform preliminary screening of the matching results. Only when the ratio of the distance between the nearest and second nearest neighbors is less than 0.75 is the match considered a clear and reliable correspondence and retained, ultimately generating an initial keypoint correspondence set containing all verified matching pairs. The hierarchical matching algorithm introduces a crucial candidate frame temporal stability verification step when determining candidate frames. This step does not determine a frame as a candidate frame simply because the similarity of a single frame exceeds the threshold. Instead, it requires that within a preset time window (5 consecutive frames), the similarity score of all frames is higher than the threshold. Only when this temporal stability condition is met will the system confirm the last frame of the window as a candidate frame and activate subsequent precise matching. This mechanism can intelligently filter out transient pseudo-candidate frames caused by dynamic effects, ensuring that computing resources are only used for scenes that truly and stably present the target, thereby significantly improving the overall efficiency and robustness of the algorithm.

[0077] By first using a computationally efficient cosine similarity algorithm to preliminarily determine candidate frames in the hierarchical matching algorithm, and then applying k-nearest neighbor search and ratio test to perform precise matching only on local features within the candidate frames, the intensive matching computation is concentrated on the most relevant subset of data, which significantly reduces the overall computational requirements and improves processing efficiency. The ratio test effectively eliminates fuzzy and ambiguous matching pairs, ensuring the accuracy of the final generated key point correspondence, thus helping to balance the requirements of processing efficiency and recognition accuracy in real-time monitoring.

[0078] Furthermore, the process of spatially clustering the set to separate the advertising instances includes:

[0079] Extract the spatial coordinates of the key points of each matching pair in the real-time video frame from the initial key point correspondence set, form a point set from all coordinates, and input the point set into a spatial density-based clustering algorithm. The clustering algorithm automatically divides these key points into independent clusters based on the spatial proximity between points. The membership of each cluster defines the advertising instance affiliation of the corresponding key point matching pair.

[0080] Specifically, the system iterates through each matching pair in the set and extracts the spatial coordinates (x, y) of the keypoints in the real-time video frames. All these coordinates extracted from the video frames are aggregated to form a two-dimensional spatial point set, which serves as the sole input to the subsequent clustering algorithm. Next, the system inputs this two-dimensional point set into a DBSCAN algorithm. This algorithm automatically divides these keypoints into one or more independent clusters based on the spatial proximity between points, using two core parameters: a neighborhood radius of 30 pixels and a minimum number of points of 5. Any isolated point not belonging to any cluster is identified as noise and discarded. Finally, the cluster membership of each keypoint coordinate in the clustering result precisely defines the instance affiliation of its corresponding original keypoint matching pair, thus separating matching pairs belonging to different ad instances. To further improve the separation accuracy under complex layouts, a cluster validity verification step is added after the spatial clustering process. This step, for each generated cluster, first calculates its minimum bounding rectangle and its aspect ratio. If the ratio of this aspect ratio to the original aspect ratio of the target ad image exceeds 2.0, the system determines that the cluster has incorrectly merged multiple instances. In this case, the system will automatically re-perform spatial clustering on the point set within that specific cluster using a stricter neighborhood radius (halving the original neighborhood radius of 30 pixels to 15 pixels) to attempt secondary separation. In this embodiment, the spatial clustering uses the DBSCAN algorithm. This verification step improves the cluster merging problem that may occur when multiple ad instances are adjacent or partially overlapping, significantly improving the accuracy of target separation in complex, high-density ad layouts and avoiding missed or false detections due to incorrect clustering.

[0081] By applying a density-based clustering algorithm to the spatial coordinates of all matching keypoints in the video frame, matching point pairs belonging to different ad instances can be automatically separated into independent clusters. This helps to address the problem of mutual interference when multiple ad instances appear simultaneously and effectively filters out discrete noise matching points, providing a cleaner and more independent subset of data for subsequent geometric transformation calculations. Thus, without significantly increasing computational complexity, the accuracy and reliability of the final target localization are improved.

[0082] Furthermore, the geometric transformation calculation includes:

[0083] For the divided independent clusters, a random sampling consensus algorithm with configured maximum number of iterations and reprojection error threshold is used to process the key point correspondences within the clusters. In each iteration, random sampling is used to estimate the minimum set of key point matching pairs for the transformation model, and the number of interior points in all correspondences that conform to the estimated transformation model is calculated. The homography matrix with the maximum number of interior points is selected and output as the optimal homography matrix for the current cluster. The coordinates of the four original vertices of the target advertisement image are multiplied with the optimal homography matrix to obtain the bounding box of the advertisement instance.

[0084] For each independent cluster, the system independently executes a RANSAC (Random Sample Consensus) algorithm configured with a maximum of 1000 iterations and a reprojection error threshold of 3.0 pixels. In each iteration, the algorithm randomly samples the minimum set of keypoint matching pairs required to estimate the transformation model from within the cluster, i.e., 4 matching pairs, to calculate a temporary homography matrix. Subsequently, the algorithm uses this matrix to transform all keypoints within the cluster from the target image to the video frame coordinate system and calculates the distance between the transformed coordinates and the actual matching coordinates, i.e., the reprojection error. If the error is less than 3.0 pixels, the matching pair is counted as an inlier. After completing all iterations, the system selects and outputs the homography matrix with the maximum number of inliers as the optimal homography matrix for that cluster.

[0085] After obtaining the optimal homography matrix of the cluster, the system enters the final bounding box generation stage. The system takes the four original vertex coordinates of the target advertisement image, namely (0,0), (width,0), (width,height), and (0,height), and performs a perspective transformation by multiplying them with the optimal homography matrix. The four new coordinate points generated by the operation constitute the bounding box of the advertisement instance in the real-time video frame, thus completing the precise positioning of the instance.

[0086] By employing a random sampling consensus algorithm to calculate the geometric transformation model for each clustered key point cluster, the accuracy and robustness of the final localization results are improved. Through iterative sampling and finding the maximum set of interior points, the optimal homography matrix can be robustly estimated from matching relationships containing a small amount of noise, thereby reducing the interference of incorrect matching on the final bounding box calculation. This is a key step in ensuring that the system outputs accurate and reliable monitoring results in complex matching scenarios.

[0087] Finally, the result generation module highlights the area where the target advertisement is located on the original video frame and adds a visual marker based on each bounding box, generating the final visual monitoring result.

[0088] On the corresponding original video frame images, the system performs visual labeling: drawing the quadrilateral bounding box of the advertisement instance using high-contrast colors. Furthermore, the system independently outputs a structured JSON metadata file as the core monitoring credential. This file contains global information such as the task ID and frame timestamp, and records each advertisement instance in detail through the `detections` array. Each object provides a unique instance ID, a bounding box composed of four [x,y] coordinate points, and other information. Separating intuitive visual evidence from precise, machine-readable data facilitates both manual verification and greatly facilitates automated report generation and in-depth data analysis.

[0089] By organically integrating phased feature extraction processes, hierarchical matching algorithms, multi-instance spatial clustering, and geometric verification techniques, and placing them within a task-driven automated control loop, the analysis scope is rapidly narrowed using computationally efficient global features, thereby improving processing efficiency in real-time video streams. Simultaneously, through a series of steps such as precise matching of local features, spatial clustering to separate multiple instances, and geometric transformation to filter noise, the recognition accuracy is improved in scenarios with complex backgrounds and multiple coexisting targets, ultimately contributing to achieving both real-time performance and reliability in monitoring tasks.

[0090] Example 2:

[0091] To achieve automated monitoring of specific advertising creatives, this invention introduces an OTT visual feature extraction method driven by a multimodal intelligent agent. The specific process is as follows: Figure 3 As shown.

[0092] The monitoring task verifies whether a specific ad creative A is displayed in the designated content recommendation area of ​​the target OTT large-screen device. The operator creates this task by calling the system API. The task content includes the image file of the target ad creative A and a sequence of navigation instructions for interface control. This instruction sequence consists of a series of specific atomic instructions, such as [{action: keypress, key: home}, {action: wait, duration: 1000}, {action: keypress, key: right}, {action: keypress, key: right}, {action: keypress, key: ok}], designed to simulate remote control operation to guide the device from the home screen interface to the expected content display page.

[0093] Once the task is initiated, the system's intelligent agent connects to the target OTT device via an IoT interface. Simultaneously, upon sending the first home button command, it activates a high-definition image capture device to begin recording a screen video stream with precise timestamps. The intelligent agent will then automatically complete the entire navigation control process strictly following the instructions and order in the JSON sequence, requiring no human intervention.

[0094] During device navigation, the system's multimodal visual perception model operates in real-time in the background. For interface frames in the video stream unrelated to ad creative A (such as the homepage, app list, etc.), although the visual spatial encoder (EfficientNet-B0) extracts their visual content, the dynamic change encoder (MobileNetV2) captures the screen changes caused by moving to the right, while the instruction context encoder parses the intent of the right instruction. This contextual information is fed into the multimodal attention fusion unit. The generated global content-aware features do not match the features of the target ad, and their similarity scores will be far below the preset threshold. Therefore, the system efficiently ignores these frames.

[0095] When the interface successfully navigates to the target page and ad creative A appears, the screen stabilizes, exhibiting weak dynamic changes, but the visual content remains highly relevant to the target. After calculating the cosine similarity between the model-generated global content-aware features and the target ad's features, if the score exceeds a preset threshold (0.9), the system immediately designates that frame as a candidate frame. Only then does the system trigger subsequent precise analysis, which involves calling the extracted local geometric features from the EfficientNet-B0 intermediate layers to perform precise feature point matching between the candidate frame and the target ad image. After obtaining the initial set of matching points, the system further processes them using a spatial clustering algorithm (DBSCAN). This algorithm aggregates the matching points representing ad creative A into a compact cluster, while identifying and removing isolated mismatches caused by other similar graphics in the background as noise, thus purifying the data used for geometric verification. Subsequently, the system uses the Random Sample Consensus Algorithm (RANSAC) to calculate the precise geometric transformation relationship only for the purified point set.

[0096] Finally, based on the calculated geometric transformation relationships, the system accurately selects the location of ad creative A on the candidate frame image using a highlighted rectangle, and can attach metadata such as task ID and timestamp to generate a complete visual monitoring report and archive it. This optimized process ensures that monitoring tasks can balance real-time performance and reliability in complex real-world application scenarios.

[0097] By adopting a phased processing flow, the process first uses computationally efficient global features for high-speed filtering to improve overall processing efficiency. Then, for only a small number of candidate frames selected, a series of refined steps, including local feature matching, multi-instance spatial clustering, and geometric verification, are applied to improve the recognition accuracy in scenarios with complex backgrounds and multiple coexisting targets. This helps to meet the application requirements of monitoring tasks in terms of real-time performance and reliability.

[0098] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An OTT visual feature extraction system driven by a multimodal intelligent agent, characterized in that, include: The monitoring task parsing module receives monitoring tasks containing target advertising images and navigation instruction sequences, sends navigation instructions to OTT devices through the Internet of Things interface to execute interface control, and captures the device's real-time screen video stream; The dual-stream heterogeneous feature extraction module processes the target advertisement image and real-time video stream, and extracts dual-stream heterogeneous visual features through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. The multimodal visual perception model includes: The visual spatial encoder receives the current frame static image from the real-time video stream as input, processes it using a first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, which serves as a local geometric structure feature. The dynamic change encoder receives the difference image between the current frame and the previous frame as input, processes it using a second convolutional neural network, captures the changing regions, directions and patterns on the screen, and generates dynamic change feature vectors. The instruction context encoder receives navigation instructions that cause the current interface state as input, and processes them using a pre-set embedding lookup table and a multilayer perceptron to map the instruction text into a low-dimensional instruction context feature vector. The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network to generate an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual spatial encoder, performs dynamic recalibration through channel attention mechanism and spatial attention mechanism, weights the feature map, and outputs global content-aware features. The multi-stage matching module executes a hierarchical matching algorithm, uses global content-aware features for calculation, filters out candidate frames, uses local geometric structure features for matching to establish an initial key point correspondence set containing all matches, performs spatial clustering on the set to separate the ad instances, and calculates the bounding box of the ad instances through geometric transformation. The results generation module highlights and adds visual markers to the area where the target advertisement is located on the original video frame based on the bounding box, generating visual monitoring results.

2. The OTT visual feature extraction system based on multimodal intelligent agent driving according to claim 1, characterized in that, The process of capturing the real-time video stream from the screen of the device includes: The system receives a structured monitoring task via an application programming interface (API) containing a unique task ID, a target advertising image, and a sequence of navigation instructions consisting of multiple atomic remote control commands. It verifies the communication connection status with the IoT human-machine interaction device. After a successful connection, the system sends the navigation instruction sequence sequentially to the OTT large screen device for navigation execution via the API. Simultaneously with sending the first instruction, the system starts an image acquisition device to capture the real-time video stream of the screen using a standard video encoding format. Each frame of data is then timestamped before being output.

3. The OTT visual feature extraction system based on multimodal intelligent agent driving according to claim 1, characterized in that, The execution method of the hierarchical matching algorithm includes: In the fast candidate frame screening step, a cosine similarity algorithm is used to iteratively calculate between the global content-aware features of the target advertisement and the global content-aware features of each frame of the real-time video stream. When the calculated similarity score is higher than a preset high similarity threshold, the current video frame is determined as a candidate frame. Within the candidate frames, the k-nearest neighbor search algorithm is used to match between local geometric structure feature descriptors. A ratio test with a preset ratio threshold is applied for preliminary screening to generate the initial key point correspondence set containing all matches.

4. The OTT visual feature extraction system based on multimodal intelligent agent driving according to claim 1, characterized in that, The process of spatially clustering the set to separate the advertising instances includes: Extract the spatial coordinates of the key points of each matching pair in the real-time video frame from the initial key point correspondence set, form a point set from all coordinates, and input the point set into a spatial density-based clustering algorithm. The clustering algorithm automatically divides these key points into independent clusters based on the spatial proximity between points. The membership of each cluster defines the advertising instance affiliation of the corresponding key point matching pair.

5. The OTT visual feature extraction system based on multimodal intelligent agent driving according to claim 1, characterized in that, The geometric transformation calculation includes: For the divided independent clusters, a random sampling consensus algorithm with configured maximum number of iterations and reprojection error threshold is used to process the key point correspondences within the clusters. In each iteration, random sampling is used to estimate the minimum set of key point matching pairs for the transformation model, and the number of interior points in all correspondences that conform to the estimated transformation model is calculated. The homography matrix with the maximum number of interior points is selected and output as the optimal homography matrix for the current cluster. The coordinates of the four original vertices of the target advertisement image are multiplied with the optimal homography matrix to obtain the bounding box of the advertisement instance.

6. A method for OTT visual feature extraction based on multimodal intelligent agent driving, characterized in that, include: Receive monitoring tasks containing target advertising images and navigation instruction sequences, send navigation instructions to OTT devices through IoT interfaces to perform interface control, and capture real-time video streams of the device screen; The target advertisement image and real-time video stream are processed, and dual-stream heterogeneous visual features are extracted through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. The multimodal visual perception model includes: a visual spatial encoder, which receives the current frame static image from the real-time video stream as input, processes it using a first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, wherein the spatial content feature map serves as a local geometric structure feature; The dynamic change encoder receives the difference image between the current frame and the previous frame as input, processes it using a second convolutional neural network, captures the changing regions, directions and patterns on the screen, and generates dynamic change feature vectors. The instruction context encoder receives navigation instructions that cause the current interface state as input, and processes them using a pre-set embedding lookup table and a multilayer perceptron to map the instruction text into a low-dimensional instruction context feature vector. The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network to generate an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual spatial encoder, performs dynamic recalibration through channel attention mechanism and spatial attention mechanism, weights the feature map, and outputs global content-aware features. The hierarchical matching algorithm is executed, and global content-aware features are used for calculation to filter out candidate frames. Within the candidate frames, local geometric structure features are used for matching to establish an initial key point correspondence set containing all matches. The set is spatially clustered to separate the advertising instances, and the bounding box of the advertising instance is obtained through geometric transformation. Based on the bounding box, the area where the target advertisement is located is highlighted and visually marked on the original video frame to generate a visual monitoring result.

Images