OTT visual feature extraction system and method based on multi-modal agent driving

Abstract

The invention relates to the technical field of Internet television services, in particular to an OTT visual feature extraction system and method based on multi-modal agent driving, and the method comprises the steps: capturing a screen real-time video stream of equipment; processing the target advertisement image and the real-time video stream, and extracting double-flow heterogeneous visual features, including global content perception features and local geometric structure features, through a multi-modal visual perception model; executing a hierarchical matching algorithm, calculating and screening out candidate frames by using global content perception features, matching in the candidate frames by using local geometric structure features to establish a corresponding relation set containing all matched initial key points, performing spatial clustering on the set to separate out advertisement instances, and obtaining bounding boxes of the instances through geometric transformation calculation; and according to the bounding box, performing highlight display on the area where the target advertisement is located on the original video frame to generate a visual broadcast monitoring result. According to the invention, through multi-mode intelligent body driving, OTT advertisement visual feature extraction and broadcast monitoring are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet television services, and in particular to an OTT visual feature extraction system and method driven by a multimodal intelligent agent. Background Art

[0002] With the development of OTT services, automated monitoring of advertisements on OTT platforms to verify the compliance and effectiveness of advertising has become a key technological development in this field. Existing technologies typically use automated scripts to control devices and combine them with computer vision algorithms to identify screen content.

[0003] Among existing visual recognition solutions, one type is based on template matching. While the computational process is relatively straightforward, its results are highly sensitive to variations in the size, angle, and appearance of the ad creative. Another type is based on matching methods based on local feature points to account for these variations. While these methods offer high recognition robustness, they also increase the computational complexity of their algorithmic processes. Furthermore, when applied to OTT interfaces with complex content elements, similar visual elements in the background may interfere with the uniqueness and accuracy of the matching results.

[0004] When existing technologies are applied to complex OTT real-time monitoring scenarios, there is room for further improvement in processing efficiency.

[0005] To this end, an OTT visual feature extraction system and method based on multimodal agent driving is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide an OTT visual feature extraction system and method based on multimodal intelligent agent drive, which realizes OTT advertising visual feature extraction and monitoring through multimodal intelligent agent drive. The real-time video stream of the device screen is captured; the target advertising image and the real-time video stream are processed, and dual-stream heterogeneous visual features, including global content-aware features and local geometric structure features, are extracted through a multimodal visual perception model; a hierarchical matching algorithm is executed, and the global content-aware features are used to calculate and filter out candidate frames. The local geometric structure features are used to match within the candidate frames to establish an initial key point correspondence relationship set containing all matches, and the set is spatially clustered to separate the advertising instances. The bounding box of the instance is obtained through geometric transformation calculation; based on the bounding box, the area where the target advertisement is located is highlighted on the original video frame to generate a visual monitoring result.

[0007] To achieve the above object, the present invention provides the following technical solutions: The OTT visual feature extraction system driven by multimodal agents includes: The monitoring task parsing module receives monitoring tasks containing target advertising images and navigation instruction sequences, sends navigation instructions to OTT devices through the IoT interface for interface control, and captures the real-time video stream of the device screen; A dual-stream heterogeneous feature extraction module processes the target advertisement image and real-time video stream and extracts dual-stream heterogeneous visual features using a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. The multi-stage matching module implements a hierarchical matching algorithm, using global content-aware features for calculations to select candidate frames. Within the candidate frames, it uses local geometric structure features for matching to establish an initial key point correspondence set containing all matches. This set is then spatially clustered to isolate ad instances, and the bounding boxes of the ad instances are calculated through geometric transformations. The result generation module highlights the target advertisement area on the original video frame based on the bounding box and adds a visual mark to generate a visual monitoring result.

[0008] Preferably, the process of capturing the real-time video stream of the screen of the device includes: Through the application program interface, a structured monitoring task containing a unique task ID, a target advertising image, and a navigation instruction sequence consisting of multiple atomic remote control instructions is received to verify the communication connection status with the IoT human-computer interaction device. After the connection is successful, the navigation instruction sequence is sent to the OTT large-screen device through the interface in sequence to execute navigation. At the same time as sending the first instruction, the image acquisition device is started, and the real-time video stream of the screen is captured using a standard video encoding format. Each frame of data is timestamped and output.

[0009] Preferably, the multimodal visual perception model includes: The visual spatial encoder receives a static image of the current frame in the real-time video stream as input, processes it using a first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, which serves as a local geometric structure feature; The dynamic change encoder receives the difference image between the current frame and the previous frame as input, processes it using a second convolutional neural network, captures the area, direction, and pattern that have changed on the screen, and generates a dynamic change feature vector; The command context encoder receives the navigation command that leads to the current interface state as input, uses a preset embedding lookup table and a multi-layer perceptron to process it, and maps the command text into a low-dimensional command context feature vector; The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network, generates an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual space encoder, dynamically recalibrates the feature map through the channel attention mechanism and the spatial attention mechanism, weights the feature map, and outputs the global content perception feature.

[0010] Preferably, the execution method of the hierarchical matching algorithm includes: In the fast candidate frame screening step, a cosine similarity algorithm is used to perform a cyclic calculation between the global content-aware features of the target advertisement and the global content-aware features of each frame of the real-time video stream. When the calculated similarity score is higher than a preset high similarity threshold, the current video frame is determined to be a candidate frame; only within the candidate frame, a k-nearest neighbor search algorithm is used to match between local geometric structure feature descriptors, and a ratio test with a preset ratio threshold is applied for preliminary screening to generate the initial key point correspondence relationship set containing all matches.

[0011] Preferably, the process of performing spatial clustering on the set to separate the advertisement instances includes: The key point spatial coordinates of each matching pair in the real-time video frame in the initial key point correspondence set are extracted, all the coordinates are formed into a point set, and the point set is input into a spatial density-based clustering algorithm. The clustering algorithm automatically divides these key points into independent clusters according to the spatial proximity relationship between the points. The membership of each cluster defines the advertising instance affiliation of the corresponding key point matching pair.

[0012] Preferably, the geometric transformation calculation includes: For the divided independent clusters, a random sampling consensus algorithm with a configured maximum number of iterations and a reprojection error threshold is used to process the key point correspondences within the cluster; in each iteration, random sampling is used to estimate the minimum key point matching pair set of the transformation model, and the number of inliers that conform to the estimated transformation model in all correspondences is calculated; the homography matrix with the largest number of inliers is selected and output as the optimal homography matrix for the current cluster, and the four original vertex coordinates of the target advertising image are matrix multiplied with the optimal homography matrix to obtain the bounding box of the advertising instance.

[0013] The OTT visual feature extraction method based on multimodal agent driving includes: Receive monitoring tasks containing target advertising images and navigation instruction sequences, send navigation instructions to OTT devices through the IoT interface for interface control, and capture the device's screen real-time video stream; The target advertising image and real-time video stream are processed, and dual-stream heterogeneous visual features are extracted through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. A hierarchical matching algorithm is implemented, using global content-aware features for calculations to select candidate frames. Within the candidate frames, local geometric structure features are used for matching to establish an initial keypoint correspondence set containing all matches. This set is then spatially clustered to isolate the ad instances, and the bounding boxes of the ad instances are calculated through geometric transformations. Based on the bounding box, the area where the target advertisement is located is highlighted on the original video frame and a visual mark is added to generate a visual monitoring result.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. By adopting a staged feature extraction and matching process, the invention first uses global features with low computational overhead for preliminary screening, limiting the number of video frames that require precise analysis to a smaller range. While maintaining high recognition accuracy, the present invention effectively reduces the overall computing requirements of the system and improves processing efficiency to meet real-time requirements.

[0015] 2. By applying a multi-instance spatial clustering algorithm to group the spatial coordinates of the key points of the preliminary match, the present invention can effectively distinguish the matching point pairs belonging to different advertising instances, suppress mismatches caused by similar elements in the background, and realize the independent recognition and positioning of multiple target instances, thereby improving the accuracy of recognition results in complex scenarios.

[0016] 3. By integrating device navigation control and visual content verification into a unified, task-driven automated process, the present invention reduces the need for manual intervention, improves the efficiency of large-scale monitoring tasks, and ensures the synchronization of visual capture and interface status through programmed control, which helps to improve the consistency and reliability of the entire monitoring process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the structure of an OTT visual feature extraction system driven by a multimodal agent provided in an embodiment of the present invention; Figure 2 A schematic diagram of the process of a multimodal visual perception model provided by an embodiment of the present invention; Figure 3 A flow chart of the OTT visual feature extraction method based on multimodal agent driving provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] Example 1: OTT visual feature extraction system driven by multimodal intelligent agent, specifically Figure 1 Shown, including: The monitoring task parsing module receives monitoring tasks containing target advertising images and navigation instruction sequences, sends navigation instructions to OTT devices through the IoT interface for interface control, and captures the real-time video stream of the device screen; A dual-stream heterogeneous feature extraction module processes the target advertisement image and real-time video stream and extracts dual-stream heterogeneous visual features using a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. The multi-stage matching module implements a hierarchical matching algorithm, using global content-aware features for calculations to select candidate frames. Within the candidate frames, it uses local geometric structure features for matching to establish an initial key point correspondence set containing all matches. This set is then spatially clustered to isolate ad instances, and the bounding boxes of the ad instances are calculated through geometric transformations. The result generation module highlights the target advertisement area on the original video frame based on the bounding box and adds a visual mark to generate a visual monitoring result.

[0020] Furthermore, the process of capturing the real-time video stream of the screen of the device includes: Through the application program interface, a structured monitoring task containing a unique task ID, a target advertising image, and a navigation instruction sequence consisting of multiple atomic remote control instructions is received to verify the communication connection status with the IoT human-computer interaction device. After the connection is successful, the navigation instruction sequence is sent to the OTT large-screen device through the interface in sequence to execute navigation. At the same time as sending the first instruction, the image acquisition device is started, and the real-time video stream of the screen is captured using a standard video encoding format. Each frame of data is timestamped and output.

[0021] Specifically, the process of capturing a real-time video stream of the device's screen begins by receiving a structured monitoring task. This task is received via an HTTP-based RESTful application programming interface (API) and encapsulated in JSON format. Its data structure includes a taskId (unique task ID), a targetImageUrl (the URL of the target ad image), and a command array named commands. The commands array consists of multiple atomic remote control commands, such as {action: keypress, key: KEYCODE_DPAD_RIGHT} to execute a key press and {action: wait, duration_ms: 2000} to control the execution rhythm.

[0022] Before executing the navigation command, the system first verifies the communication connection status with the IoT human-computer interaction device. Specifically, it sends a GET request to a specific heartbeat detection API endpoint of the target device and only continues execution after receiving a response with an HTTP status code of 200 OK.

[0023] After a successful connection, the system sends the first command of a command sequence to the OTT large-screen device via the interface and simultaneously activates the image capture device. In this embodiment, this device is an external HDMI video capture card, which connects to the output port of the OTT device via an HDMI cable to capture the screen digital signal. The capture card uses H.264 as the video encoding format for capture and sets a constant bit rate factor of 18 to control the data bit rate while ensuring the fidelity of the visual information.

[0024] To ensure the time traceability required for event analysis and auditing, the system adds a timestamp to each frame of data in the captured video stream. In this embodiment, the timestamp format uses a string with time zone information and ISO 8601 format, and then the video stream with the timestamp is output.

[0025] By including steps such as structured task reception, connection status verification, acquisition synchronization startup, and data frame timestamp addition in the task process, the consistency and reliability of the execution of automated monitoring tasks are improved. In addition, the time-stamped video data output by this process also provides information basis with temporal traceability for subsequent event analysis and auditing.

[0026] Furthermore, the multimodal visual perception model has the following specific process: Figure 2 Shown, including: The visual spatial encoder receives the static image of the current frame in the real-time video stream as input, processes it using the first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, which serves as the local geometric structure feature; a dynamic change encoder configured to receive a differential image between a current frame and a previous frame as input, process the differential image using a second convolutional neural network, capture the area, direction, and pattern of changes on the screen, and generate a dynamic change feature vector; The command context encoder receives the navigation command that leads to the current interface state as input, uses a preset embedding lookup table and a multi-layer perceptron to process it, and maps the command text into a low-dimensional command context feature vector; The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network, generates an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual space encoder, dynamically recalibrates the feature map through the channel attention mechanism, weights the feature map, and outputs the global content perception feature.

[0027] Specifically, the visual spatial encoder extracts rich spatial details and geometric information from a single static image frame. Its design aims to provide highly accurate and robust local features for subsequent feature matching. In practice, the encoder receives the current frame of a live video stream from the screen as input. To ensure consistent processing and optimal model performance, the input image is first preprocessed and normalized, with pixel values ​​standardized.

[0028] The main body of the encoder uses the EfficientNet-B0 model as the core of the first convolutional neural network. In order to obtain geometric texture rather than highly abstract semantic features, the model does not use its final classification layer. Specifically, the feature map after the fourth MBConv module in the network structure is intercepted as the output. The feature map at this stage is relatively large in size, retaining sufficient spatial resolution. At the same time, its receptive field is large enough to capture key local structures. The high-resolution spatial content feature map rich in spatial details generated by this process is defined as the local geometric structure feature, which provides a solid foundation for subsequent precise key point matching.

[0029] To capture temporal changes and understand the user's intent, the model employs two independent encoders in parallel: a dynamic change encoder processes temporal visual changes, while a command context encoder converts discrete, human-readable navigation commands into mathematical representations with continuous semantics that neural networks can understand and process.

[0030] The system receives the current and previous frames from a live video stream as input. To focus on intensity changes in content and structure rather than color noise, the system first converts the two frames into single-channel grayscale images. It then generates a raw difference image by calculating the absolute difference between corresponding pixels in the two grayscale images. This difference image intuitively reflects all areas that have changed on the screen. To further enhance signal robustness and filter out slight acquisition noise, the system applies a binary thresholding process to the difference image, generating a binary mask image that clearly identifies areas of significant change. This processed binary image serves as the final input to the dynamic change encoder.

[0031] In terms of network structure, the lightweight MobileNetV2 model is used as the core of the second convolutional neural network. This second convolutional network receives a binary mask image and processes it through depthwise separable convolutional layers to effectively capture the changing areas, directions, and patterns.

[0032] In the final processing stage, the two-dimensional feature map output by the network passes through a global average pooling layer, which averages the feature map across the spatial dimensions, compresses it, and generates a fixed-dimensional dynamically changing feature vector. This dynamically changing feature vector represents the key information of changes on the screen, stably representing the interface dynamics and providing critical temporal context for subsequent multimodal attention fusion.

[0033] The command context encoder is a collaborative effort between a pre-configured embedding lookup table and a multi-layer perceptron. First, the lookup table is constructed, and the system defines a comprehensive command vocabulary. In this embodiment of the present invention, navigation commands consist of a series of atomic remote control commands. The system parses all possible atomic commands, such as {action: keypress, key: KEYCODE_DPAD_RIGHT}, and breaks them down into their smallest semantic units, such as keypress, wait, KEYCODE_DPAD_RIGHT, and KEYCODE_DPAD_LEFT. These unique semantic units together constitute the system's command vocabulary.

[0034] After the vocabulary is built, each unique instruction unit in it is assigned a fixed integer index, for example, keypress corresponds to index 1, wait corresponds to index 2, KEYCODE_DPAD_RIGHT corresponds to index 3, and so on. The preset embedding lookup table is created as a two-dimensional matrix. The number of rows in the matrix is ​​equal to the total number of unique instruction units in the vocabulary, and the number of columns is a preset, lower vector dimension. Each row of the matrix is ​​a randomly initialized low-dimensional vector representing the corresponding instruction unit.

[0035] When the command context encoder receives a specific navigation command text (such as KEYCODE_DPAD_RIGHT), it first looks up the integer index corresponding to the command in the vocabulary (such as 3). It then uses this index to find and extract the 128-dimensional vector corresponding to the third row of the embedding lookup table matrix. This vector is the initial numerical representation of the command. This process successfully maps discrete text symbols into a low-dimensional continuous vector.

[0036] Finally, the low-dimensional vector extracted from the lookup table is fed into a multi-layer perceptron (MLP) for nonlinear transformation and deep feature extraction, ultimately outputting a command context feature vector that fully characterizes the current operation intent. Specifically, the multi-layer perceptron consists of two fully connected layers: the first layer receives a 128-dimensional embedding vector, outputs 256 neurons, and uses a ReLU activation function; the second layer receives a 256-dimensional vector and outputs a final 128-dimensional command context feature vector.

[0037] Furthermore, the multimodal attention fusion unit receives and integrates contextual information from two different modalities. Specifically, it concatenates the dynamic change feature vector from the dynamic change encoder and the instruction context feature vector from the instruction context encoder in terms of dimension, forming a higher-dimensional composite context vector. This composite vector provides a comprehensive, multimodal mathematical description of the current monitoring state.

[0038] Next, the composite context vector is fed into the attention generation network, which consists of two parallel, independently structured multilayer perceptrons (MLPs), one for generating guidance signals for channel attention and the other for spatial attention. The first MLP receives the composite context vector and outputs a channel attention guidance vector with a dimension equal to the number of channels in the visual spatial encoder's output feature map. The second MLP also receives the composite context vector and outputs a spatial attention guidance vector for generating spatial attention. The attention generation network consists of two parallel, independently structured multilayer perceptrons (MLPs). The first MLP generates the channel attention guidance vector, with a structure of input layer - activation function (ReLU) - output layer. The second MLP generates the spatial attention guidance vector, with the same structure as the first MLP, and the number of input layers equal to the dimension of the concatenated composite vector.

[0039] The system then performs the first dynamic recalibration using a channel-wise attention mechanism. It passes the channel-wise attention guidance vector through a sigmoid activation function, normalizing its value to between 0 and 1, forming a set of explicit channel weights. These weights are then applied to the raw spatial content feature map output by the visual spatial encoder, weighting each channel of the feature map by channel-by-channel multiplication. This step intelligently enhances or suppresses the importance of different feature channels based on the current dynamics and instruction context.

[0040] After channel recalibration, the system continues to perform a second fine-tuning of the spatial attention mechanism. To incorporate contextual information into the spatial attention generation process, the system first spatially expands the spatial attention guidance vector and copies it into a context feature map with the same spatial dimensions as the channel-weighted feature map. The system then concatenates the context feature map with the channel-weighted visual feature map along the channel dimension to form a fused feature map that contains the original visual information and multimodal contextual information. Finally, the system uses this fused feature map as input and passes it through a spatial convolution layer with a larger convolution kernel (7x7 in this example) to analyze the response of the fused feature at different spatial locations and generate a single-channel heat map of the same size as the original feature map. The sigmoid activation function is then used to convert it into a spatial attention map. The value of each pixel in the map represents the importance of the corresponding spatial location. This spatial attention map is element-wise multiplied by the channel-weighted feature map. This step enables the model to focus computing resources on specific key areas in the image based on the context.

[0041] Finally, the feature map that has been dynamically recalibrated by the channel-space dual attention mechanism is passed through a global average pooling layer, which compresses its spatial dimensions and ultimately generates a fixed-dimensional global content-aware feature vector that contains the core content of the image and deeply incorporates temporal and instruction context information. This feature vector has a high degree of generalization and contextual awareness, providing high-quality input for subsequent efficient matching and screening.

[0042] By adopting a staged feature extraction and matching method, using global features with low computational overhead for high-speed preliminary screening, and limiting the high computational complexity of local feature extraction steps to only a small number of relevant candidate frames, while maintaining high recognition accuracy, the overall computational load of the system is effectively reduced and processing efficiency is improved, so that the monitoring solution can better meet the application requirements of real-time and accuracy.

[0043] Furthermore, the detailed implementation of the hierarchical matching algorithm includes: In the fast candidate frame screening step, a cosine similarity algorithm is used to perform a cyclic calculation between the global content-aware features of the target advertisement and the global content-aware features of each frame of the real-time video stream. When the calculated similarity score is higher than a preset high similarity threshold, the current video frame is determined to be a candidate frame; only within the candidate frame, a k-nearest neighbor search algorithm is used to match between local geometric structure feature descriptors, and a ratio test with a preset ratio threshold is applied for preliminary screening to generate the initial key point correspondence relationship set containing all matches.

[0044] Specifically, the system uses a cosine similarity algorithm to calculate the similarity score between the global content-aware feature vector of the target ad image and the global content-aware feature vector of each frame in the live video stream. Only when the calculated score exceeds the preset similarity threshold of 0.9 is the video frame considered a candidate frame and enters the subsequent processing flow.

[0045] During the precise matching step, the system processes only the candidate frames selected in the previous stage. For each frame, the system uses a FLANN-based k-nearest neighbor search algorithm to match the target ad image's local geometric structure feature descriptors with the current candidate frame's descriptor set. Setting the k value to 2 allows the system to identify the nearest and next-nearest neighbors for each descriptor. The system then applies a ratio test with a preset ratio threshold of 0.75 to perform a preliminary screening of the matching results. Only when the ratio of the nearest neighbor distance to the next-nearest neighbor distance is less than 0.75 is the match considered a clear and reliable correspondence and retained. Ultimately, an initial keypoint correspondence set containing all verified matching pairs is generated. The hierarchical matching algorithm incorporates a critical temporal stability check for candidate frames when determining candidate frames. This step does not determine a candidate frame simply when the similarity of a single frame exceeds a threshold. Instead, it requires that all frames within a preset time window (five consecutive frames) have similarity scores above the threshold. Only when this temporal stability condition is met will the system confirm the last frame in the window as a candidate frame and activate subsequent precise matching. This mechanism can intelligently filter out short-lived pseudo candidate frames caused by dynamic effects, ensuring that computing resources are only used for scenes that truly and stably present the target, thereby significantly improving the overall efficiency and robustness of the algorithm.

[0046] In the hierarchical matching algorithm, the computationally efficient cosine similarity algorithm is first used to preliminarily determine the candidate frame, and then only the local features within the candidate frame are subjected to the k-nearest neighbor search and ratio test for precise matching. The intensive matching calculation is concentrated on the most relevant data subset, which significantly reduces the overall computing requirements and improves processing efficiency. With the help of the ratio test, ambiguous and unclear matching pairs are effectively eliminated, ensuring the accuracy of the final generated key point correspondence, which helps to balance the requirements of processing efficiency and recognition accuracy in real-time monitoring.

[0047] Furthermore, the process of performing spatial clustering on the set to separate the advertisement instances includes: The key point spatial coordinates of each matching pair in the real-time video frame in the initial key point correspondence set are extracted, all the coordinates are formed into a point set, and the point set is input into a spatial density-based clustering algorithm. The clustering algorithm automatically divides these key points into independent clusters according to the spatial proximity relationship between the points. The membership of each cluster defines the advertising instance affiliation of the corresponding key point matching pair.

[0048] Specifically, the system iterates through each matching pair in the set and extracts the spatial coordinates (x, y) of the keypoints in the live video frame. All these extracted coordinates are combined to form a two-dimensional spatial point set, which serves as the sole input for the subsequent clustering algorithm. This two-dimensional point set is then fed into a DBSCAN algorithm. This algorithm automatically partitions keypoints into one or more independent clusters based on the spatial proximity between points using two core parameters: a neighborhood radius set to 30 pixels and a minimum number of points set to 5. Any isolated points that do not belong to a cluster are identified as noise and removed. Ultimately, the cluster membership of each keypoint coordinate in the clustering results precisely defines the instance affiliation of the original keypoint matching pair, thereby separating matching pairs belonging to different ad instances. To further improve separation accuracy in complex layouts, the spatial clustering process includes a cluster validity verification step after processing. For each generated cluster, this step first calculates its minimum bounding rectangle (MBR) and its aspect ratio. If the ratio of this MBR to the original aspect ratio of the target ad image exceeds 2.0, the system determines that the cluster has incorrectly merged multiple instances. At this point, the system automatically re-clusters the points within that particular cluster using a stricter neighborhood radius (halving the original 30-pixel neighborhood radius to 15 pixels) to attempt a secondary separation. In this embodiment, the DBSCAN algorithm is used for spatial clustering. This verification step mitigates the potential for cluster merging that can occur when multiple ad instances are closely adjacent or partially overlap, significantly improving the accuracy of target separation in complex, high-density ad layouts and avoiding missed or false detections due to incorrect clustering.

[0049] By applying a density-based clustering algorithm to the spatial coordinates of all matching key points in the video frame, the matching point pairs belonging to different advertising instances can be automatically separated into independent clusters. This helps to deal with the problem of mutual interference when multiple advertising instances appear at the same time, and can effectively filter out discrete noise matching points, providing a purer and independent data subset for subsequent geometric transformation calculations, thereby improving the accuracy and reliability of the final target positioning without significantly increasing the computational complexity.

[0050] Furthermore, the geometric transformation calculation includes: For the divided independent clusters, a random sampling consensus algorithm with a configured maximum number of iterations and a reprojection error threshold is used to process the key point correspondences within the cluster; in each iteration, random sampling is used to estimate the minimum key point matching pair set of the transformation model, and the number of inliers that conform to the estimated transformation model in all correspondences is calculated; the homography matrix with the largest number of inliers is selected and output as the optimal homography matrix for the current cluster, and the four original vertex coordinates of the target advertising image are matrix multiplied with the optimal homography matrix to obtain the bounding box of the advertising instance.

[0051] For each independent cluster divided, the system independently executes a RANSAC random sampling consensus algorithm configured with a maximum number of iterations of 1000 and a reprojection error threshold of 3.0 pixels. In each iteration, the algorithm randomly samples the minimum set of key point matching pairs required to estimate the transformation model from the cluster, that is, 4 matching pairs, to calculate a temporary homography matrix. The algorithm then uses this matrix to transform all key points in the cluster from the target image to the video frame coordinate system, and calculates the distance between the transformed coordinates and the actual matching coordinates, that is, the reprojection error. If the error is less than 3.0 pixels, the matching pair is counted as an inlier. After completing all iterations, the system selects and outputs the homography matrix with the largest number of inliers as the optimal homography matrix for the cluster.

[0052] After obtaining the optimal homography matrix for the cluster, the system enters the final bounding box generation stage. The system performs matrix multiplication on the four original vertex coordinates of the target advertising image, namely (0,0), (width,0), (width,height), and (0,height), with the optimal homography matrix and performs perspective transformation. The four new coordinate points generated by the operation constitute the bounding box of the advertising instance in the real-time video frame, thereby completing the precise positioning of the instance.

[0053] By using a random sampling consensus algorithm to calculate the geometric transformation model for each clustered key point cluster, the accuracy and robustness of the final positioning result are improved. By iterative sampling and finding the largest inlier set, the optimal homography matrix can be robustly estimated from the matching relationship containing a small amount of noise, thereby reducing the interference of incorrect matches on the final bounding box calculation, ensuring the key link of the system to output accurate and reliable monitoring results in complex matching scenarios.

[0054] Finally, the result generation module highlights the area where the target advertisement is located on the original video frame according to each of the boundary boxes and adds a visual mark to generate the final visual monitoring result.

[0055] The system visually marks the corresponding original video frame image by drawing a quadrilateral bounding box around the ad instance using high-contrast colors. Furthermore, the system independently outputs a structured JSON metadata file as core monitoring evidence. This file contains global information such as the task ID and frame timestamp, and records each ad instance in detail through a detections array. Each object provides a unique instance ID and a bounding box consisting of four [x, y] coordinate points. Separating intuitive visual evidence from precise, machine-readable data facilitates both manual verification and automated report generation and in-depth data analysis.

[0056] By organically integrating phased feature extraction processes, hierarchical matching algorithms, multi-instance spatial clustering, and geometric verification technologies and placing them in a task-driven automated control closed loop, computationally efficient global features are used to quickly narrow the analysis scope, thereby improving processing efficiency under real-time video streams. At the same time, through a series of steps such as precise matching of local features, spatial clustering to separate multiple instances, and geometric transformation to filter out noise, the recognition accuracy is improved in scenarios with complex backgrounds and the coexistence of multiple targets, ultimately helping to achieve both real-time and reliability of monitoring tasks.

[0057] Example 2: In order to realize the automatic monitoring of specific advertising creativity, the OTT visual feature extraction method based on multimodal agent driving provided by the present invention is introduced. The specific process is as follows: Figure 3 shown.

[0058] The monitoring task is to verify whether a specific ad creative, A, is displayed within the designated content recommendation area of ​​the target OTT large-screen device. The operator creates this task by calling a system API. The task content includes the image file of the target ad creative, A, and a sequence of navigation instructions for interface manipulation. This instruction sequence consists of a series of specific atomic instructions, such as [{action: keypress, key: home}, {action: wait, duration: 1000}, {action: keypress, key: right}, {action: keypress, key: right}, {action: keypress, key: ok}], which is designed to simulate remote control operation to guide the device from the home screen interface to the desired content display page.

[0059] After the mission is initiated, the system's intelligent agent connects to the target OTT device via an IoT interface. Simultaneously with the initial home button command, it activates a high-definition image capture device and begins recording a screen video stream with precise timestamps. The intelligent agent strictly follows the instructions and sequence in the JSON sequence, automatically completing the entire navigation process without any human intervention.

[0060] During device navigation, the system's multimodal visual perception model operates in real time in the background. For interface frames in the video stream that are unrelated to ad creative A (such as the homepage and app list), while the visual spatial encoder (EfficientNet-B0) extracts their visual content, the dynamic change encoder (MobileNetV2) captures the image changes caused by moving to the right, while the command context encoder parses the intent of the "right" command. This contextual information is fed into the multimodal attention fusion unit. The resulting global content-aware features do not match the features of the target ad, and their similarity scores will be far below the preset threshold. Therefore, the system effectively ignores these frames.

[0061] When the interface successfully redirects to the target page and Ad Creative A appears, the image is stable, with minimal dynamic changes, but the visual content is highly correlated with the target. When the cosine similarity between the model-generated global content-aware features and the target ad's features exceeds a preset threshold (0.9), the system immediately identifies the frame as a candidate. Only then does the system trigger the subsequent refined analysis, utilizing the extracted local geometric structure features from the intermediate layers of EfficientNet-B0 to perform precise feature point matching between the candidate frame and the target ad image. After obtaining a preliminary set of matching points, the system further processes them using a spatial clustering algorithm (DBSCAN). This algorithm clusters matching points representing Ad Creative A into a compact cluster while identifying and removing isolated mismatches caused by other similar background images as noise, thereby purifying the data used for geometric verification. The system then uses the Random Sample Consensus Algorithm (RANSAC) algorithm to calculate the precise geometric transformation relationship for this purified point set.

[0062] Finally, based on the calculated geometric transformation, the system accurately selects the location of Ad Creative A within the candidate frame using a highlighted rectangle. It also adds metadata such as the task ID and timestamp, generating a complete visual monitoring report for archiving. This optimized process ensures both real-time and reliable monitoring tasks in complex real-world scenarios.

[0063] By adopting a staged processing flow, which first uses computationally efficient global features for high-speed screening to improve overall processing efficiency, and then applies a series of refinement steps including local feature matching, multi-instance spatial clustering and geometric verification to only a small number of screened candidate frames, it can improve recognition accuracy in scenes with complex backgrounds and multiple targets, thus helping to meet the application requirements of monitoring tasks in terms of real-time performance and reliability at the same time.

[0064] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An OTT visual feature extraction system driven by a multimodal agent, characterized by: include: The monitoring task parsing module receives monitoring tasks containing target advertising images and navigation instruction sequences, sends navigation instructions to OTT devices through the IoT interface for interface control, and captures the real-time video stream of the device screen; A dual-stream heterogeneous feature extraction module processes the target advertisement image and real-time video stream and extracts dual-stream heterogeneous visual features using a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. The multi-stage matching module implements a hierarchical matching algorithm, using global content-aware features for calculations to select candidate frames. Within the candidate frames, it uses local geometric structure features for matching to establish an initial key point correspondence set containing all matches. This set is then spatially clustered to isolate ad instances, and the bounding boxes of the ad instances are calculated through geometric transformations. The result generation module highlights the target advertisement area on the original video frame based on the bounding box and adds a visual mark to generate a visual monitoring result.

2. The OTT visual feature extraction system based on multimodal agent driving according to claim 1 is characterized in that: The process of capturing the real-time video stream of the screen of the device includes: Through the application program interface, a structured monitoring task containing a unique task ID, a target advertising image, and a navigation instruction sequence consisting of multiple atomic remote control instructions is received to verify the communication connection status with the IoT human-computer interaction device. After the connection is successful, the navigation instruction sequence is sent to the OTT large-screen device through the interface in sequence to execute navigation. At the same time as sending the first instruction, the image acquisition device is started, and the real-time video stream of the screen is captured using a standard video encoding format. Each frame of data is timestamped and output.

3. The OTT visual feature extraction system based on multimodal agent driving according to claim 1 is characterized in that: The multimodal visual perception model includes: The visual spatial encoder receives a static image of the current frame in the real-time video stream as input, processes it using a first convolutional neural network, and generates a high-resolution spatial content feature map containing rich spatial details, which serves as a local geometric structure feature; The dynamic change encoder receives the difference image between the current frame and the previous frame as input, processes it using a second convolutional neural network, captures the area, direction, and pattern that have changed on the screen, and generates a dynamic change feature vector; The command context encoder receives the navigation command that leads to the current interface state as input, uses a preset embedding lookup table and a multi-layer perceptron to process it, and maps the command text into a low-dimensional command context feature vector; The multimodal attention fusion unit concatenates the dynamically changing feature vector with the instruction context feature vector, inputs the concatenated composite vector into the attention generation network, generates an attention weight vector, applies the attention weight vector to the spatial content feature map output by the visual space encoder, dynamically recalibrates the feature map through the channel attention mechanism and the spatial attention mechanism, weights the feature map, and outputs the global content perception feature.

4. The OTT visual feature extraction system based on multimodal agent driving according to claim 1 is characterized in that: The execution method of the hierarchical matching algorithm includes: In the fast candidate frame screening step, a cosine similarity algorithm is used to perform a cyclic calculation between the global content-aware features of the target advertisement and the global content-aware features of each frame of the real-time video stream. When the calculated similarity score is higher than a preset high similarity threshold, the current video frame is determined to be a candidate frame; only within the candidate frame, a k-nearest neighbor search algorithm is used to match between local geometric structure feature descriptors, and a ratio test with a preset ratio threshold is applied for preliminary screening to generate the initial key point correspondence relationship set containing all matches.

5. The OTT visual feature extraction system based on multimodal agent driving according to claim 1 is characterized in that: The process of performing spatial clustering on the set to separate the advertisement instances includes: The key point spatial coordinates of each matching pair in the real-time video frame in the initial key point correspondence set are extracted, all the coordinates are formed into a point set, and the point set is input into a spatial density-based clustering algorithm. The clustering algorithm automatically divides these key points into independent clusters according to the spatial proximity relationship between the points. The membership of each cluster defines the advertising instance affiliation of the corresponding key point matching pair.

6. The OTT visual feature extraction system based on multimodal agent driving according to claim 1 is characterized in that: The geometric transformation calculation includes: For the divided independent clusters, a random sampling consensus algorithm with a configured maximum number of iterations and a reprojection error threshold is used to process the key point correspondences within the cluster; in each iteration, random sampling is used to estimate the minimum key point matching pair set of the transformation model, and the number of inliers that conform to the estimated transformation model in all correspondences is calculated; the homography matrix with the largest number of inliers is selected and output as the optimal homography matrix for the current cluster, and the four original vertex coordinates of the target advertising image are matrix multiplied with the optimal homography matrix to obtain the bounding box of the advertising instance.

7. An OTT visual feature extraction method based on multimodal agent driving is characterized by: include: Receive monitoring tasks containing target advertising images and navigation instruction sequences, send navigation instructions to OTT devices through the IoT interface for interface control, and capture the device's screen real-time video stream; The target advertising image and real-time video stream are processed, and dual-stream heterogeneous visual features are extracted through a multimodal visual perception model. The dual-stream heterogeneous visual features include global content perception features and local geometric structure features. A hierarchical matching algorithm is implemented, using global content-aware features for calculations to select candidate frames. Within the candidate frames, local geometric structure features are used for matching to establish an initial keypoint correspondence set containing all matches. This set is then spatially clustered to isolate the ad instances, and the bounding boxes of the ad instances are calculated through geometric transformations. Based on the bounding box, the area where the target advertisement is located is highlighted on the original video frame and a visual mark is added to generate a visual monitoring result.

Images