Marketing webpage dynamic content identification and tampering detection method based on multi-modal fusion

Abstract

The invention relates to a marketing webpage dynamic content identification and tampering detection method based on multi-modal fusion. The method comprises the following steps: S1, obtaining page snapshots of the same page reported by channel merchants at multiple times; s2, key features and descriptors are taken from the page snapshots, matching is carried out, and a dynamic feature set is obtained; s3, generating a page snapshot only containing static content according to the position of the dynamic feature in the state feature set; s4, inputting the page snapshots only containing the static content into the visual feature extraction network, the structural feature extraction network and the semantic feature extraction network to obtain page feature representation; and S5, inputting the page feature representation into a tampering detection model, and visualizing a tampering detection result. Compared with the prior art, the method has the advantages of reducing false alarms of dynamic elements, improving the accuracy of tampering detection and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a marketing webpage dynamic content recognition and tampering detection based on multimodal fusion. Background Art

[0002] With the rapid development of the Internet, digital marketing has become one of the main means for enterprises to promote products and services. Enterprises usually establish digital marketing platforms and cooperate with third-party channel providers (such as agents, distributors, alliance partners, etc.) to promote products using the marketing pages they develop. Marketing pages are designed to attract potential customers, provide product information and promotional content, and guide users to complete ordering or purchasing operations. These pages usually integrate with the enterprise's digital marketing platform by calling the enterprise's back-end interface to implement functions such as order processing and payment settlement.

[0003] However, in actual operation, there are some prominent problems in the cooperation model with third-party channel providers:

[0004] (1) Weak controllability of marketing page content: Since marketing pages are independently developed and maintained by third-party channel vendors, it is difficult for enterprises to fully monitor changes in page content. Some channel vendors may modify page content without authorization, add misleading elements, exaggerate publicity, or embed illegal information. Such non-compliant content may increase sales in the short term, but in the long run, it will lead to user complaints, damage corporate reputation, and even cause legal risks.

[0005] (2) Limitations of traditional monitoring methods: Enterprises usually use manual review and fixed rule detection to manage channel partners’ marketing pages. However, this method has the following shortcomings:

[0006] Dynamic elements in the page are easily identified as tampered, resulting in false alarms. In addition, tampering includes subtle visual changes, structural element adjustments, and modifications to text content. Current detection methods are difficult to fully cover all types of tampering, and the detection accuracy is not high. Summary of the invention

[0007] The purpose of the present invention is to provide a marketing web page dynamic content recognition and tampering detection method based on multimodal fusion in order to reduce false positives of dynamic elements and improve the accuracy of tampering detection.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A marketing web page dynamic content recognition and tampering detection method based on multimodal fusion, the method comprising the following steps:

[0010] S1. Obtain page snapshots of the same page reported by channel dealers at multiple times to obtain time series data;

[0011] S2, extract key features and descriptors from the page snapshot, match the key features and descriptors on the time series data, and obtain a dynamic feature set;

[0012] S3, determining the dynamic feature boundary according to the position of the dynamic feature in the dynamic feature set, shielding the part within the dynamic feature boundary, and generating a page snapshot containing only static content;

[0013] S4, inputting the page snapshot containing only static content into the visual feature extraction network, the structural feature extraction network, and the semantic feature extraction network, fusing the visual features, structural features, and semantic features to obtain the page feature representation;

[0014] S5. Input the page feature representation into the tampering detection model and visualize the tampering detection results.

[0015] Furthermore, the specific steps of S2 are:

[0016] Select the feature extraction algorithm to extract key points from the page snapshot and obtain the key point set K t ={k t，i}, where k t，i Represents the i-th key point of the t-th snapshot. For each key point, calculate the corresponding feature descriptor D t，i , forming a feature set T t ={(k t，i , D t，i )};

[0017] For the feature set T at adjacent time points t and t+Δt t and T t+Δt Using the KNN method, find each feature descriptor D t，i In T t+Δt The nearest neighbor in the,ratio test method is used to obtain the matching pair set and the unmatched features;

[0018] For the matching key points in the matching pair set, calculate the position change and the difference of the corresponding feature descriptor. If the position change exceeds the threshold or the difference of the feature descriptor exceeds the threshold, it is judged as a feature with significant change;

[0019] The features that change significantly and the features that are not matched are taken as dynamic features to obtain a dynamic feature set.

[0020] Furthermore, the specific steps of using the ratio test method to obtain the matching pair set and the unmatched features are as follows:

[0021] If each feature descriptor D t，iIn T t+Δt If the ratio of the distance between the nearest neighbor and the next nearest neighbor in is less than the threshold, the match is reliable and the feature descriptor D t，i The corresponding key point k t，i and feature descriptor D t，i The adjacent feature descriptor D t+Δt，i The corresponding key point k t+Δt，j The two pairs are matched with each other, and the matching pair set M is obtained. t ={{k t，i , k t+Δt，j}}, the key points that do not belong to the matching pair set are unmatched features.

[0022] Furthermore, the distance is calculated using Euclidean distance.

[0023] Furthermore, the position changes as follows:

[0024]

[0025] Δx i =x t+ Δ t，j -x t，j , Δy i =y t+Δt，j -y t，j

[0026] Where Δx i is the x-axis position change of the i-th matching pair, Δy i is the change in the y-axis position of the i-th matching pair, where (x t,i ,y t，i ) and (x t+Δt,j ,y t+Δt，j ) are the coordinates of the two key points of the matching pair.

[0027] Furthermore, the difference in feature descriptors is:

[0028] ΔD i =||D t，i -D t+Δt，j ||

[0029] Among them, D t，i , D t+Δt，j are the feature descriptors corresponding to the two key points of the matching pair.

[0030] Furthermore, the specific steps for determining the dynamic feature boundary are:

[0031] Use K-Means clustering algorithm to cluster the key points of dynamic features;

[0032] For each cluster, calculate its minimum enclosing rectangle and get the bounding box R of the dynamic area. j , and the bounding box R j Expand and get the dynamic feature boundary.

[0033] Furthermore, the page features are represented as:

[0034] F=α v ⊙V+α s ⊙S+α v ⊙E

[0035] Among them, F represents the page feature representation, α v , α s , α e represents the visual, structural, and semantic attention weights, and V, S, and E represent the visual, structural, and semantic features.

[0036] Furthermore, visual features are extracted by inputting page snapshots into image convolutional neural networks, structural features are aggregated by inputting the DOM tree structure of page snapshots into graph neural networks, and semantic features are obtained by inputting the text sequence corresponding to the text content of the page snapshots into a bidirectional LSTM model.

[0037] Furthermore, the specific steps for obtaining page snapshots of the same page reported by channel dealers at multiple times are as follows:

[0038] Get screenshots of the same page reported by channel dealers at multiple times, compress and denoise the screenshots, and obtain page snapshots.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention combines the visual features, structural features and semantic features of the page to comprehensively capture the multi-dimensional information of the page and improve the perception of tampering behavior. Through the fusion strategy of unified feature representation, the feature dimensions are uniformly mapped: the feature vectors of different modalities are mapped to the same dimension through the fully connected layer to create conditions for fusion. Combined with the adaptive attention mechanism, the effective integration of multimodal features is achieved, and the expressive power of feature representation is improved. Multimodal feature fusion enables the model to identify various forms of tampering behavior, including subtle visual changes, structural element adjustments and text content modifications. It has strong adaptability to different tampering strategies and can maintain stable detection performance in complex and changeable actual environments.

[0041] (2) The present invention utilizes multiple quick snapshots to detect elements that change in a short period of time and marks them as dynamic, and uses time series analysis and feature extraction to identify and separate dynamic content and static content on marketing pages. By isolating dynamic elements (such as rotating banners, real-time dynamics), the system focuses on detecting unauthorized changes in static content that are more likely to cause violations, thereby reducing false positives. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] The present invention proposes a marketing web page dynamic content recognition and tampering detection method based on multimodal fusion. The flow chart of the method is as follows: Figure 1 The method comprises the following steps:

[0046] S1. Obtain page snapshots of the same page reported by channel dealers at multiple times to obtain time series data;

[0047] S2, extract key features and descriptors from the page snapshot, match the key features and descriptors on the time series data, and obtain a dynamic feature set;

[0048] S3, determining the dynamic feature boundary according to the position of the dynamic feature in the dynamic feature set, shielding the part within the dynamic feature boundary, and generating a page snapshot containing only static content;

[0049] S4, inputting the page snapshot containing only static content into the visual feature extraction network, the structural feature extraction network, and the semantic feature extraction network, fusing the visual features, structural features, and semantic features to obtain the page feature representation;

[0050] S5. Input the page feature representation into the tampering detection model and visualize the tampering detection results.

[0051] The improvements of the present invention are:

[0052] (1) Use advanced AI technology for multimodal feature fusion:

[0053] Innovation:

[0054] A multimodal approach is adopted: integrating visual features (from web page snapshot images), structural features (from web page DOM analysis) and semantic features (from web page text content) to comprehensively determine whether a web page has been tampered with.

[0055] Neural network structure design in multimodal feature fusion: This design innovatively combines the visual features, structural features and semantic features of the page to comprehensively capture the multi-dimensional information of the page and improve the ability to perceive tampering behavior.

[0056] Fusion strategy of unified feature representation, unified mapping of feature dimensions: feature vectors of different modalities are mapped to the same dimension through a fully connected layer to create conditions for fusion.

[0057] Optimization of weighted fusion method: In the fusion process, combined with the adaptive attention mechanism, effective integration of multimodal features is achieved, which improves the expressive power of feature representation.

[0058] advantage:

[0059] Comprehensive coverage of tampering types: Multimodal feature fusion enables the model to identify various forms of tampering, including subtle visual changes, structural element adjustments, and modifications of text content.

[0060] Reduce false positive and false negative rates: The adaptive attention mechanism adjusts the feature weights according to the actual situation, reduces the misjudgment caused by a single modality, and improves the accuracy of detection.

[0061] Enhanced model robustness and generalization ability: The model has strong adaptability to different tampering strategies and can maintain stable detection performance in complex and changing actual environments.

[0062] Reduce operating costs and increase business value: Higher detection accuracy reduces reliance on manual review, saving human resources and operating costs. Timely detection and handling of tampering behaviors can maintain the company's brand image and reduce legal and reputation risks.

[0063] (2) Intelligent dynamic content detection and separation:

[0064] Innovation:

[0065] A new method is introduced that employs time series analysis and feature extraction to identify and separate dynamic and static content on marketing pages.

[0066] Leverage multiple quick snapshots to detect elements that change in a short period of time and mark them as dynamic.

[0067] advantage:

[0068] Reduced false positives: By isolating dynamic elements (e.g. rotating banners, live feeds), the system focuses on detecting unauthorized changes in static content that are more likely to indicate a violation.

[0069] Improve detection accuracy: Improve the reliability of change detection by filtering expected or benign changes, enabling more precise compliance monitoring.

[0070] (3) Adaptive Attention Mechanism:

[0071] Innovation:

[0072] Introducing an adaptive attention mechanism in deep learning models to prioritize key areas of a page, such as key promotional information or legal disclaimers.

[0073] Dynamically adjust focus based on importance and likelihood of compliance violations in different page areas.

[0074] advantage:

[0075] Increased sensitivity to critical changes: Improves the model’s ability to detect unauthorized modifications in high-risk areas, thereby improving overall compliance enforcement.

[0076] Resource optimization: Achieve better scalability by allocating computing resources more efficiently by focusing on what matters.

[0077] The present invention addresses the limitations of the prior art through an innovative approach combining advanced artificial intelligence techniques, intelligent content analysis, and scalable system design. Specifically, it introduces:

[0078] (1) A sophisticated dynamic content processing method that uses time series analysis and feature extraction to identify and separate dynamic and static content on marketing pages, ensuring the focus and accuracy of compliance monitoring.

[0079] (2) A multimodal analysis framework that leverages visual, structural, and semantic data to detect complex compliance violations.

[0080] (3) The neural network structure in multimodal feature fusion aims to address the limitations of existing tampering detection technology when dealing with complex and diverse marketing pages. To this end, a neural network structure for multimodal feature fusion is designed, which comprehensively utilizes visual features, structural features, and semantic features to comprehensively and accurately characterize the content and structure of marketing pages. Through the fusion of deep learning models, the accuracy, robustness, and adaptability of tampering detection are improved.

[0081] (4) Scalable and efficient architecture, suitable for real-time monitoring in large enterprise environments.

[0082] By overcoming the challenges associated with current solutions, the present invention provides a comprehensive and effective tool for ensuring the integrity and compliance of third-party marketing pages, thereby protecting the reputation of an organization and minimizing legal and regulatory risks.

[0083] The present invention aims to provide a method and system for tampering detection and compliance monitoring of marketing pages of third-party channel merchants, so as to overcome the shortcomings of the prior art in terms of efficiency, accuracy, adaptability and scalability. Specifically, the purpose of the present invention is to:

[0084] (1) Improve detection accuracy and robustness: By introducing multimodal feature fusion technology, the visual features, structural features, and semantic features of the page are comprehensively utilized to construct a comprehensive page feature representation, enhance the detection capability of complex tampering behaviors, and reduce the false positive and false negative rates.

[0085] (2) Intelligent identification and separation of dynamic content: Using intelligent dynamic content detection methods based on time series analysis and feature extraction, the dynamic elements in the page are automatically identified and their interference with the comparison results is eliminated, ensuring that tampering detection focuses on static core content and improving detection accuracy.

[0086] (3) Enhance the model's adaptability and learning ability: Adopt an adaptive attention mechanism and deep learning model to enable the system to dynamically adjust its focus based on the importance of different areas of the page, automatically learn and adapt to new violations and tampering methods, and improve the effectiveness and adaptability of detection.

[0087] (4) Realize real-time and efficient tampering detection: Design a scalable system architecture and efficient processing flow to support real-time monitoring of a large number of channel marketing pages, meet the enterprise's requirements for real-time and performance, and reduce computing resource consumption.

[0088] (5) Support for multiple scenarios and scalable applications: Build a modular and configurable system architecture that is not only suitable for advertising monitoring of value-added telecommunications services, but can also be extended to compliance monitoring of marketing pages in other industries such as finance, insurance, and healthcare.

[0089] In general, the present invention will provide an efficient, intelligent and reliable marketing page tampering detection and compliance monitoring solution for Internet marketing. This solution helps enterprises effectively manage the marketing activities of third-party channel merchants, maintain brand image, improve user experience, reduce legal and compliance risks, and promote the healthy development of business.

[0090] This invention proposes a method and system for tampering detection and compliance monitoring of marketing pages of third-party channel merchants. The solution comprehensively uses advanced technologies such as multimodal feature fusion, adaptive attention mechanism, deep learning model and intelligent dynamic content detection, aiming to efficiently and accurately detect unauthorized modification of marketing pages and ensure the legality and compliance of page content.

[0091] By multimodally fusing the visual, structural, and semantic features of marketing pages, a comprehensive page feature representation is constructed. At the same time, time series analysis and feature extraction techniques are used to intelligently identify and separate dynamic content in the page to avoid interference of dynamic changes on the detection results. Adaptive attention mechanisms and deep learning models are used to detect tampering of processed pages and accurately identify unauthorized modification behaviors. The entire system is efficient, accurate, and scalable, and can meet the needs of large-scale real-time monitoring.

[0092] The system corresponding to the method of the present invention is mainly divided into the following modules: page snapshot acquisition module, dynamic content detection and separation module, multimodal feature fusion module, tampering detection module, alarm and visualization module, system management and security module.

[0093] Relationships and interactions between modules:

[0094] The page snapshot collection module transmits the collected page snapshots to the dynamic content detection module.

[0095] The page snapshot processed by the dynamic content detection module is passed to the multimodal feature fusion module.

[0096] The feature representation generated by the multimodal feature fusion module is input into the tampering detection module for comparison.

[0097] The detection results of the tampering detection module are processed by the alarm and visualization module.

[0098] The system management and security module runs through the entire system and is responsible for the coordination and security of each module.

[0099] The following is a detailed discussion of each module of the present invention:

[0100] Page snapshot collection module: The page snapshot collection module is one of the basic modules of the present invention, which is responsible for efficiently and accurately obtaining snapshots of third-party channel marketing pages, and providing raw data support for subsequent dynamic content detection, multimodal feature fusion and tampering detection. This module uses automated page loading and rendering technology, combined with reasonable scheduling strategies and data preprocessing methods, to ensure that the collected page snapshots are consistent, complete and high-quality, laying the foundation for the performance and accuracy of the entire system. The technical implementation details are as follows:

[0101] 1. Page snapshot collection process:

[0102] Goal: Automatically and efficiently obtain a complete snapshot of a channel partner's marketing page, including all visual content and related resources of the page.

[0103] method:

[0104] (1) Page list acquisition: Channel merchant reporting page: Get the page URLs to be collected from the marketing page list submitted by the channel merchant. Page update monitoring: Regularly check the channel merchant's page updates and dynamically maintain the list of pages to be collected.

[0105] (2) Page loading and rendering: Browser automation tools: Use automation tools such as Selenium, Puppeteer or Playwright. The present invention uses Selenium to simulate the real browser environment to load the page. Environment configuration: Browser type: Select the mainstream browser kernel (such as Chrome, Firefox) to ensure consistent page rendering effects. User agent settings: Simulate the access behavior of real users to prevent being blocked by the anti-crawler mechanism of the channel dealer. Network condition simulation: Different network delays and bandwidths can be simulated as needed to ensure that page resources are loaded. Waiting strategy: Fixed delay: Wait for a certain period of time (such as 5 seconds) after the page is loaded to ensure that dynamic resources are loaded. Element loading detection: Determine whether the page is fully rendered by detecting the loading status of specific elements in the page. Network request monitoring: Listen to network requests and wait for all necessary resources to be loaded.

[0106] (3) Page snapshot capture: Full-screen screenshot: Get a screenshot of the entire page, including the part outside the window range, to ensure the integrity of the page content. Screenshot format: Select a lossless compression format (such as PNG) to save the screenshot and retain image details. Screenshot size: Uniformly set the screenshot resolution (such as 1920x1080) to ensure consistency in subsequent processing.

[0107] (4) Data preprocessing: Image optimization: compress and denoise the screenshots to reduce storage and transmission costs. File naming and storage: save the screenshots according to a unified naming rule (such as page ID + timestamp) for easy retrieval and management.

[0108] 2. Scheduling and management:

[0109] Objective: To efficiently manage snapshot collection tasks for a large number of pages, ensuring the rational use of system resources and the timely completion of collection work.

[0110] method:

[0111] (1) Task scheduling system: Task queue: Add the URL of the page to be collected to the task queue and process it according to priority and time order. Multithreading or multi-process: Use parallel computing to run multiple browser instances at the same time to improve collection efficiency. Load balancing: Dynamically adjust the execution frequency and concurrency of tasks according to system resource usage.

[0112] (2) Failure retry mechanism: Exception handling: For abnormal situations such as loading failure and timeout, log and retry. Retry strategy: Set the maximum number of retries and retry interval to avoid infinite loops and resource waste.

[0113] (3) Resource monitoring and management: System resource monitoring: Real-time monitoring of CPU, memory, network and other resource usage to prevent overload. Current limiting strategy: When resource usage reaches the threshold, limit the launch of new tasks to ensure system stability.

[0114] 3. Data consistency and reliability assurance

[0115] Objective: Ensure that the collected page snapshots are highly consistent and reliable, and provide accurate data for subsequent detection.

[0116] method:

[0117] (1) Environmental consistency: Fixed browser version: Use the same version of the browser to avoid rendering differences. Disable cache: Disable browser cache every time you load a page to ensure that the latest page content is obtained. Time synchronization: The server time is synchronized with the standard time to facilitate logging and task scheduling.

[0118] (2) Network stability: Network environment optimization: Use a stable network connection to reduce page loading failures caused by network fluctuations. Proxy server: Use a proxy server when necessary to avoid IP restrictions and regional access restrictions.

[0119] (3) Data verification: File integrity verification: Generate a verification code (such as MD5, SHA-256) for the screenshot file to verify whether the file is complete. Duplicate data detection: Detect and remove duplicate or erroneous snapshots to ensure the uniqueness and accuracy of the data.

[0120] 4. Privacy and Security

[0121] Objective: To protect the privacy of distributors and users and comply with relevant laws and regulations during the page snapshot collection process.

[0122] (1) Sensitive information filtering: Content identification: Before saving the snapshot, identify and blur sensitive information (such as user accounts, personal information) on the page.

[0123] (2) Data encryption storage: Encryption algorithm: Encrypt the snapshot files to prevent unauthorized access. Access control: Set up strict permission management so that only authorized modules and personnel can access the snapshot data.

[0124] Dynamic content detection and separation module:

[0125] Dynamic content detection and separation is one of the key technologies of this invention, which aims to identify dynamically loaded or changing content in marketing pages (such as advertising carousels, animation effects, real-time updated data, etc.), and block or remove it from the page to generate a page snapshot containing only static content. This can avoid the interference of dynamic content on tampering detection and improve the accuracy and reliability of detection. The technical implementation details are as follows:

[0126] 1. Multiple page snapshot collection

[0127] Objective: Get multiple snapshots of the same page in a short period of time to form time series data and capture the changing characteristics of dynamic content.

[0128] Collection time interval setting: Set a short time interval Δt (such as 1-3 seconds) to collect N page snapshots within T time to form a snapshot sequence {i t}, where t=t0, t0+Δt, t0+2Δt,..., t0+(N-1)Δt.

[0129] Get page snapshot: Use browser automation tools (such as Selenium) to load the page, ensuring that the environment is consistent each time (such as browser window size, user agent, etc.). Take a full-screen screenshot of the page to get a complete page image. t

[0130] Preprocessing: Preprocess the snapshots, including resizing and color space conversion (such as conversion to grayscale images) to ensure image comparability.

[0131] 2. Feature extraction:

[0132] Objective: Extract key features and descriptors from each page snapshot as the basis for subsequent matching and change detection.

[0133] (1) Selected feature extraction algorithm: SIFT (Scale-Invariant Feature Transform): It is invariant to scale and rotation and is suitable for detecting complex texture and edge features.

[0134] (2) Feature point detection: For each snapshot i t , use the selected algorithm to detect the key point set K t ={k t，i}, where kt，i represents the i-th key point of the t-th snapshot.

[0135] (3) Feature descriptor calculation: For each key point, calculate the corresponding feature descriptor D t，i , forming a feature set F t ={(k t,i , D t，i )}.

[0136] 3. Time series analysis and matching

[0137] Objective: Match feature points in time series, analyze feature changes, and identify dynamic content.

[0138] Feature matching: For the feature set T of adjacent time points t and t+Δt t and T t+Δt , perform feature matching.

[0139] The specific matching algorithm uses the KNN (K-Nearest Neighbors) method to find each feature descriptor D t，i In T t+Δt According to the type of feature descriptor (SIFT), the Euclidean distance is used as the distance metric, and the ratio test method is adopted. That is, if the distance ratio between the nearest neighbor and the next nearest neighbor is less than the threshold θ (such as 0.75), the match is considered reliable. Finally, the matching pair set M is formed. t ={{k t，i , k t+Δt，j}}

[0140] Feature change analysis: Calculate the position change of matching key points:

[0141] Δ xi =x t+Δt,j -x t，j , Δ yi =y t+Δt,j -y t,j , where (x t,i ,y t，i ) and (x t+Δt,j ,y t+Δt，j ) are the coordinates of the matching key points. Calculate the difference in feature descriptors:

[0142] ΔD i =||D t，i -D t+Δt，j ||

[0143] Determine whether the feature change is significant: if the position changes Exceeding the threshold T p , or the descriptor difference ΔD iExceeding the threshold T d , then the feature is considered to have changed significantly.

[0144] Dynamic feature marking: Features that cannot be matched at adjacent time points or features that change significantly are marked as dynamic features. The occurrence and change of each feature in the entire time series are counted to form a dynamic feature set D dyn .

[0145] 4. Dynamic content area positioning

[0146] Goal: Determine the area where dynamic content is located based on the location of dynamic features.

[0147] Key point clustering: Use the K-Means clustering algorithm to cluster the key points of dynamic features to form a dynamic region candidate set.

[0148] Region boundary determination: For each cluster, calculate its minimum enclosing rectangle and obtain the bounding box R of the dynamic region. j Considering the detection error of feature points and the continuity of the region, the bounding box needs to be appropriately expanded (such as increasing a certain pixel margin).

[0149] 5. Dynamic content blocking

[0150] Objective: Block the identified dynamic content areas and generate pages containing only static content.

[0151] Use pixel filling to fill the pixels in the dynamic area with a uniform color (such as black or white), and then traverse each dynamic area R j , snapshot I of the original page t Perform shielding operation to obtain the final processed page T′ t .

[0152] 6 Dynamic content mask generation (optional)

[0153] Objective: Generate a mask map of dynamic content for use in subsequent feature extraction and comparison.

[0154] (1) Mask initialization: Create a binary mask image M with the same size as the page snapshot t , the initial value is 0.

[0155] (2) Mask update: For each dynamic region R j , set the value of the corresponding area on the mask map to 1.

[0156] (3) Applying masks: In the subsequent feature extraction and comparison process, ignore the areas with mask value 1 and focus on the static content.

[0157] Multimodal feature fusion module: Multimodal feature fusion is one of the core technologies of this invention, which aims to comprehensively utilize the visual features, structural features and semantic features of the marketing page to build a comprehensive and unified page feature representation. By fusing information from different modalities, the system can more accurately capture the complex changes in page content and improve the detection accuracy of unauthorized tampering. The technical implementation details are as follows:

[0158] 1. CNN network structure design for visual feature extraction:

[0159] Network input: Input data: Marketing page screenshot image after dynamic content masking. Input size: Assume the image size is H×W×3, where H and W are the height and width of the image respectively, and 3 represents the 3 channels of RGB. Network architecture: Use classic convolutional neural network (CNN) architecture, such as ResNet, VGG, or customized shallow network to balance feature extraction effect and computational efficiency. Convolution layer block 1: Convolution layer: kernel size 3×3, number of kernels 32, stride 1, padding "same". Activation function: ReLU. Normalization layer: Normalize the convolution output to stabilize the training process. Max pooling layer: Pooling window 2×2, stride 2, reduce the size of feature map. Convolution layer block 2: Convolution layer: kernel size 3×3, number of kernels 64, stride 1, padding "same". Activation function: ReLU. Normalization layer: Normalize the convolution output to stabilize the training process. Max pooling layer: pooling window 2×2, stride 2. Convolutional layer block 3 Convolutional layer: kernel size 3×3, number of kernels 128, stride 1, padding "same". Activation function: ReLU. Normalization layer: normalize the convolution output to stabilize the training process. Max pooling layer: pooling window 2×2, stride 2. Global average pooling layer: perform global average pooling on the feature map to obtain a fixed-length feature vector. Fully connected layer: output dimension: d v (Visual feature dimension, 256 or 512 Activation function: ReLU. Network output: visual feature vector

[0160] 2. Visual feature extraction:

[0161] Get the visual characteristics of the page, including layout, color, images and other visual elements, reflecting the overall visual effect of the page.

[0162] 1) Page snapshot processing: pre-process the collected page snapshots, adjust the size and normalize them to meet the model input requirements. 2) Convolutional neural network (CNN) feature extraction. Model selection: use pre-trained CNN models such as ResNet, VGG, Inception, etc. Feature layer selection: extract high-level visual features from a certain layer of the model (such as the feature layer before the fully connected layer). Feature vector acquisition: flatten the extracted feature layer output to form a visual feature vector where d v is the feature dimension.

[0163] 3) CNN network structure for visual feature extraction

[0164] 3. Neural network structure design for structural feature extraction:

[0165] Network input: Input data: DOM tree structure of the page, including node tag names, attributes and hierarchical relationships. Data preprocessing: Node representation: Tag name encoding: Convert HTML tag names (such as "div", "span") into one-hot encoding or embedding vectors. Attribute encoding: Encode node attributes (such as "id", "class") through hashing techniques or embedding methods. Feature vector s i : Concatenate the encoding results of the tag name and the attribute to form the initial feature vector of the node. Construct the graph structure: Node set N: all nodes in the DOM tree. Edge set E: parent-child and sibling relationships between nodes, represented as directed or undirected edges. Graph Neural Network (GNN) Architecture Design Input Layer: Node Feature Matrix: Where N is the number of nodes, d s is the initial feature dimension. Adjacency matrix: Represents the connection relationship between nodes. Graph convolution layer: Update rule for the lth layer: Add self-connected adjacency matrix. yes The degree matrix of H (l -1) : is the node feature matrix of the l-1th layer. W (l) : is the trainable weight matrix of the lth layer. σ: activation function, such as ReLU. Readout layer: node feature aggregation: aggregate the features of all nodes, such as summing or averaging, to obtain the feature representation of the entire graph. Fully connected layer: output dimension: d ′ s (Structural feature dimension, such as 256 or 512). Activation function: such as ReLU. Network output: Structural feature vector: Represents the high-level features of the page structure, which are used for subsequent feature fusion.

[0166] 4. Structural feature extraction:

[0167] Objective: Obtain the structural characteristics of the page, reflecting the DOM tree structure and element hierarchical relationship of the page.

[0168] 1) DOM tree parsing: HTML parsing: Use a parser (such as BeautifulSoup, lxml) to parse the HTML source code of the page and build a DOM tree.

[0169] 2) Node feature representation: Node attribute extraction: For each node, extract the label name, attributes (such as id, class), location and other information. Feature encoding: Encode the node attributes into feature vectors where d s It is the structural feature dimension.

[0170] 3) Graph representation construction: Node set N: all nodes of the DOM tree. Edge set E: parent-child and sibling relationships between nodes. Construct graph G = (N, E) for subsequent graph neural network processing.

[0171] 5. Network structure design for semantic feature extraction:

[0172] Network input: Input data: text content in the page, preprocessed text sequence. Data preprocessing: Text cleaning: remove HTML tags, special symbols, and extract plain text. Word segmentation and annotation: divide the text into words or subwords (for Chinese, characters or words can be used as basic units). Word embedding layer: Pre-trained word vectors: Use pre-trained word vector models (such as Word2Vec, GloVe, BERT) to map words to embedding vectors. Embedding matrix: Where T is the length of the text sequence, d e is the embedding dimension. Network structure: bidirectional LSTM model is used to encode text. Bidirectional LSTM layer: number of hidden units: h. Forward LSTM: processes text sequence from front to back. Backward LSTM: processes text sequence from back to front. Output: concatenate the hidden states in both directions to get Fully connected layer: Output dimension: d ′ e (semantic feature dimension, 256 or 512). Activation function: ReLU. Network output: semantic feature vector: Represents the high-level semantic features of the page text content, which are used for subsequent feature fusion.

[0173] 6. Semantic feature extraction:

[0174] Goal: To obtain semantic features of a page, capturing the meaning and theme of the text content.

[0175] 1) Text content extraction: Text node acquisition: Extract all nodes containing text (such as titles, paragraphs, button texts) from the DOM tree.

[0176] Text cleaning: Remove HTML tags and special characters, and retain only the pure text content.

[0177] 2) Text preprocessing: Word segmentation and part-of-speech tagging: Segment the text and tag the part of speech. Stop word filtering: Remove common meaningless words (such as "de", "le", "he").

[0178] 3) Text embedding representation: Word vector model: Use a pre-trained word vector model (such as Word2Vec, GloVe, BERT) to convert words into vector representations. Text representation generation: Average, weighted average the word vectors of a sentence or paragraph, or obtain the overall semantic vector through a sentence embedding model (such as Sentence-BERT)

[0179] 4) Semantic feature vector: Fuse the semantic vectors of all text content to form the semantic feature vector of the page

[0180] 7. Network structure design of the feature fusion layer: Acquisition of feature vectors Visual features: Structural features: Semantic features: Adjust each feature vector to the same dimension d f (such as 256 or 512), and use a fully connected layer to map the feature vectors to a unified dimension.

[0181] Adaptive attention mechanism: Calculate the weights of each modality feature: α v = σ(W v V + b v ), α s = σ(W s S + b s ), α e = σ(W e E + b e ), where W v , W s , W e are weight matrices, b v , b s , b e are bias vectors. σ is the activation function sigmoid. Normalize the weights to ensure that the sum of the weights is 1:

[0182] Feature fusion: Weighted fusion: F = β v V + β s S + βe E where

[0183] 8. Fusion layer feature extraction:

[0184] Objective: To effectively integrate visual, structural, and semantic features and construct a unified page feature representation F.

[0185] 1) Feature alignment: Dimension unification: adjust the dimensions of feature vectors of different modes to ensure fusion. Normalization: normalize the feature vectors to eliminate dimensional differences.

[0186] 2) Fusion strategy: Direct concatenation: directly concatenate the feature vectors of each modality F = [V; S; E]. Weighted fusion: assign weights to each modality feature F = α v V+α s S+α e E, where α v , α s , α e is the weight parameter, which can be obtained through training.

[0187] Adaptive fusion: Attention mechanism: Introduce the attention mechanism to calculate the importance weight of each modality feature. Calculate the attention weight: α v =softmax(W v V), α s =softmax(W s S), α e =softmax(W e E)W v , W s , W e is a trainable parameter matrix. Fusion calculation: F = α v ⊙V+α s ⊙S+α v ⊙E, where ⊙ represents the bit-wise multiplication of elements.

[0188] 3) Fusion model training: Objective function: In the subsequent tampering detection tasks, the parameters of the fusion model are updated in reverse by optimizing the objective function (such as similarity loss, classification loss). Prevent overfitting: Use regularization techniques (such as Dropout, L2 regularization) to prevent model overfitting and improve generalization ability. Generate unified feature representation: Through feature fusion, a unified page feature representation is obtained. Used for subsequent tampering detection.

[0189] Tamper Detection Module:

[0190] Tamper detection is the core functional module of this invention, which aims to analyze the marketing pages after dynamic content shielding and multimodal feature fusion processing to detect whether there is unauthorized modification or tampering. This module uses an adaptive attention mechanism and a deep learning model to accurately identify subtle changes and complex tampering of the page, improving the accuracy and robustness of detection. The technical implementation details are as follows:

[0191] 1. Adaptive Attention Mechanism

[0192] Objective: To introduce an adaptive attention mechanism into the tampering detection model, dynamically adjust the model's attention weight according to the importance and degree of change of different areas of the page, and enhance the sensitivity to tampering of key areas.

[0193] Attention weight calculation:

[0194] Input features: unified feature representation F generated by multimodal feature fusion module, attention score calculation: A = softmax(W a F+b a ), where W a and b a is the trainable parameter matrix and bias vector, is the attention weight vector, and A represents the degree of attention the model pays to different features of the page.

[0195] Weighted feature representation: Weighted calculation: F′=A⊙F, where ⊙ represents the bit-by-bit multiplication of elements and F′ is the weighted feature representation, which strengthens the influence of important features.

[0196] 2. Deep Learning Comparison Model

[0197] Objective: Build a deep learning model to compare the feature representations of the current page and the benchmark page, calculate the similarity, and determine whether there is tampering.

[0198] Model selection: Siamese neural network e Network): It consists of two sub-networks that share parameters and can effectively compare the similarity of two inputs.

[0199] Model structure: Input layer: weighted current page feature F′ current and the benchmark page feature F′ baseline . Feature extraction subnetwork: Shared parameters: The two subnetworks share the same weight θ to perform further high-level feature extraction on the input features. Extraction process: E current =f θ (F′ current ), E baseline =f θ (F′ baseline ) Among them, E currentand E baseline is the extracted high-dimensional feature vector. Similarity calculation layer: Calculation method: Euclidean distance D = || E current -E baseline ||Output layer: Tampering judgment: Based on the similarity or distance, the judgment result y∈{0, 1} is output, where 0 indicates no tampering and 1 indicates tampering. Model training: Loss function: Contrastive Loss: L=y·D 2 +(1-y)·max(0,mD) 2 , where m is the set distance threshold and y is the label. Training data: Positive sample pairs: normal page pairs, label 0. Negative sample pairs: pages containing tampering, label 1. Optimization method: Use the gradient descent algorithm (SGD) to update the model parameters.

[0200] 3. Anomaly detection mechanism

[0201] Objective: In addition to detecting known types of tampering, it is also possible to identify unknown types of tampering and enhance the robustness of the model.

[0202] Method: An autoencoder is used, which consists of an encoder and a decoder, to learn the low-dimensional representation of data. The training uses the feature F of normal pages for unsupervised training, so that the model can reconstruct the input. During the detection, for the test page, the reconstruction error is calculated: in is the reconstructed feature. If the reconstruction error exceeds the threshold T e , it is considered that there is an abnormality and it may be tampered.

[0203] 4. Judgment strategy and threshold setting

[0204] Multi-indicator fusion judgment: Combine the similarity S of the twin network, the reconstruction error E of the autoencoder, and other possible indicators, and use weighted or rule-based comprehensive judgment. Threshold setting: Use the validation set, adjust the threshold, draw the ROC curve, and select the best threshold.

[0205] Alarm and visualization module, system management and security module: Alarm and visualization module, system management and security module are important auxiliary parts of the present invention, responsible for processing and displaying tampering detection results, as well as overall management and security of the system. Although they are not the core innovation, they are crucial to the integrity and practicality of the system.

[0206] 1. Alarm and visualization module

[0207] Function Overview: Alarm function: When unauthorized tampering is detected on the marketing page, an alarm message is generated in time to notify relevant personnel or systems to handle it. Visualization function: Display the detection results in an intuitive way, including the specific location of the tampering, content differences, etc., to help users quickly understand and deal with the problem.

[0208] Specific technical implementation:

[0209] Alarm mechanism: Alarm trigger: According to the output results of the tampering detection module, when it is determined that there is tampering, the alarm is triggered. Alarm method: Supports multiple notification methods, such as email, SMS, system message, etc. Alarm content: Contains key information such as page identification, detection time, tampering details, etc.

[0210] Visual display: Difference highlighting: Compare the original page and the current page, and highlight the tampered part in a bright color. Comparison view: Provides a side-by-side display of the original page and the current page, allowing users to intuitively view the differences. Interactive interface: Supports operations such as zooming in, zooming out, and dragging to help users drill down to specific details.

[0211] 2. System management and security module

[0212] Functional Overview:

[0213] System management: responsible for the system's user management, authority control, configuration management, log recording, etc., to ensure the stable operation and maintenance of the system. Security assurance: provide data security, access control, protection mechanisms, etc. to protect the system from attacks and data leaks.

[0214] Specific technical implementation:

[0215] User management and permission control: User role definition: Set different user roles (such as administrator, auditor) and grant corresponding permissions. Identity authentication: Use secure authentication methods (such as OAuth, JWT) to ensure the legitimacy of user identities. Permission management: Role-based access control (RBAC) limits user access to system functions and data. Configuration management: Parameter setting: Provides a friendly interface to facilitate administrators to adjust system parameters (such as detection thresholds, alarm methods). Configuration backup and recovery: Supports backup and recovery of system configurations to ensure system stability. Logging and auditing: Operation log: Records user operation behaviors to facilitate auditing and problem tracking. System log: Records system operating status and abnormal information to assist in system maintenance.

[0216] Security protection: Data encryption: Encrypt sensitive data for storage and transmission to prevent data leakage. Protection mechanism: Detect and defend against common security threats, such as SQL injection, XSS attack, etc. Abnormal monitoring: Real-time monitoring of abnormal system behavior, timely warning and handling of security incidents.

[0217] The overall operation process of the present invention consists of the following main steps. According to the time sequence and data flow, the system realizes tampering detection and compliance monitoring of the marketing pages of third-party channel merchants:

[0218] Page snapshot collection:

[0219] Step 1: The system obtains the page to be tested from the marketing page URL list reported by the channel dealer.

[0220] Step 2: Use a browser automation tool (such as Selenium) to automatically load each page through the page snapshot collection module to ensure that the page content is fully rendered.

[0221] Step 3: Take multiple snapshots of each page within a preset time interval to generate a time series of page screenshots to ensure that changes in dynamic content are captured.

[0222] Step 4: Preprocess the captured snapshots, including size normalization, format conversion, and image optimization, in preparation for subsequent processing.

[0223] Dynamic content detection and separation:

[0224] Step 5: Input the snapshots of the same page at different time points into the dynamic content detection module.

[0225] Step 6: Use a feature extraction algorithm (such as SIFT) to extract feature points from each snapshot and obtain key feature descriptors.

[0226] Step 7: Match the snapshot features of adjacent time points, analyze the changes in feature points, and identify dynamically changing content elements.

[0227] Step 8: Based on the identified dynamic content, a mask of the dynamic area is generated or the dynamic area is directly shielded to obtain a page snapshot containing only static content.

[0228] Multimodal feature fusion:

[0229] Step 9: Extract multimodal features from the processed static page snapshot:

[0230] Visual features: Use convolutional neural networks (CNN) to extract the visual feature vector of the page. Structural features: Use graph neural networks (GNN) to extract the structured feature representation of the page. Semantic features: Perform natural language processing (BERT) on the page text content to obtain the semantic feature vector.

[0231] Step 10: Fusion layer neural network (designed by the present invention) and adaptive attention mechanism are used to fuse the features of different modalities and generate a unified page feature representation.

[0232] Tamper Detection:

[0233] Step 11: Input the fusion features of the current page and the fusion features of the reference page (original reporting page) into the tampering detection module.

[0234] Step 12: Calculate the similarity or distance between the features of two pages through the Siamese Network to determine whether the page has been tampered with.

[0235] Step 13: At the same time, use the autoencoder to perform anomaly detection and identify unknown types of tampering behaviors.

[0236] Step 14: Combine multiple detection indicators and use the set threshold or strategy to ultimately determine whether there is unauthorized modification on the page.

[0237] Alarm and Visualization:

[0238] Step 15: If it is detected that the page has been tampered with, the alarm mechanism is triggered and a notification is sent to relevant personnel.

[0239] Step 16: In the visualization module, a highlighted display of the tampered content is generated, providing a comparison view of the original page and the current page to facilitate manual review and processing.

[0240] System Administration and Security:

[0241] Step 17: The system management module monitors and coordinates the entire process, including task scheduling, resource management, logging, etc.

[0242] Step 18: The security module ensures the secure transmission and storage of data, protects user privacy, and guards against potential security threats.

[0243] Result feedback and iterative optimization:

[0244] Step 19: Based on the results of manual review and processing, update and optimize the detection model to improve the detection accuracy and adaptability of the system.

[0245] Step 20: The system runs continuously and monitors the channel partners’ marketing pages regularly or in real time, forming a closed-loop monitoring and optimization mechanism.

[0246] Through the above operation process, the present invention realizes the all-round and intelligent tampering detection and compliance monitoring of the marketing pages of third-party channel merchants. The various modules work closely together to form an efficient and reliable system: the page snapshot collection module provides the system with high-quality and consistent raw data, ensuring the reliability of subsequent processing. The dynamic content detection and separation module effectively identifies and shields dynamic content, eliminates the interference of dynamic changes on tampering detection, and improves the accuracy of detection. The multimodal feature fusion module comprehensively utilizes visual, structural and semantic features to construct a comprehensive page feature representation, enhancing the system's ability to identify complex tampering behaviors. The tampering detection module combines the adaptive attention mechanism and the deep learning model to accurately determine whether the page has been tampered with, and has the ability to detect unknown types of tampering. The alarm and visualization module promptly conveys the detection results, provides an intuitive display of tampered content, and assists manual rapid problem handling. The system management and security module ensures the stable operation and data security of the system, and provides a complete management and protection mechanism. The entire process reflects the innovation and practicality of the present invention in the technical solution, and has good scalability and adaptability. The system can monitor a large number of channel marketing pages in real time or regularly, promptly detect and handle unauthorized modifications, and safeguard the company's brand image and legitimate rights and interests.

[0247] When the above steps are actually executed, they are as follows:

[0248] Page snapshot collection module:

[0249] Step 1: Get the URL of the page to be collected from the list reported by the channel provider and add it to the task queue.

[0250] Step 2: Start a browser automation tool (such as Selenium) and configure the browser environment (Chrome browser, headless mode, disable cache).

[0251] Step 3: Set the user agent to "Mozilla / 5.0 (Windows NT 10.0; Win64; x64)" to simulate real user access.

[0252] Step 4: Load the page URL and set the maximum loading time to 30 seconds.

[0253] Step 5: Wait for a specific element in the page (such as a DIV with ID "content") to load, or wait for a fixed time of 5 seconds.

[0254] Step 6: Call the full-screen screenshot function of the browser, get the page snapshot, and save it as "page_snapshot_20241001_101500.png".

[0255] Step 7: Compress and encrypt the screenshots and store them in the specified folder.

[0256] Step 8: Record logs, including page URL, collection time, file path, collection status, and other information.

[0257] Technical feasibility: Tool maturity: Browser automation tools (such as Selenium and Puppeteer) are powerful, support multiple programming languages ​​and browsers, and have mature technology. Controllable resource requirements: Through reasonable scheduling and resource management, a large number of page collection tasks can be completed under limited hardware conditions. Scalability: Modular design makes it easy to expand new functions (such as supporting more browsers and adding page loading strategies).

[0258] Key technical difficulties and solutions:

[0259] Handling dynamic content loading

[0260] Difficulty: Dynamic content in the page may cause unpredictable loading time and affect collection efficiency.

[0261] Solution: Combine fixed waiting time and element loading detection to ensure that the main content of the page is loaded; set a timeout to prevent long waiting times.

[0262] Response to anti-crawler mechanisms:

[0263] Difficulty: Some channel partners may set up anti-crawler mechanisms to prevent access by automated tools.

[0264] Solution: Simulate real user behavior, use random user agents and request headers; use human-computer interaction verification (such as CAPTCHA) solutions when necessary.

[0265] Network fluctuations and access restrictions:

[0266] Difficulty: Unstable network or restricted IP may cause page loading failure.

[0267] Solution: Use a stable network environment; configure a proxy server, switch IP addresses; set up a retry mechanism.

[0268] Differences from existing technologies: Traditional methods: Manual screenshots or simple script collection may be used, which is inefficient and cannot meet large-scale and real-time requirements.

[0269] Innovation of this invention: High degree of automation: Fully adopts automated tools to achieve unattended page collection. Intelligent scheduling: Through task queues and multi-threaded concurrency, optimize resource utilization and improve efficiency. Data quality assurance: Focus on data consistency and reliability, and provide high-quality snapshot data

[0270] Application effect: Improved collection efficiency: Compared with manual collection, the efficiency is improved by dozens of times, and snapshot collection of thousands of pages can be completed in a short time. Improved data quality: Unified environment and parameter settings ensure the comparability and reliability of snapshot data. Reduced abnormality rate: The perfect exception handling and retry mechanism reduces the collection failure rate to less than 1%. The above detailed discussion of the implementation example of the page snapshot collection module demonstrates the integrity and practicality of the technical solution of the present invention. This module provides a solid data foundation for subsequent dynamic content detection, multimodal feature fusion and tampering detection, ensuring the efficient operation and detection accuracy of the entire system.

[0271] Dynamic content detection and separation module:

[0272] Step 1: Starting from t=0 seconds, multiple snapshots are collected for a marketing page, with the number of collections N=3 and the time interval Δt=2 seconds, to obtain snapshots I0, I2, and I4.

[0273] Step 2: Extract features from each snapshot using the SIFT algorithm to obtain feature sets F0, F2, and F4.

[0274] Step 3: Match and analyze the features of adjacent time points:

[0275] Match F0 and F2, and calculate the displacement and descriptor difference of the matching pair.

[0276] Match F2 and F4 and repeat the above process.

[0277] Step 4: Identify dynamic features and find that some features have changed significantly in position and descriptor, and mark them as dynamic features.

[0278] Step 5: Cluster the dynamic features and determine the dynamic content area. For example, the carousel advertisement area at the top of the page is found to be dynamic content.

[0279] Step 6: Mask the original snapshot and fill the dynamic area with black to generate the processed page I t ′ .

[0280] Technical feasibility: Algorithm maturity: The feature extraction and matching algorithms (such as SIFT, SURF, ORB) and clustering algorithms (such as DBSCAN) used are mature and reliable. Real-time: The computational complexity of feature extraction and matching is controllable, and combined with efficient algorithm implementation, it can meet the needs of real-time processing. Scalability: Parameters (such as the number of acquisitions, time interval, threshold) can be adjusted according to actual conditions to adapt to the dynamic characteristics of different pages.

[0281] Key technical difficulties and solutions:

[0282] Complexity of dynamic content: Some dynamic content may not change noticeably or may be mixed with static content.

[0283] Solution: Increase the number of acquisitions, extend the time span, and improve the detectability of feature changes; combine the structural information of the page to assist in judgment. Errors in feature matching: Feature matching may have mismatches, affecting the recognition of dynamic content.

[0284] Solution: Collect stricter matching and screening strategies, such as two-way matching verification; use more stable feature descriptors. Computing resource limitations: When processing a large number of pages, computing resources may become a bottleneck.

[0285] Solution: Optimize algorithm implementation and use parallel computing; in case of resource constraints, reduce the number of acquisitions or simplify feature extraction.

[0286] Differences from existing technologies: Traditional methods: usually cannot effectively distinguish dynamic content from tampered content, or require pre-definition of dynamic areas. Innovations of the present invention: Time series analysis: by analyzing the changes of the page on the timeline, dynamic content can be automatically identified without pre-setting. Feature level detection: Change analysis is performed at the feature level, which is more accurate and can identify subtle dynamic changes. High degree of automation: the entire process is completed automatically without human intervention, suitable for large-scale applications.

[0287] Application effect: Reduced false alarm rate: By shielding dynamic content, false alarms caused by dynamic changes are reduced, and the false alarm rate is reduced by 5%. Improved detection accuracy: Tampering detection for static content has increased the accuracy by 9%. Improved processing efficiency: Optimized algorithms and processes ensure high efficiency when processing pages in batches. The above detailed discussion of the embodiments of dynamic content detection and separation technology demonstrates the innovation and practicality of the technical solution of the present invention. This technology effectively solves the problem of interference of dynamic content on tampering detection, ensures the accuracy of subsequent detection, and provides important support for the efficient operation of the entire system.

[0288] Multimodal feature fusion module:

[0289] Step 1: Extract visual features from the page snapshot to obtain

[0290] Step 2: Parse the DOM description, extract structural features, and use GNN to generate structural feature vectors

[0291] Step 3: Extract the page text content and use the BERT model to obtain the semantic feature vector

[0292] Step 4: Normalize and resize the feature vector so that

[0293] Step 5: Use the adaptive attention mechanism to calculate the weight of each modality:

[0294] α v =0.4,α s =0.3,α e =0.3

[0295] Step 6: Fuse the feature vectors to obtain a unified feature representation:

[0296] F=0.4V+0.3S+0.3E

[0297] Technical feasibility: Algorithm maturity: The feature extraction and fusion algorithms used, such as CNN, GNN, BERT and attention mechanism, are all mature and widely used technologies. Computing resource requirements: Under reasonable hardware configuration (such as GPU acceleration), it can meet the real-time requirements. Scalability: The model can be adjusted according to actual needs, such as replacing the feature extraction model and adjusting the fusion strategy to adapt to pages of different scales and complexities.

[0298] Key technical difficulties and solutions:

[0299] Heterogeneity of features from different modalities: Visual, structural, and semantic features differ in data form and distribution.

[0300] Solution: Eliminate the impact of heterogeneity through normalization, dimension adjustment, and adaptive fusion strategies.

[0301] Weight determination for feature fusion: The importance of each modal feature may vary from page to page.

[0302] Solution: Introduce an adaptive attention mechanism so that the model automatically learns and adjusts the weights of each modality.

[0303] Difference from the prior art: Traditional methods: usually only use features of a single modality, such as only visual features or text features, which cannot fully reflect the content of the page. Innovation of the present invention: Multimodal fusion: Comprehensive use of visual, structural and semantic features to build a comprehensive page representation. Adaptive fusion: Through the attention mechanism, the model can be dynamically adjusted according to actual conditions and give higher weights to key features. Improve detection accuracy: The combination of multimodal fusion and adaptive mechanism makes the model more accurate and robust when dealing with complex tampering methods.

[0304] Application effect: Improved detection accuracy: In actual tests, the tampering detection accuracy of the multimodal feature fusion model increased by 9% compared with the single modality model. Real-time performance meets the requirements: The optimized model can meet the requirements of real-time detection in actual applications, and the processing delay is controlled within 500 milliseconds.

[0305] The above detailed description of the multimodal feature fusion technology demonstrates the innovation and feasibility of the technical solution of the present invention. This technology provides a solid foundation for subsequent tampering detection, ensuring that the system can accurately and efficiently identify unauthorized modifications to marketing pages.

[0306] Tamper Detection Module:

[0307] Step 1: Get the weighted feature representation F of the current page and the benchmark page c ′ urrent 、F b ′ aseline .

[0308] Step 2: Input the features into the twin neural network to obtain high-level features E c ′ urrent 、E b ′ aseline .

[0309] Step 3: Calculate similarity S and distance D

[0310] Step 4: Use the autoencoder to derive the current page feature F c ′ urrent Perform reconstruction and calculate the reconstruction error E.

[0311] Step 5: According to the preset threshold T s , T d , T e , determine whether there is tampering.

[0312] Step 6: If it is determined to be tampered, an alarm message is generated and the visualization module is called to display the tampered area.

[0313] Technical feasibility: Model maturity: Models such as twin neural networks and autoencoders have been widely used in the field of pattern recognition and are mature. Computing resources: Model compression and optimization technologies (such as quantization and pruning) can reduce the computing and storage requirements of the model and meet real-time requirements. Data acquisition: Using historical data and simulated tampered data, rich training and verification data sets can be constructed.

[0314] Key technical difficulties and solutions:

[0315] Data imbalance and insufficient samples:

[0316] Solution: Use data augmentation technology to generate more tampered samples; adopt transfer learning and use pre-trained models to improve performance.

[0317] The generalization ability of the model:

[0318] Solution: Introduce regularization, Dropout and other techniques to prevent overfitting; use cross-validation to evaluate the generalization ability of the model.

[0319] Fusion and matching of multimodal features:

[0320] Solution: Design special modules in the model (feature normalization processing, adaptive attention mechanism, feature weight design) to handle the differences in multimodal features and ensure the effectiveness of the fused features.

[0321] Differences from existing technologies: Traditional methods: rely on simple difference calculations, are sensitive to noise and small changes, and have a high false alarm rate. Innovations of this invention: Application of deep learning models: Using twin neural networks and autoencoders to improve the ability to detect complex tampering. Adaptive attention mechanism: Enhances the model's focus on important features and improves detection accuracy. Multi-indicator fusion judgment: Comprehensively utilizes multiple detection indicators to reduce false alarm and missed alarm rates.

[0322] Application effect: Accuracy: The accuracy of tampering detection reaches more than 98%, which is significantly improved compared with traditional methods. False alarm rate: The false alarm rate is reduced to less than 2%, effectively reducing unnecessary alarms. Processing speed: The detection time of a single page is less than 500 milliseconds, meeting the requirements of real-time monitoring.

[0323] The above detailed discussion of the tampering detection module embodiment demonstrates the innovation and practicality of the technical solution of the present invention. The module uses deep learning and adaptive attention mechanism to achieve efficient and accurate detection of marketing page tampering behavior, providing strong support for the functional improvement and performance improvement of the entire system.

[0324] Alarm and visualization module, system management and security module:

[0325] The backend uses Spring boot and Spring Cloud distributed frameworks, MySQL as the database, and Kafka as the message middleware to implement the alarm and visualization modules as well as the system management and security modules. The front end uses Vue3.0 and Echart to display the page. Alarm and visualization implementation example

[0326] Step 1: Acquisition of tampering detection results: When the tampering detection module determines that a marketing page has been tampered with, it generates detection result data containing information such as page identification, tampering type, and tampering location.

[0327] Step 2: Alarm information generation: Alarm content construction: Page ID: uniquely identifies the tampered page. Tampering time: timestamp when tampering is detected. Tampering details: describe the specific content or type of tampering, such as "unauthorized price modification found", "illegal advertising implantation detected", etc. Priority level: set the alarm level (high, medium, low) according to the severity of the tampering. Alarm template configuration: define the format of the alarm message in a templated manner, support variable replacement and customized content. Alarm method selection: supports multiple alarm methods, including email, SMS, instant messaging tools (such as WeChat, DingTalk), etc. Alarm sending: call the corresponding API or service interface to send the alarm information to the pre-set recipients or groups.

[0328] Step 3: Visual interface display: Difference highlighting: In a web browser, load the baseline snapshot and current snapshot of the detected page. Use image difference algorithms (such as pixel difference, highlight overlay) to mark the tampered area and highlight the changed part in a striking color. Provide multiple comparison modes: Side-by-side comparison: The baseline page and the current page are displayed side by side, and users can view the differences between the two at the same time. Sliding comparison: Through the slider control, users can view the two versions of the page switch in the same position. Transparent overlay: Overlay the two pages and observe the differences by adjusting the transparency. Interactive function: Zoom in / out: Support zooming in and out of the page to facilitate viewing details. Marking and annotation: Users can add marks or annotations to the tampered area to record processing opinions. Filtering and searching: Filter alarm records according to conditions such as page ID, time, and tampering type. Step 4: Alarm recording and tracking. Alarm log storage. All alarm information and processing status are recorded in the database for subsequent query and statistics. Processing process tracking: Establish a processing process for each alarm event, and record information such as the processor, processing time, and processing results. Statistics and reports: Generate alarm statistics reports regularly, including the number of alarms, processing efficiency, common tampering types, etc., to provide support for management decisions.

[0329] Implementation example of the system management and security module:

[0330] Step 1: User management and authority control, user role definition: Administrator: has all system authority, responsible for user management, system configuration, etc. Auditor: responsible for viewing alarm information and handling tampering events. Observer: can only view alarm statistics and reports, no right to operate. Identity authentication: adopt two-factor authentication (2FA) to ensure the legitimacy of user identity through passwords and dynamic verification codes. Support single sign-on (SSO) integration and docking with the enterprise's unified identity authentication system. Role-based access control (RBAC), administrators can assign specific operation permissions to different roles.

[0331] Step 2: System configuration management: Parameter setting interface: Provides a friendly web interface for administrators to adjust system parameters, such as: tamper detection threshold setting. Alarm mode and recipient configuration. Page snapshot collection frequency and time interval. Configuration file management: All configuration parameters are stored in the form of files or databases, supporting version control and change records.

[0332] Step 3: Logging and auditing: Operation log: records all users' login, logout, and operation behaviors, including time, IP address, operation content, etc. System log: records system operation status, abnormal errors, resource usage, and other information.

[0333] Through the above implementation examples, the specific implementation methods of the alarm and visualization module and the system management and security module are demonstrated in detail. The design and implementation of these modules ensure the practicality, security and reliability of the system, and provide solid support for the overall functional improvement and user experience improvement of the present invention.

[0334] Aiming at the deficiencies of existing technologies in third-party channel marketing page tampering detection and compliance monitoring, this paper proposes a comprehensive solution that combines multimodal feature fusion, intelligent dynamic content detection, adaptive attention mechanism and deep learning model. Through innovative technical means, the system can efficiently and accurately detect unauthorized modifications of marketing pages, ensure the legality and compliance of page content, and provide strong support for enterprises to maintain their brand image and reduce legal risks.

[0335] Based on time series analysis and feature extraction, the present invention extracts and matches feature points of multiple snapshots of a marketing page in a short period of time, automatically identifies dynamically changing content elements (such as advertising carousels, animations, etc.), and blocks or removes them, generating page snapshots containing only static content.

[0336] The present invention simultaneously extracts the visual features (image), structural features (DOM structure) and semantic features (text content) of the page, designs a feature fusion layer network structure and an adaptive attention mechanism, effectively fuses features of different modalities, generates a unified high-dimensional feature representation, and comprehensively reflects the page content.

[0337] The present invention introduces an adaptive attention mechanism into the tampering detection model, dynamically adjusts the attention weights of different areas of the page, and enhances the sensitivity to tampering of key areas. The method of combining Siamese Network and Autoencoder is used to perform tampering detection and anomaly recognition, improving the detection ability of subtle tampering and unknown type tampering.

[0338] The present invention designs a page snapshot collection module based on browser automation tools and intelligent scheduling strategies to achieve multiple snapshot collection of a large number of channel marketing pages, ensure the consistency, integrity and high quality of snapshot data, and provide a reliable data basis for subsequent detection.

[0339] The benefits include:

[0340] (1) Improve detection accuracy and robustness: Through intelligent dynamic content detection and separation and multimodal feature fusion technology, the system can accurately identify complex and subtle tampering behaviors, reduce false positives and false negatives, and improve detection accuracy and reliability.

[0341] (2) Enhance the system's adaptability and self-learning capabilities: By introducing an adaptive attention mechanism and deep learning model, the system can automatically learn and adapt to new violations and tampering methods, improve the ability to detect unknown types of tampering, and enhance the system's adaptability.

[0342] (3) Improve detection efficiency and meet real-time monitoring needs: Through efficient page snapshot collection and processing methods, combined with optimized algorithm design, we ensure the efficient operation of the system under large-scale page monitoring and realize real-time or near real-time monitoring of channel marketing pages.

[0343] (4) Reduce manual review costs and improve work efficiency: Through automated detection, alarms, and visual displays, we can reduce reliance on manual review, reduce labor costs, and improve the speed and efficiency of problem handling.

[0344] (5) Ensure corporate brand image and compliance: Effectively monitor the marketing pages of third-party channel vendors to prevent unauthorized modifications and the appearance of non-compliant content, maintain the corporate brand image, and reduce legal risks and the probability of user complaints.

[0345] The present invention aims to address the deficiencies of the prior art in third-party channel marketing page tampering detection and compliance monitoring, and proposes an innovative comprehensive solution. Compared with the prior art, the present application has the following significant advantages and effects:

[0346] Intelligent dynamic content detection to improve detection accuracy

[0347] Advantages: Existing technologies usually have difficulty distinguishing between dynamic content and tampered content on a page, which can easily lead to false positives and false negatives. This application uses an intelligent dynamic content detection method based on time series analysis and feature extraction to automatically identify and block dynamic content, ensuring that tampering detection focuses on static core content.

[0348] Effect: It effectively reduces the interference of dynamic content on detection results, improves the accuracy and reliability of tampering detection, and significantly reduces the false alarm rate.

[0349] Multimodal feature fusion to fully reflect page content

[0350] Advantages: Traditional methods often only use features of a single modality (such as visual features or text features) and cannot fully capture page changes. This application comprehensively extracts features of three modalities: visual, structural, and semantic, and uses graph neural networks and adaptive attention mechanisms to fuse them to generate a unified page feature representation.

[0351] Effect: It fully reflects all aspects of the page, improves the detection capability of complex tampering behaviors, and can identify subtle modifications and hidden tampering.

[0352] Deep learning model with adaptive attention mechanism to enhance detection robustness

[0353] Advantages: Existing tampering detection models have a fixed degree of attention to page regions and cannot adapt to the characteristics of different pages. This application introduces an adaptive attention mechanism, and the model can dynamically adjust the attention weight according to the importance and changes of different regions of the page.

[0354] Effect: The model's sensitivity to key tampering areas is enhanced, the detection accuracy is improved, and it also has a strong detection capability for unknown types of tampering, which improves the robustness of the system.

[0355] Efficient page snapshot collection and processing to achieve real-time monitoring

[0356] Advantages: The existing technology has bottlenecks in page collection efficiency and data quality. This application designs an efficient page snapshot collection module, using browser automation tools and intelligent scheduling strategies to achieve multiple snapshot collection of a large number of pages in a short period of time.

[0357] Effect: It meets the needs of real-time monitoring, ensures the consistency and high quality of snapshot data, and provides a reliable data basis for subsequent detection.

[0358] Reduce labor costs and improve work efficiency

[0359] Advantages: Traditional monitoring methods rely on manual review, which is inefficient and costly. This application reduces reliance on manual work through automated detection, alarming, and visual display.

[0360] Effect: It greatly reduces the cost of manual review, improves the speed and efficiency of problem handling, and enables enterprises to effectively monitor channel marketing pages at a lower cost.

[0361] The system has strong scalability and wide application range

[0362] Advantages: The system architecture of this application has good scalability, and the modular design facilitates the expansion and upgrading of functions.

[0363] Effect: It is not only applicable to advertising monitoring of value-added telecommunications services, but can also be extended to compliance monitoring of marketing pages in other industries such as finance, insurance, and medical care. It has broad application prospects and market value.

[0364] Strong innovation and advanced technology

[0365] Advantages: This application comprehensively uses advanced technologies such as time series analysis, feature extraction, graph neural network, adaptive attention mechanism, deep learning, etc., and proposes a new technical solution.

[0366] Effect: It breaks through the limitations of existing technologies, has significant innovation and advancement, and provides new ideas and methods for the development of industry technology.

[0367] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A marketing web page dynamic content recognition and tampering detection method based on multimodal fusion, characterized in that: The method comprises the following steps: S1. Obtain page snapshots of the same page reported by channel dealers at multiple times to obtain time series data; S2, extract key features and descriptors from the page snapshot, match the key features and descriptors on the time series data, and obtain a dynamic feature set; S3, determining the dynamic feature boundary according to the position of the dynamic feature in the dynamic feature set, shielding the part within the dynamic feature boundary, and generating a page snapshot containing only static content; S4, inputting the page snapshot containing only static content into the visual feature extraction network, the structural feature extraction network, and the semantic feature extraction network, fusing the visual features, structural features, and semantic features to obtain the page feature representation; S5. Input the page feature representation into the tampering detection model and visualize the tampering detection results.

2. According to claim 1, a marketing web page dynamic content recognition and tampering detection method based on multimodal fusion is characterized in that: The specific steps of S2 are: Select the feature extraction algorithm to extract key points from the page snapshot and obtain the key point set K t ={k t，i }, where k t,i Represents the i-th key point of the t-th snapshot. For each key point, calculate the corresponding feature descriptor D t，i , forming a feature set T t ={(k t，i , D t，i )}; For the feature set T at adjacent time points t and t+Δt t and T t+Δt Using the KNN method, find each feature descriptor D t，i In T t+Δt The nearest neighbor in the,ratio test method is used to obtain the matching pair set and the unmatched features; For the matching key points in the matching pair set, calculate the position change and the difference of the corresponding feature descriptor. If the position change exceeds the threshold or the difference of the feature descriptor exceeds the threshold, it is judged as a feature with significant change; The features that change significantly and the features that are not matched are taken as dynamic features to obtain a dynamic feature set.

3. According to claim 2, a marketing web page dynamic content recognition and tampering detection method based on multimodal fusion is characterized in that: The specific steps of using the ratio test method to obtain the matching pair set and unmatched features are: If each feature descriptor D t，i In T t+Δt If the distance ratio between the nearest neighbor and the next nearest neighbor in is less than the threshold, the match is reliable and the feature descriptor D t，i The corresponding key point k t，i and feature descriptor D t，i The adjacent feature descriptor D t+Δt，i The corresponding key point k t+Δt，j Match each other as a pair, and get the matching pair set M t ={{k t，i , k t+Δt，j }}, the key points that do not belong to the matching pair set are unmatched features.

4. According to claim 3, a marketing web page dynamic content recognition and tampering detection method based on multimodal fusion is characterized in that: The distance is calculated using Euclidean distance.

5. According to claim 1, a marketing web page dynamic content recognition and tampering detection method based on multimodal fusion is characterized in that: The position changes are: Δx i =x t+Δt,j -x t，j ,Δy i =y t +Δ t,j -y t,j Where Δx i is the x-axis position change of the i-th matching pair, Δy i is the change in the y-axis position of the i-th matching pair, where (x t,i ,y t，i ) and (x t+Δt,j ,y t+Δt，j ) are the coordinates of the two key points of the matching pair.

6. The method for dynamic content recognition and tampering detection of marketing web pages based on multimodal fusion according to claim 5 is characterized in that: The difference in feature descriptors is: ΔD i =||D t，i -D t+Δt，j || Among them, D t，i , D t+Δt，j are the feature descriptors corresponding to the two key points of the matching pair.

7. The method for dynamic content recognition and tampering detection of marketing web pages based on multimodal fusion according to claim 6 is characterized in that: The specific steps to determine the dynamic feature boundary are: Use K-Means clustering algorithm to cluster the key points of dynamic features; For each cluster, calculate its minimum enclosing rectangle and get the bounding box R of the dynamic area. j , and the bounding box R j Expand and get the dynamic feature boundary.

8. The method for dynamic content recognition and tampering detection of marketing web pages based on multimodal fusion according to claim 1 is characterized in that: The page features are represented as follows: F=a v ⊙V+α s ⊙S+α v ⊙E Among them, F represents the page feature representation, α v , α s , α e represents the visual, structural, and semantic attention weights, and V, S, and E represent the visual, structural, and semantic features.

9. The method for dynamic content recognition and tampering detection of marketing web pages based on multimodal fusion according to claim 8 is characterized in that: Visual features are extracted by inputting page snapshots into image convolutional neural networks, structural features are aggregated by inputting the DOM tree structure of page snapshots into graph neural networks, and semantic features are obtained by inputting the text sequence corresponding to the text content of the page snapshots into a bidirectional LSTM model.

10. The method for dynamic content recognition and tampering detection of marketing web pages based on multimodal fusion according to claim 1, characterized in that: The specific steps to obtain page snapshots of the same page reported by channel vendors at multiple times are as follows: Get screenshots of the same page reported by channel dealers at multiple times, compress and denoise the screenshots, and obtain page snapshots.

Images