Webpage monitoring system and method based on dynamic perception

Abstract

The application provides a webpage monitoring system and method based on dynamic perception, wherein the system comprises: an image acquisition module, used for acquiring historical webpage monitoring images; a feature extraction module, used for extracting multi-modal features in the historical webpage monitoring images and constructing multi-modal feature vectors; a training module, used for training a webpage dynamic perception model based on the multi-modal feature vectors and in combination with a time series analysis technique; and an output module, used for inputting real-time access webpages of a user into the webpage dynamic perception model to obtain webpage monitoring results. The webpage monitoring system and method based on dynamic perception of the application vectorize multi-modal feature vectors and train a webpage dynamic perception model in combination with a time series analysis technique; the webpage dynamic perception model dynamically perceives tampering results of real-time access webpages of a user, thereby avoiding the problem that webpage tampering causes poor user browsing experience and increasing system security.

Description

Dynamic Awareness-Based Web Monitoring System and Method Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a webpage monitoring system and method based on dynamic perception. Background Technology

[0002] Modern web pages typically use technologies like JavaScript and AJAX to dynamically load content, such as ad carousels, comment section updates, and real-time data displays. This dynamic content means the state of a webpage can change at any time; when a user visits the same page, the content they see may differ due to time, interaction, or changes in backend data. While this dynamism enhances the user experience, it also presents significant challenges for webpage monitoring. Traditional static webpage monitoring methods (such as simple image comparison or text matching) struggle to effectively handle dynamic content, easily leading to false positives (e.g., a normal ad carousel might be misjudged as image tampering) or missed detections.

[0003] More seriously, website tampering may lead to problems such as leakage of user information and discrepancies between the viewed website content and expectations. Such misleading behavior not only damages the user experience but may also have a long-term impact on the platform's reputation and trustworthiness, making the platform highly risky.

[0004] In view of this, there is an urgent need for web monitoring systems and methods based on dynamic perception, in order to at least address the above-mentioned shortcomings. Summary of the Invention

[0005] One of the objectives of this invention is to provide a webpage monitoring system and method based on dynamic perception. This system features historical webpage monitoring images to obtain multimodal visual, textual, and structural features. These multimodal features are then vectorized and combined with time series analysis techniques to train a dynamic webpage perception model. The system inputs the user's real-time webpage access into the dynamic webpage perception model to detect webpage tampering results in real time, thus avoiding the problem of poor user browsing experience caused by webpage tampering and increasing system security.

[0006] The webpage monitoring system based on dynamic perception provided in this embodiment of the invention includes:

[0007] The image acquisition module is used to acquire historical webpage monitoring images;

[0008] The feature extraction module is used to extract multimodal features from historical webpage surveillance images and construct multimodal feature vectors; the multimodal features include: visual features, text features, and structural features;

[0009] The training module is used to train a dynamic webpage perception model based on multimodal feature vectors and time series analysis techniques.

[0010] The output module is used to input the user's real-time access to the webpage into the webpage dynamic perception model to obtain the webpage monitoring results.

[0011] Preferably, the image acquisition module acquires historical webpage monitoring images, including:

[0012] Obtain the target records; the target records include: historical user-uploaded complaint records and proactive upload records by website administrators;

[0013] Analyze the target record to obtain the snapshot time and the first screen snapshot;

[0014] The first screen snapshots belonging to the same webpage tag are summarized and sorted according to the chronological order of the snapshot times to obtain the first screen snapshot sequence;

[0015] Obtain the submission node of the annotator;

[0016] The first screen snapshot sequence is packaged and delivered to the annotation personnel's delivery node, and the annotation personnel are assisted in annotating the first screen snapshot in the first screen snapshot sequence;

[0017] Once the first screenshot sequence under all webpage tabs has been labeled, historical webpage monitoring images are obtained.

[0018] Preferably, the image acquisition module assists the annotator in annotating the first screen snapshot in the first screen snapshot sequence, including:

[0019] Determine the first screen snapshot that the annotator is currently viewing, and use it as the second screen snapshot;

[0020] Identify a third screen snapshot in the first screen snapshot sequence that is similar to the second screen snapshot, excluding the second screen snapshot.

[0021] Based on the snapshot time, and according to the second and third screen snapshots, extract the DOM structure sequence;

[0022] Based on the DOM structure sequence and the target webpage DOM structure, obtain the DOM tree structure difference sequence;

[0023] Based on the DOM tree structure difference sequence, obtain knowledge of webpage tampering annotations;

[0024] Visualize the knowledge of webpage tampering annotation to the annotation personnel.

[0025] Preferably, the image acquisition module obtains webpage tampering annotation knowledge based on the DOM tree structure difference sequence, including:

[0026] Parse the differences in the DOM tree structure to obtain the difference tree nodes and difference branches;

[0027] Obtain the knowledge graph of webpage tampering;

[0028] Based on preset knowledge delineation conditions and differences in DOM tree structure, obtain the delineated local graph in the webpage tampering knowledge graph;

[0029] The criteria for defining knowledge circles include:

[0030] The cost of delineating the boundary is less than or equal to the preset delineation cost threshold;

[0031] Each difference tree node in the circled graph nodes can find a graph node with a node correlation degree greater than or equal to the preset node correlation degree threshold.

[0032] Among the circled graph branches, there are graph branches whose branch correlation degree with the branch corresponding to the second screen snapshot is greater than or equal to the preset branch correlation degree threshold.

[0033] In the circled graph branches, apart from the graph branches associated with the difference branch corresponding to the second screen snapshot, the source DOM tree structure differences and the target DOM tree structure differences corresponding to the second screen snapshot of the remaining graph branches conform to the standard sequence arrangement.

[0034] Preferably, the image acquisition module also performs the following operations:

[0035] Based on the differences in the DOM tree structure corresponding to the second screen snapshot, determine the first sub-graph in the local graph;

[0036] Based on the differences in the DOM tree structure corresponding to the third screen snapshot, determine the second sub-graph in the local graph;

[0037] The first knowledge presentation screen corresponding to the webpage tampering annotation knowledge of the first sub-graph is determined;

[0038] Based on the preset graph feature template, extract graph features according to the first sub-graph;

[0039] Based on the pre-defined knowledge organization logic of the graph and the characteristics of the graph, multiple knowledge organization routes for the graph are determined.

[0040] The regularity perception of the knowledge graph organization route is calculated based on the order in which the route first passes through the second subgraph and the standard order in which the route passes through the second subgraph.

[0041] The target coverage rate is calculated based on the knowledge routing route through the second sub-graph and the preset knowledge weights of the second sub-graph.

[0042] Summing and calculating the pattern perception and target coverage corresponding to the knowledge map sorting route, and obtaining the route selection value;

[0043] Based on the knowledge graph with the highest route selection value, the second knowledge presentation screen corresponding to the second sub-graph that the route passes through in sequence is used to construct a knowledge presentation screen sequence;

[0044] The first knowledge presentation screen is inserted at the beginning of the knowledge presentation screen sequence, and the knowledge presentation screens are displayed cyclically according to the preset highlighting rules.

[0045] Preferably, the image acquisition module also performs the following operations:

[0046] Obtain the questioning images of the marked personnel in the knowledge presentation sequence;

[0047] The third sub-map corresponding to the questionable image is shown to the annotator to obtain the annotator's first line of sight trajectory in the third sub-map.

[0048] Based on the first line-of-sight trajectory, determine the second line-of-sight trajectory between the target entities in the third sub-map that are viewed alternately, and take the second line-of-sight trajectory most recently generated at the current moment as the third line-of-sight trajectory.

[0049] Determine the differences in map features of the third sub-map traversed by the second line of sight trajectory (excluding the third line of sight trajectory);

[0050] Extract the semantic description of the differences in the graph features and input it into the knowledge annotation model corresponding to the third subgraph. Then, mark the corresponding annotation knowledge on the third subgraph through which the third line of sight trajectory passes.

[0051] The webpage monitoring method based on dynamic perception provided in this embodiment of the invention includes:

[0052] Step 1: Obtain historical webpage monitoring images;

[0053] Step 2: Extract multimodal features from historical webpage surveillance images and construct multimodal feature vectors; multimodal features include: visual features, text features, and structural features;

[0054] Step 3: Train a webpage dynamic perception model based on multimodal feature vectors and time series analysis techniques;

[0055] Step 4: Input the user's real-time accessed webpages into the webpage dynamic perception model to obtain webpage monitoring results.

[0056] Preferably, step 1: Obtain historical webpage monitoring images, including:

[0057] Obtain the target records; the target records include: historical user-uploaded complaint records and proactive upload records by website administrators;

[0058] Analyze the target record to obtain the snapshot time and the first screen snapshot;

[0059] The first screen snapshots belonging to the same webpage tag are summarized and sorted according to the chronological order of the snapshot times to obtain the first screen snapshot sequence;

[0060] Obtain the submission node of the annotator;

[0061] The first screen snapshot sequence is packaged and delivered to the annotation personnel's delivery node, and the annotation personnel are assisted in annotating the first screen snapshot in the first screen snapshot sequence;

[0062] Once the first screenshot sequence under all webpage tabs has been labeled, historical webpage monitoring images are obtained.

[0063] Preferably, the method of assisting annotators in annotating the first screen snapshot in the first screen snapshot sequence includes:

[0064] Determine the first screen snapshot that the annotator is currently viewing, and use it as the second screen snapshot;

[0065] Identify a third screen snapshot in the first screen snapshot sequence that is similar to the second screen snapshot, excluding the second screen snapshot.

[0066] Based on the snapshot time, and according to the second and third screen snapshots, extract the DOM structure sequence;

[0067] Based on the DOM structure sequence and the target webpage DOM structure, obtain the DOM tree structure difference sequence;

[0068] Based on the DOM tree structure difference sequence, obtain knowledge of webpage tampering annotations;

[0069] Visualize the knowledge of webpage tampering annotation to the annotation personnel.

[0070] Preferably, based on the DOM tree structure difference sequence, knowledge of webpage tampering annotation is obtained, including:

[0071] Parse the differences in the DOM tree structure to obtain the difference tree nodes and difference branches;

[0072] Obtain the knowledge graph of webpage tampering;

[0073] Based on preset knowledge delineation conditions and differences in DOM tree structure, obtain the delineated local graph in the webpage tampering knowledge graph;

[0074] The criteria for defining knowledge circles include:

[0075] The cost of delineating the boundary is less than or equal to the preset delineation cost threshold;

[0076] Each difference tree node in the circled graph nodes can find a graph node with a node correlation degree greater than or equal to the preset node correlation degree threshold.

[0077] Among the circled graph branches, there are graph branches whose branch correlation degree with the branch corresponding to the second screen snapshot is greater than or equal to the preset branch correlation degree threshold.

[0078] In the circled graph branches, apart from the graph branches associated with the difference branch corresponding to the second screen snapshot, the source DOM tree structure differences and the target DOM tree structure differences corresponding to the second screen snapshot of the remaining graph branches conform to the standard sequence arrangement.

[0079] Based on local maps, obtain knowledge about webpage tampering annotations.

[0080] Preferably, obtaining webpage tampering annotation knowledge based on the DOM tree structure difference sequence also includes:

[0081] Based on the differences in the DOM tree structure corresponding to the second screen snapshot, determine the first sub-graph in the local graph;

[0082] Based on the differences in the DOM tree structure corresponding to the third screen snapshot, determine the second sub-graph in the local graph;

[0083] The first knowledge presentation screen corresponding to the webpage tampering annotation knowledge of the first sub-graph is determined;

[0084] Based on the preset graph feature template, extract graph features according to the first sub-graph;

[0085] Based on the pre-defined knowledge organization logic of the graph and the characteristics of the graph, multiple knowledge organization routes for the graph are determined.

[0086] The regularity perception of the knowledge graph organization route is calculated based on the order in which the route first passes through the second subgraph and the standard order in which the route passes through the second subgraph.

[0087] The target coverage rate is calculated based on the knowledge routing route through the second sub-graph and the preset knowledge weights of the second sub-graph.

[0088] Summing and calculating the pattern perception and target coverage corresponding to the knowledge map sorting route, and obtaining the route selection value;

[0089] Based on the knowledge graph with the highest route selection value, the second knowledge presentation screen corresponding to the second sub-graph that the route passes through in sequence is used to construct a knowledge presentation screen sequence;

[0090] The first knowledge presentation screen is inserted at the beginning of the knowledge presentation screen sequence, and the knowledge presentation screens are displayed cyclically according to the preset highlighting rules.

[0091] The beneficial effects of this invention are as follows:

[0092] This invention features historical webpage monitoring images to obtain multimodal visual, textual, and structural features. These multimodal features are then vectorized and combined with time series analysis techniques to train a dynamic webpage perception model. The model inputs the user's real-time webpage access into the dynamic webpage perception model to detect webpage tampering results in real time, thus avoiding the problem of webpages being tampered with, resulting in a poor user browsing experience, and also increasing system security.

[0093] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0094] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0095] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0096] Figure 1 is a schematic diagram of a webpage monitoring system based on dynamic perception in an embodiment of the present invention;

[0097] Figure 2 is a schematic diagram of the webpage monitoring method based on dynamic perception in an embodiment of the present invention. Detailed Implementation

[0098] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0099] This invention provides a webpage monitoring system based on dynamic perception, as shown in Figure 1, comprising:

[0100] Image acquisition module 1 is used to acquire historical webpage monitoring images; historical webpage monitoring images are: webpage images that have been acquired in the past by the webpage monitoring platform and have been annotated, with the webpage tampered with and the type of tampering annotated in the historical webpage monitoring images;

[0101] Feature extraction module 2 is used to extract multimodal features from historical webpage monitoring images and construct multimodal feature vectors. Multimodal features include visual features, text features, and structural features. Visual features are visual information extracted from historical webpage monitoring images, including color distribution, layout structure, and the position and shape of key elements (such as buttons and images). Text features are the text content extracted from historical webpage monitoring images. Structural features are information extracted from the DOM (Document Object Model) structure of the corresponding webpage from the historical webpage monitoring images, such as the hierarchical relationship of HTML tags, node attributes, and dynamically loaded content. The multimodal feature vector is a vector representation of the visual, text, and structural features after processing, and can be used as input to a machine learning model.

[0102] Training module 3 is used to train a dynamic webpage perception model based on multimodal feature vectors and time series analysis techniques. When training the dynamic webpage perception model based on multimodal feature vectors and time series analysis techniques, the multimodal feature vectors are input into the YOLOv8 model and the LSTM network is used to learn the relationship between the multimodal feature vectors and their corresponding labels and the tampering types to obtain the dynamic webpage perception model.

[0103] Output module 4 is used to input the user's real-time access to the webpage into the webpage dynamic perception model to obtain the webpage monitoring results.

[0104] The working principle and beneficial effects of the above technical solution are as follows:

[0105] This invention features historical webpage monitoring images to obtain multimodal visual, textual, and structural features. These multimodal features are then vectorized and combined with time series analysis techniques to train a dynamic webpage perception model. The model inputs the user's real-time webpage access into the dynamic webpage perception model to detect webpage tampering results in real time, thus avoiding the problem of webpages being tampered with, resulting in a poor user browsing experience, and also increasing system security.

[0106] In one embodiment, the image acquisition module acquires historical webpage monitoring images, including:

[0107] Obtain target records; target records include: historical user-uploaded complaint records and proactive upload records by web administrators on the web platform; web platform refers to the platform that needs web monitoring; historical users refer to users who have historically generated complaints related to webpage tampering on the web platform;

[0108] Parse the target record to obtain the snapshot time and the first screen snapshot; the first screen snapshot is a screenshot of the webpage in the target record, and the snapshot time is the time of the webpage screenshot.

[0109] The first screen snapshots belonging to the same webpage tag are summarized and sorted according to the chronological order of the snapshot times to obtain the first screen snapshot sequence; webpage tags are: marks used to classify or identify webpages, distinguishing different webpages or different versions of the same webpage;

[0110] Obtain the annotation personnel's submission node; the annotation personnel's submission node is the annotation personnel's communication node;

[0111] Package and deliver the first screen snapshot sequence to the annotation personnel's delivery node, and assist the annotation personnel in annotating the first screen snapshot in the first screen snapshot sequence; when assisting, display relevant information to the annotation personnel who need it to help them mark the tampered content or abnormal areas in the webpage;

[0112] Once the first screenshot sequence under all webpage tabs has been labeled, historical webpage monitoring images are obtained.

[0113] The working principle and beneficial effects of the above technical solution are as follows:

[0114] The target records include historical user complaints about the webpage and images of the webpage that have been tampered with (first screen snapshots) uploaded by webpage administrators, along with the time of tampering (snapshot time). Snapshots belonging to the same webpage are categorized and displayed according to time to obtain a sequence of first screen snapshots, which facilitates subsequent annotation by personnel to analyze the dynamic changes in webpage content.

[0115] The annotation process connects to the annotation personnel's delivery nodes, packages and delivers the first screen snapshot sequence, and assists the annotation personnel in annotation. Once the first screen snapshot sequence under all webpage tags has been annotated, historical webpage monitoring images (training data) are obtained, making the annotation process more intelligent and reasonable.

[0116] In one embodiment, the image acquisition module assists the annotator in annotating the first screen snapshot in the first screen snapshot sequence, including:

[0117] The first screen snapshot that the annotator is currently viewing is identified and designated as the second screen snapshot; the second screen snapshot is a specific snapshot selected from the sequence of first screen snapshots.

[0118] Identify a third screen snapshot in the first screen snapshot sequence that is similar to the second screen snapshot, excluding the second screen snapshot; similar to the second screen snapshot means that the image feature similarity with the second screen snapshot is greater than or equal to a preset similarity threshold (e.g., 70%).

[0119] Based on the snapshot time, and according to the second and third screen snapshots, the DOM structure sequence is extracted; during extraction, the DOM document object model structure (DOM structure) of the webpage corresponding to the screen snapshot is sorted in order from earliest to latest according to the snapshot time.

[0120] Based on the DOM structure sequence and the target webpage DOM structure, obtain the DOM tree structure difference sequence; the DOM tree structure difference refers to the differences between the DOM structure and the target webpage DOM structure, including: newly added, deleted, or modified nodes and their attributes; the target webpage DOM structure is the preset DOM structure of the webpage corresponding to the second and third screen snapshots;

[0121] Based on the DOM tree structure difference sequence, obtain webpage tampering annotation knowledge; webpage tampering annotation knowledge: specific information about webpage tampering obtained from analyzing DOM tree structure differences, including tampering type, location and content;

[0122] Visualize the knowledge of webpage tampering annotation to the annotation personnel.

[0123] The working principle and beneficial effects of the above technical solution are as follows:

[0124] Since webpage tampering is dynamic, annotators cannot understand the patterns of tampering based on a single second-screen snapshot. Furthermore, due to insufficient webpage-related knowledge, annotators cannot effectively identify tampered content and mark the type of tampering. Therefore, assistance is needed for annotators.

[0125] In specific assistance, the system identifies the second screen snapshot that the annotator is currently viewing. Simultaneously, it searches for a third screen snapshot similar to the first screen snapshot sequence. The DOM structure of the screen snapshots is extracted in chronological order from earliest to latest to obtain a DOM structure sequence. The DOM structure sequence is then compared with the DOM structure of the target webpage to obtain a DOM tree structure difference sequence. Based on the DOM tree structure difference sequence, the system retrieves webpage tampering annotation knowledge for the annotator to view, making the annotation process more intelligent.

[0126] In one embodiment, the image acquisition module obtains webpage tampering annotation knowledge based on the DOM tree structure difference sequence, including:

[0127] Parse the differences in the DOM tree structure to obtain the difference tree nodes and difference branches; the difference tree nodes are the DOM tree nodes that differ between the DOM structure and the target webpage's DOM structure, including: newly added nodes (e.g., a certain...).<scr i pt> Tag insertion), node deletion (e.g., a specific tag insertion),<d i v> (Label removed) and modified nodes (e.g., a certain) (The text content of the tag has been changed); the difference branches are: the subtrees or node paths that have changed in the DOM tree structure differences, used to describe the scope and impact of the tampering behavior, such as: a certain<d i v> The tag and its child nodes were deleted entirely.

[0128] Obtain the webpage tampering knowledge graph; the webpage tampering knowledge graph is a tampering knowledge base represented by a graph structure, where graph nodes represent tampering-related entities (e.g., DOM nodes, tampering types), and edges represent relationships between entities (e.g., code injection leading to tampering).<scr i pt> (Label change);

[0129] Based on preset knowledge delineation conditions and differences in DOM tree structure, the local graphs delineated in the webpage tampering knowledge graph are obtained; the knowledge delineation conditions are: selecting constraints that adapt to the interpretation of differences in DOM tree structure.

[0130] The criteria for defining knowledge circles include:

[0131] The delineation cost is less than or equal to the preset delineation cost threshold; the delineation cost is the access cost (computing resources and time cost) of all delineated graph nodes and graph branches; the preset delineation cost threshold is set manually in advance;

[0132] In the circled graph nodes, each difference tree node can find a graph node with a node correlation degree greater than or equal to a preset node correlation degree threshold. Node correlation degree refers to the relevance between graph nodes and difference tree nodes in the webpage tampering knowledge graph, such as the graph node of "code injection" and the graph node of "...".<scr i pt> The correlation degree of the "tag" difference tree node is 0.9. The correlation degree between the difference tree node and the DOM tree node is predefined manually. The preset node correlation degree threshold is set manually, for example: 0.85.

[0133] Among the circled graph branches, there are graph branches whose branch correlation with the difference branch corresponding to the second screen snapshot is greater than or equal to the preset branch correlation threshold; the branch correlation is: the semantic similarity between the semantics of the graph branch and the semantics of the difference branch, for example: 0.7; the preset branch correlation threshold is set manually, for example: 0.8;

[0134] In the circled graph branches, except for the graph branch associated with the difference branch corresponding to the second screen snapshot, the source DOM tree structure differences and the target DOM tree structure differences of the difference branches associated with the remaining graph branches conform to the standard sequence arrangement. The source DOM tree structure differences are: the DOM tree structure differences from which the difference branches associated with the remaining graph branches are parsed; the target DOM tree structure differences are the DOM tree structure differences corresponding to the second screen snapshot; the standard sequence arrangement is: the source DOM tree structure differences and the target DOM tree structure differences, after being sorted according to the snapshot time of the corresponding extracted second or third screen snapshot, constitute a local subsequence of the DOM tree structure difference sequence;

[0135] Based on local maps, obtain knowledge about webpage tampering annotations.

[0136] The working principle and beneficial effects of the above technical solution are as follows:

[0137] When acquiring knowledge about webpage tampering annotations, displaying all relevant knowledge to the user is inefficient. Therefore, we introduce knowledge delineation conditions and delineate local graphs based on the DOM tree structure difference sequence. The corresponding graph knowledge of these local graphs is then used as the webpage tampering annotation knowledge. Specifically, the knowledge delineation conditions are as follows:

[0138] Condition 1: The cost of mapping is less than the cost threshold for mapping; Since accessing web pages to tamper with the knowledge graph requires computing resources and time, excessively high mapping costs will lead to untimely assistance. Setting this condition improves the timeliness of assistance presentation.

[0139] Condition 2: For each difference tree node in the circled graph nodes, there can be a graph node with a node correlation degree greater than or equal to the preset node correlation degree threshold. The difference tree node is the DOM tree node that is different between each DOM structure and the target webpage DOM structure in the DOM tree structure difference sequence. Specifically, it is all the DOM tree nodes that caused the webpage tampering phenomenon in the second and third screen snapshots. It is the basic unit for analyzing tampering behavior and helps to locate the specific location of tampering. Therefore, it is necessary to obtain the graph node corresponding to each difference tree node to improve the comprehensiveness and continuity of webpage tampering annotation knowledge.

[0140] Condition 3: Among the circled graph branches, there are graph branches whose branch correlation with the difference branch corresponding to the second screen snapshot is greater than or equal to the preset branch correlation threshold; since graph branches contain more detailed information, and the presentation priority of detailed information is different, for example: the graph branch that the annotator is currently annotating the difference branch corresponding to the second screen snapshot must be presented with the highest priority.

[0141] Condition 4: In the circled graph branches, except for the graph branches associated with the difference branch corresponding to the second screen snapshot, the source DOM tree structure differences and the target DOM tree structure differences corresponding to the second screen snapshot of the remaining graph branches conform to the standard sequence arrangement. Since it is dynamically perceived tampering information, the graph branches associated with the difference branches extracted from the third screen snapshot that are closer to the second screen snapshot in time are circled first. Therefore, the source DOM tree structure differences and the target DOM tree structure differences are constrained to be local subsequences of the DOM tree structure difference sequence after being sorted according to the snapshot time of the corresponding extracted second or third screen snapshot. Under the condition of limited circled resources, more web page tampering annotation knowledge with higher relevance is presented, and the circled resource utilization rate is improved.

[0142] In one embodiment, the image acquisition module also performs the following operations:

[0143] Based on the differences in the DOM tree structure corresponding to the second screen snapshot, the first sub-graph in the local graph is determined; the first sub-graph is the part of the local graph corresponding to the relevant knowledge that caused the differences in the DOM tree structure corresponding to the second screen snapshot.

[0144] Based on the differences in the DOM tree structure corresponding to the third screen snapshot, the second subgraph in the local graph is determined; the second subgraph is the part of the local graph corresponding to the relevant knowledge that caused the differences in the DOM tree structure corresponding to the third screen snapshot.

[0145] Determine the first knowledge presentation screen corresponding to the webpage tampering annotation knowledge of the first sub-graph; the first knowledge presentation screen is: the presentation screen area when the knowledge corresponding to the first sub-graph is visualized.

[0146] Based on the preset graph feature template, graph features are extracted according to the first sub-graph; the graph features are: edge graph entities and edge graph relationships of the first sub-graph.

[0147] Based on the pre-set knowledge organization logic of the graph and the characteristics of the graph, multiple knowledge organization routes are determined. The pre-set knowledge organization logic of the graph is set by personnel with experience in teaching knowledge tampering on web pages, including: which part of the knowledge to learn first and which part of the knowledge to learn next.

[0148] The pattern perception of the knowledge graph organization route is calculated based on the order in which the route first passes through the second subgraph and the standard passing order of the second subgraph. The standard passing order of the second subgraph is determined by the positional interval between the DOM tree structure difference of the second subgraph and the DOM tree structure difference of the second subgraph in the sequence of DOM tree structure differences. For example, if the interval is 1, the standard passing order is 1. The pattern perception is the sum of the absolute values ​​of the passing order of each second subgraph minus the absolute value of its corresponding standard passing order. For example, 16.

[0149] The target coverage rate is calculated based on the knowledge routing route through the second subgraph, the graph coverage rate through the second subgraph, and the preset knowledge weight of the second subgraph. The preset knowledge weight of the second subgraph is the reciprocal of the standard traversal order value of the second subgraph.

[0150] Summing and calculating the pattern perception and target coverage corresponding to the knowledge map sorting route, and obtaining the route selection value;

[0151] Based on the knowledge graph with the highest route selection value, the second knowledge presentation screen corresponding to the second sub-graph that the route passes through in sequence is used to construct a knowledge presentation screen sequence;

[0152] The first knowledge presentation screen is inserted at the beginning of the knowledge presentation screen sequence, and the knowledge presentation screens are displayed cyclically according to the knowledge presentation screen sequence based on preset highlighting rules. The cyclic display of knowledge presentation screens according to the knowledge presentation screen sequence based on preset highlighting rules is as follows: when a certain knowledge presentation screen in the knowledge presentation screen sequence is looped to, it is highlighted (for example, highlighting the background color of the knowledge).

[0153] The working principle and beneficial effects of the above technical solution are as follows:

[0154] The presentation of webpage tampering annotation knowledge to annotators involves a viewing logic. Directly displaying all the webpage tampering annotation knowledge to annotators yields poor learning results. Therefore, the first sub-graph corresponding to the second screen snapshot to be viewed first is determined. The knowledge in the first sub-graph has the highest relevance to the second screen snapshot and should be viewed first, thus determining the first knowledge presentation screen. Next, the graph features of the first sub-graph in the local graph (edge ​​graph entities and edge graph relationships) are extracted. Based on the graph knowledge organization logic set by experienced webpage tampering knowledge teachers, multiple graph knowledge organization routes are determined. The standard traversal order of the second sub-graph is an ideal access order that can achieve dynamic comparison of associated webpage tampering knowledge, but the continuity of knowledge understanding also needs to be considered. Therefore, based on the traversal order of the first visit to the second sub-graph and the standard traversal order of the graph knowledge organization route, calculations are performed. The pattern perception of the knowledge graph organization route is considered. The higher the pattern perception, the higher the priority of the corresponding knowledge graph organization route being selected. In addition, the graph coverage rate of the knowledge graph organization route passing through the second sub-graph needs to be considered. Different second sub-graphs have different knowledge weights. The graph coverage rate is assigned a knowledge weight to obtain the target coverage rate. The pattern perception rate and the target coverage rate corresponding to the knowledge graph organization route are summed to obtain the route selection value. According to the order of the second sub-graphs passed through by the knowledge graph organization route with the highest route selection value, the second knowledge presentation screens presented in the corresponding second sub-graphs are sorted to form a knowledge presentation screen sequence. The first knowledge presentation screen is inserted at the beginning of the sequence and displayed cyclically. The knowledge screens displayed in the cycle are highlighted, so that the annotators can view the webpage tampering knowledge in a reasonable logical order, improve the understanding effect, and further improve the annotation quality of auxiliary annotation.

[0155] In one embodiment, displaying knowledge presentation screens cyclically according to a sequence of knowledge presentation screens further includes:

[0156] Acquire the questioning images of the annotators in the sequence of knowledge presentation images; the questioning images are the knowledge presentation images that elicit a questioning expression from the annotators when they view them.

[0157] The third sub-map corresponding to the questionable image is shown to the annotator to obtain the annotator's first line of sight trajectory in the third sub-map.

[0158] Based on the first line-of-sight trajectory, the second line-of-sight trajectory between the target entities of the third sub-map that are viewed alternately is determined, and the second line-of-sight trajectory most recently generated at the current time is taken as the third line-of-sight trajectory; viewing the target entities of the third sub-map alternately means that there is more than one line-of-sight trajectory (second line-of-sight trajectory) connecting the target entities of the third sub-map that are viewed.

[0159] Determine the differences in map features of the third sub-map traversed by the second line of sight trajectory (excluding the third line of sight trajectory); the differences in map features are: the difference in third sub-map entities and the difference in third sub-branches.

[0160] Extract the semantic description of the differences in the graph features and input it into the knowledge annotation model corresponding to the third subgraph. Then, annotate the corresponding knowledge onto the third subgraph traversed by the third line-of-sight trajectory. The semantic description of the differences describes the graph differences between the entities in the third subgraph where the differences begin and end. The knowledge annotation model corresponding to the third subgraph is an AI model that automatically annotates the third subgraph, trained based on the principle explanation of the third subgraph's construction by the builder.

[0161] The working principle and beneficial effects of the above technical solution are as follows:

[0162] When annotators view the knowledge presentation screens sequentially according to their understanding of the logical order, they may encounter content items that they do not understand. Therefore, based on facial expression analysis technology, we can identify the screens that annotators find confusing and extract the corresponding third sub-graphs. The third sub-graphs effectively organize the knowledge outlines corresponding to the screens that annotators find confusing, making it easier to quickly locate the knowledge that annotators do not understand in the future.

[0163] Next, the annotator will examine the third sub-map. Based on the line-of-sight extraction technology, the first line-of-sight trajectory of the annotator in the third sub-map will be obtained, and it will be determined that there is more than one line-of-sight trajectory connecting the target entities in the third sub-map, i.e., the second line-of-sight trajectory. Alternating viewing indicates that the annotator has viewed the third sub-map area corresponding to the second line-of-sight trajectory more than once, which indicates that the annotator has repeated viewing behavior and needs to be annotated.

[0164] The differences in graph features between the third line-of-sight trajectory and the remaining second line-of-sight trajectories are determined. These differences characterize the differences in graph entities and graph branches of the repeatedly viewed third sub-graph, including the knowledge outline information of the differences that are repeatedly compared. The more likely the difference is to be annotated, the more likely it is to be annotated. Therefore, a knowledge annotation model corresponding to the third sub-graph is introduced. The semantic description of the difference is input into the model and the output annotation is marked on the third sub-graph traversed by the latest viewed third line-of-sight trajectory in real time. This realizes the automatic annotation of graph knowledge that the annotator is confused about, and further assists the annotator in understanding.

[0165] This invention provides a webpage monitoring method based on dynamic perception, as shown in Figure 2, including:

[0166] Step 1: Obtain historical webpage monitoring images;

[0167] Step 2: Extract multimodal features from historical webpage surveillance images and construct multimodal feature vectors; multimodal features include: visual features, text features, and structural features;

[0168] Step 3: Train a webpage dynamic perception model based on multimodal feature vectors and time series analysis techniques;

[0169] Step 4: Input the user's real-time accessed webpages into the webpage dynamic perception model to obtain webpage monitoring results.

[0170] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A webpage monitoring system based on dynamic perception, characterized in that, include: The image acquisition module is used to acquire historical webpage monitoring images; The feature extraction module is used to extract multimodal features from historical webpage monitoring images and construct multimodal feature vectors. Multimodal features include visual features, text features, and structural features; the training module is used to train a dynamic webpage perception model based on multimodal feature vectors and time series analysis techniques; the output module is used to input the user's real-time access to the dynamic webpage perception model to obtain webpage monitoring results; among them, the image acquisition module acquires historical webpage monitoring images, including: acquiring target records; target records include: historical user-uploaded complaint records and active upload records by webpage administrators; parsing target records to obtain snapshot timestamps and first screen snapshots; summarizing first screen snapshots belonging to the same webpage tag and sorting them according to the chronological order of snapshot times to obtain the first screen snapshot sequence; acquiring annotation personnel submission nodes; packaging and submitting the first screen snapshot sequence to the annotation personnel submission nodes, and assisting in... The annotation team annotates the first screen snapshot in the first screen snapshot sequence. Once the first screen snapshot sequence under all webpage tags has been annotated, historical webpage monitoring images are obtained. The image acquisition module assists the annotation team in annotating the first screen snapshot in the first screen snapshot sequence, including: determining the first screen snapshot currently being viewed by the annotation team and designating it as the second screen snapshot; determining a third screen snapshot in the first screen snapshot sequence that is similar to the second screen snapshot, excluding the second; extracting the DOM structure sequence based on the snapshot time and according to the second and third screen snapshots; obtaining a DOM tree structure difference sequence based on the DOM structure sequence and the target webpage DOM structure; obtaining webpage tampering annotation knowledge based on the DOM tree structure difference sequence; and visually displaying the webpage tampering annotation knowledge to the annotation team.

2. The webpage monitoring system based on dynamic perception as described in claim 1, characterized in that, The image acquisition module acquires webpage tampering annotation knowledge based on the DOM tree structure difference sequence, including: parsing DOM tree structure differences to obtain difference tree nodes and difference branches; acquiring a webpage tampering knowledge graph; and acquiring the circled local graphs in the webpage tampering knowledge graph based on preset knowledge circled conditions and DOM tree structure differences. The knowledge circled conditions include: circled cost less than or equal to a preset circled cost threshold; each difference tree node in the circled graph nodes can find a graph node with a node correlation degree greater than or equal to a preset node correlation degree threshold; the circled graph branches contain graph branches with a branch correlation degree greater than or equal to a preset branch correlation degree threshold for the difference branch corresponding to the second screen snapshot; and, except for the graph branches associated with the difference branch corresponding to the second screen snapshot, the source DOM tree structure differences and the target DOM tree structure differences corresponding to the second screen snapshot in the circled graph branches conform to a standard sequence arrangement. Based on the local graphs, webpage tampering annotation knowledge is acquired.

3. The webpage monitoring system based on dynamic perception as described in claim 2, characterized in that, The image acquisition module also performs the following operations: determining the first sub-map in the local map based on the DOM tree structure differences corresponding to the second screen snapshot; determining the second sub-map in the local map based on the DOM tree structure differences corresponding to the third screen snapshot; and determining the first knowledge presentation screen corresponding to the webpage tampering annotation knowledge of the first sub-map. Based on the preset graph feature template, extract graph features according to the first sub-graph; Based on the pre-defined knowledge organization logic of the graph and the characteristics of the graph, multiple knowledge organization routes for the graph are determined. The regularity perception of the knowledge graph organization route is calculated based on the order in which the route first passes through the second subgraph and the standard order in which the route passes through the second subgraph. The target coverage rate is calculated based on the knowledge routing route through the second sub-graph and the preset knowledge weights of the second sub-graph. Summing and calculating the pattern perception and target coverage corresponding to the knowledge map sorting route, and obtaining the route selection value; Based on the knowledge graph with the highest route selection value, the second knowledge presentation screen corresponding to the second sub-graph that the route passes through in sequence is used to construct a knowledge presentation screen sequence; The first knowledge presentation screen is inserted at the beginning of the knowledge presentation screen sequence, and the knowledge presentation screens are displayed cyclically according to the preset highlighting rules.

4. A webpage monitoring method based on dynamic perception, characterized in that, include: Step 1: Obtain historical webpage monitoring images; Step 2: Extract multimodal features from historical webpage monitoring images and construct multimodal feature vectors; Multimodal features include: visual features, text features, and structural features; Step 3: Based on multimodal feature vectors and combined with time series analysis technology, train the webpage dynamic perception model; Step 4: Input the user's real-time access to the webpage into the webpage dynamic perception model to obtain webpage monitoring results; Among them, Step 1: Obtain historical webpage monitoring images, including: obtaining target records; target records include: historical user-uploaded complaint records and active upload records of webpage administrators on the webpage platform; parse the target records to obtain the snapshot time and the first screen snapshot; summarize the first screen snapshots belonging to the same webpage tag and sort them according to the chronological order of the snapshot time to obtain the first screen snapshot sequence; obtain the annotation personnel delivery node; package the first screen snapshot sequence and deliver it to the annotation personnel delivery node, and assist the annotation personnel in marking the first screen snapshot sequence. Annotators label the first screen snapshot in the first screen snapshot sequence; after all the first screen snapshot sequences under all webpage tags have been labeled, historical webpage monitoring images are obtained; the annotation of the first screen snapshot in the first screen snapshot sequence is assisted by: determining the first screen snapshot currently being viewed by the annotator and using it as the second screen snapshot; determining the third screen snapshot in the first screen snapshot sequence that is similar to the second screen snapshot, excluding the second screen snapshot; extracting the DOM structure sequence based on the snapshot time, the second screen snapshot, and the third screen snapshot; obtaining the DOM tree structure difference sequence based on the DOM structure sequence and the target webpage DOM structure; obtaining webpage tampering annotation knowledge based on the DOM tree structure difference sequence; and visually displaying the webpage tampering annotation knowledge to the annotator.

5. The webpage monitoring method based on dynamic perception as described in claim 4, characterized in that, Based on the DOM tree structure difference sequence, acquire webpage tampering annotation knowledge, including: parsing DOM tree structure differences to obtain difference tree nodes and difference branches; acquiring a webpage tampering knowledge graph; based on preset knowledge delineation conditions and according to DOM tree structure differences, acquire the delineated local graphs in the webpage tampering knowledge graph; the knowledge delineation conditions include: delineation cost is less than or equal to a preset delineation cost threshold; each difference tree node in the delineated graph nodes can find a graph node with a node correlation degree greater than or equal to a preset node correlation degree threshold; there are graph branches in the delineated graph branches with a branch correlation degree greater than or equal to a preset branch correlation degree threshold for the difference branch corresponding to the second screen snapshot; in the delineated graph branches, except for the graph branches associated with the difference branch corresponding to the second screen snapshot, the source DOM tree structure differences and the target DOM tree structure differences corresponding to the second screen snapshot conform to a standard sequence arrangement for the difference branches associated with the remaining graph branches; acquire webpage tampering annotation knowledge based on the local graphs.

6. The webpage monitoring method based on dynamic perception as described in claim 5, characterized in that, Based on the DOM tree structure difference sequence, the process of obtaining webpage tampering annotation knowledge includes: determining the first sub-graph in the local graph based on the DOM tree structure differences corresponding to the second screen snapshot; determining the second sub-graph in the local graph based on the DOM tree structure differences corresponding to the third screen snapshot; determining the first knowledge presentation screen corresponding to the webpage tampering annotation knowledge of the first sub-graph; extracting graph features based on a preset graph feature template and the first sub-graph; determining multiple graph knowledge organization routes based on preset graph knowledge organization logic and graph features; and determining the second sub-graph based on the order in which the graph knowledge organization routes first pass through the second sub-graph. The standard passes through the spectrum in sequence, and the regularity perception of the knowledge sorting route is calculated. Based on the spectrum coverage of the second sub-graph through which the knowledge sorting route passes and the preset knowledge weight of the second sub-graph, the target coverage is calculated. The regularity perception and target coverage corresponding to the knowledge sorting route are summed to obtain the route selection value. Based on the second knowledge presentation screens corresponding to the second sub-graphs through which the knowledge sorting route with the largest route selection value passes, a knowledge presentation screen sequence is constructed. The first knowledge presentation screen is inserted at the beginning of the knowledge presentation screen sequence, and the knowledge presentation screens are displayed cyclically according to the knowledge presentation screen sequence based on the preset highlighting rules.

Images