Webpage monitoring method and device

Abstract

The invention provides a webpage monitoring method and device.The method comprises the steps that based on a first request, a first feature corresponding to the first request under a preset dimension is extracted from a target webpage; the preset dimensions comprise a structure dimension, a text dimension and a link dimension; based on the second request, extracting a second feature corresponding to the second request under a preset dimension from the target webpage; the second request is obtained by adding identification information in the first request; and monitoring the target webpage based on the first feature and the second feature. According to the embodiment, the target webpage can be monitored based on the multi-dimensional features of the target webpage, the false alarm rate and the missing report rate of webpage monitoring are reduced, and the webpage monitoring accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a web page monitoring method and device. Background Art

[0002] By monitoring web pages, abnormal activities or content tampering can be discovered in a timely manner, potential security vulnerabilities can be identified, and attackers can be prevented from implanting malicious code, stealing data, or destroying web page functions, thereby ensuring web page security. Existing technologies usually monitor web pages by installing agents on the server, based on rule recognition, or based on changes in text and image features in web pages, but there are still high false positive and false negative rates, resulting in low accuracy of web page monitoring. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure at least provide a web page monitoring method, device, electronic device and storage medium, which can reduce the false alarm rate and missed alarm rate of web page monitoring and improve the accuracy of web page monitoring.

[0004] In a first aspect, an embodiment of the present disclosure provides a webpage monitoring method, comprising:

[0005] Based on the first request, extracting a first feature corresponding to the first request under a preset dimension from the target webpage; the preset dimension includes a structure dimension, a text dimension, and a link dimension;

[0006] Based on the second request, extracting a second feature corresponding to the second request under a preset dimension from the target webpage; the second request is obtained by adding identification information to the first request;

[0007] Based on the first feature and the second feature, the target webpage is monitored.

[0008] Optionally, extracting a first feature corresponding to the first request under a structural dimension from the target webpage includes:

[0009] Determine the level of each hypertext node in the target webpage in the structure tree of the target webpage and the total number of nodes under the corresponding level;

[0010] Determine the node weight of each hypertext node based on the level of each hypertext node in the structure tree and the total number of nodes under the corresponding level;

[0011] Based on the node weight of each hypertext node and the label value corresponding to the node type of each hypertext node, a first feature corresponding to the first request under the structural dimension in the target webpage is determined.

[0012] Optionally, extracting a first feature corresponding to the first request in a text dimension from the target webpage includes:

[0013] Extracting text information from a node of a preset node type in a structure tree of a target web page;

[0014] The text information is converted into a vector representation to obtain a first feature corresponding to the first request under a text dimension in the target webpage.

[0015] Optionally, extracting a first feature corresponding to the first request under the link dimension from the target webpage includes:

[0016] Get the full links corresponding to the target web page and extract the domain names corresponding to each link;

[0017] Based on the domain name corresponding to the full link, the main domain name corresponding to the target web page, the subdomain name associated with the main domain name, and the domain names in the preset list are eliminated to obtain the first feature corresponding to the first request under the link dimension in the target web page.

[0018] Optionally, monitoring the target webpage based on the first feature and the second feature includes:

[0019] Calculate the cosine similarity between the first feature and the second feature in the structure dimension and the text dimension;

[0020] Extracting the difference set and the full set of domain name lists corresponding to the first feature and the second feature under the link dimension, and taking the ratio between the difference set and the full set as the similarity between the first feature and the second feature under the link dimension;

[0021] Based on the cosine similarity between the first feature and the second feature in the structure dimension and the text dimension and the similarity between the first feature and the second feature in the link dimension, the target similarity between the target web page corresponding to the first request and the target web page corresponding to the second request is determined to monitor the target web page.

[0022] Optionally, the method further comprises:

[0023] In response to a preset time interval, extracting a third feature corresponding to the first request under a preset dimension from the target webpage based on the first request;

[0024] Based on the first feature and the third feature, the target web page is monitored.

[0025] In a second aspect, an embodiment of the present disclosure provides a webpage monitoring device, including:

[0026] A first extraction module is used to extract, based on the first request, a first feature corresponding to the first request under a preset dimension from the target webpage; the preset dimension includes a structure dimension, a text dimension, and a link dimension;

[0027] A second extraction module is used to extract a second feature corresponding to the second request under a preset dimension from the target webpage based on the second request; the second request is obtained by adding identification information to the first request;

[0028] The monitoring module is used to monitor the target web page based on the first feature and the second feature.

[0029] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the above-mentioned first aspect, or the steps in any optional implementation of the first aspect are performed.

[0030] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect, or any optional implementation of the first aspect are executed.

[0031] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor.

[0032] Any of the above aspects or any implementation methods of any of the aspects, by extracting preset multi-dimensional features from the target web page, including structural dimension, text dimension and link dimension, constructing feature sets for different requests, and realizing comprehensive and accurate web page monitoring. In the feature extraction process of the first request and the second request, attention is paid to the content changes of the target web page in different dimensions, for example, the structural dimension can capture the changes of the HTML (HyperText Markup Language) tag level, the text dimension can detect the update of the text content, and the link dimension can detect the changes of external or internal links. The process of the user's normal request for the target web page is simulated by the first request, and the corresponding multi-dimensional features are extracted based on the first request, and then the identification information is added on the basis of the first request to simulate the search engine crawler to form the second request. Since the tampered content is only displayed to the search engine crawler, after obtaining the multi-dimensional features corresponding to the second request, the multi-dimensional features corresponding to the two requests can be compared to cross-verify the web page changes from multiple angles, reduce the false alarms caused by the abnormality of a single dimension, and avoid missing subtle but critical content changes, thereby significantly reducing the false alarm rate and the missed alarm rate. In addition, by extracting the multi-dimensional features of the target web page, it can adapt to complex web page structures or dynamic content environments, improve the robustness and applicability of monitoring in diverse scenarios, and thus help to further improve the accuracy of web page monitoring.

[0033] The effects of the above-mentioned web page monitoring device, electronic device and storage medium refer to the description of the above-mentioned web page monitoring method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0035] Figure 1 A flow chart of a webpage monitoring method provided by an embodiment of the present disclosure is shown;

[0036] Figure 2 A schematic diagram of a webpage monitoring method provided by an embodiment of the present disclosure is shown;

[0037] Figure 3 A schematic diagram of a webpage monitoring device provided by an embodiment of the present disclosure is shown;

[0038] Figure 4 An exemplary system architecture is shown in which embodiments of the present disclosure may be applied;

[0039] Figure 5 A schematic diagram of the structure of a computer system of a terminal device or a server used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0041] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0042] The raising of the above problems and the solutions are the results obtained by the inventor after practice and careful research. The discovery process of the above problems and the solutions proposed for the above problems should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0043] The technical solutions in the present disclosure will be clearly and completely described below in conjunction with the drawings in the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] To facilitate understanding of this embodiment, a webpage monitoring method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the webpage monitoring method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the webpage monitoring method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0046] See also Figure 1 FIG. 1 is a flowchart of a webpage monitoring method provided by an embodiment of the present disclosure, and the method includes S101 to S103, wherein:

[0047] S101: Based on a first request, extract a first feature corresponding to the first request under a preset dimension from a target webpage; the preset dimension includes a structure dimension, a text dimension, and a link dimension.

[0048] In the disclosed embodiment, the first request can be obtained by simulating a normal user's request for the target web page, and is used to simulate the user's actions such as clicking a button, filling out a form, or scrolling the page on the target web page. In a specific implementation, a dynamic crawler can be used to simulate a normal user's request for the target web page; a dynamic crawler is a crawler tool that can process and parse dynamically loaded content, can simulate browser behavior, run Javascript code, and thus capture content dynamically generated by the front end, and can obtain dynamically loaded data by performing operations similar to human users. In addition to using the dynamic crawler method to simulate a normal user's request for the target web page, a packet capture tool can also be used to analyze the request when the web page is loaded to obtain the first request, or a headless browser can be used to manually write a script to simulate the execution of user operations; the disclosed embodiment does not specifically limit the method for obtaining the first request, and in actual applications it can be set according to actual conditions, subject to the ability to achieve its functions.

[0049] The target web page is the web page to be monitored in the embodiment of the present disclosure. After obtaining the first request, it can be sent to the target web page so that the target web page responds to the first request. In a specific implementation, if the simulated first request itself is complete and conforms to the request format expected by the target web page, it can be directly sent to the target web page for interaction. However, in the face of the target web page relying on Javascript rendering, dynamic content loading or user behavior interaction, only sending the original data of the first request may not obtain the complete response content. At this time, the first request can be loaded into the browser simulation environment to truly interact with the target web page. In addition, the first request can also be sent to the target web page first to obtain the initial response data returned by the target web page based on the first request, and then these initial response data can be used as input for further rendering in the simulated browser, thereby obtaining complete page response data.

[0050] In the embodiment of the present disclosure, after the complete response data of the target webpage is obtained based on the first request, the first feature corresponding to the first request under the preset dimension can be extracted from the complete response data. The preset dimensions here may include structural dimensions, text dimensions and link dimensions; wherein the first feature under the structural dimension refers to those features in the target webpage that can reflect the structured information of the webpage; the first feature under the text dimension may refer to the content and its semantic features in the target webpage in the form of text; the first feature under the link dimension may describe those references to external resources or links in the target webpage, which can reflect the interaction between the target webpage and the external system. It should be noted that the preset dimensions include structural dimensions, text dimensions and link dimensions only as a possible implementation method of the embodiment of the present disclosure, and do not constitute an improper limitation of the present invention. In actual applications, specific preset dimensions can also be added or adjusted according to actual monitoring needs to achieve similar effects, and the embodiment of the present disclosure does not make specific limitations on this. For example, in addition to the structural dimension, text dimension and link dimension, the preset dimension may also include a performance dimension to reflect the loading speed, resource consumption and response efficiency of the target webpage.

[0051] In the disclosed embodiment, the first feature under the structural dimension can be determined based on the structure tree of the target web page. Specifically, extracting the first feature corresponding to the first request under the structural dimension from the target web page includes: determining the level of each hypertext node in the structure tree of the target web page and the total number of nodes under the corresponding level; determining the node weight of each hypertext node based on the level of each hypertext node in the structure tree and the total number of nodes under the corresponding level; determining the first feature corresponding to the first request under the structural dimension in the target web page based on the node weight of each hypertext node and the label value corresponding to the node type of each hypertext node.

[0052] The DOM tree (Document Object Model Tree) is a tree structure representation of the document object model of a web page. It is a programmatic structure used to represent the content and structure of HTML and XML (Extensible Markup Language) documents, and organizes the elements, attributes, and text information of the document into hierarchical relationships through nodes. In the structure tree, each node represents an HTML tag, such as 、 、 In a specific implementation, a value may be predefined for each HTML tag, such as The label can correspond to 3, The label can correspond to 6, and the value here can be used to distinguish the different node types of each label. When defining the value of the HTML tag, the HTML tag is a fixed finite set, and the label value can be randomly defined for each tag type in the set. In the structure tree, the nodes have different depths and breadths, where the depth refers to the number of layers of a certain node, such as the root node. For example, the depth of the root node is 0, and the depth of the direct child node of the root node is 1, and so on; the breadth refers to the number of nodes at the same level, that is, how many brother nodes are there on the same depth layer. For any node, in addition to determining its corresponding level depth and the breadth of the same level, the node weight of the node can be determined; the method for determining the node weight here can be selected according to actual needs, for example, the node weight can be set to the product of the depth and breadth, or the depth and breadth of the node can be used as parameters, and the node weight can be calculated through linear change and other calculation processes. The embodiment of the present disclosure does not make specific limitations on this. After determining the node weight of each node, the characteristic value of each node can be further obtained, for example, the label value corresponding to each node can be multiplied by the node weight to obtain the characteristic value of the node, or the characteristic value of each node can be calculated by other methods, such as the node type and the node attribute can be spliced ​​into a string, the string can be hashed and the remainder can be taken, and the string can be spliced ​​with the number corresponding to the node type to form a characteristic value, etc. The disclosed embodiment does not make specific restrictions on this. The entire structure tree is accessed through pre-order traversal, that is, the root node is first visited, then the left child node is visited, and finally the right child node is visited. The characteristic values ​​of each node are sequentially combined into a characteristic vector, which is the first feature finally obtained, used to represent the structural characteristics of the entire target web page.

[0053] In another possible implementation, it can also be determined by generating a unique path representation for each node; or, the structural modeling method of the graph neural network can be used to capture the first feature of the target web page under the structural dimension. It should be noted that the above method of extracting the first feature under the structural dimension from the target web page is only used as an example of several feasible implementation methods of the embodiment of the present disclosure, and does not constitute an improper limitation on the present invention. In actual applications, the method of extracting the first feature under the structural dimension from the target web page can be selected or set according to actual conditions. The embodiment of the present disclosure does not make specific limitations on this, and its function can be realized.

[0054] In an embodiment of the present disclosure, extracting a first feature corresponding to a first request under a text dimension from a target web page includes: extracting text information from a node of a preset node type in a structure tree of the target web page; converting the text information into a vector representation to obtain a first feature corresponding to the first request under a text dimension in the target web page.

[0055] In a specific implementation, text information in a preset node type may be extracted. The preset node type may include <title> and< / title> <meta> The tags in the structure tree usually contain metadata, such as web page title, description, keywords, etc. This information is very important for the functionality and search engine optimization of the target web page; <title> The title text of the target web page is usually displayed in the browser tab bar and describes the core content of the target web page;< / title> <meta> contains metadata about the target page, such as a description <meta name="description"> , Keywords <meta name="keywords"> The text information in these tags usually includes the inherent information of the target web page, which can reflect the main goals and functions of the target web page. After extracting the text information in the nodes of these preset node types, the text information can be vectorized using a text processing model or other algorithms to obtain the first feature of the target web page in the text dimension.

[0056] It should be noted that the above-mentioned preset node type for extracting text information is only used as an example of a possible implementation of the embodiment of the present disclosure. In actual applications, the preset node type can be set according to actual needs and situations. For example, in addition to the node types described above, you can also extract <footer> The text information in the footer tag usually includes the copyright statement, company name and contact information of the target page, which can help identify the legal information and registration information of the target page. Alternatively, you can also extract <nav>The text information in the navigation tag is usually used to define the navigation area in the web page, including links to other parts of the web page. By extracting the text information in the node of this node type, it can help identify the overall structure and theme of the target web page. The embodiment of the present disclosure does not specifically limit the preset node type for extracting text information, and it can be set according to actual conditions in actual applications.

[0057] In an embodiment of the present disclosure, extracting a first feature corresponding to the first request under a link dimension from a target web page includes: obtaining all links corresponding to the target web page, and extracting domain names corresponding to each link; based on the domain names corresponding to the all links, removing the main domain name corresponding to the target web page, the subdomain name associated with the main domain name, and the domain names in a preset list, so as to obtain the first feature corresponding to the first request under the link dimension in the target web page.

[0058] In a specific implementation, all links in the target web page can be scanned by a crawler, and the target URL (Uniform Resource Locator) corresponding to each link can be extracted. Then, the target URL of each link can be analyzed to determine the corresponding domain name and IP (Internet Protocol) address. After obtaining the domain name and IP address of each link, the links can be filtered and screened based on the domain name and IP address of each link. Specifically, the part that does not belong to the external link can be filtered out. Since this part belongs to the website's own information, when judging whether the target web page has been maliciously tampered with, it is not necessary to judge whether the website's own links have increased, and only the external links need to be judged. In a specific implementation, the main domain name and the subdomain name associated with the main domain name can be removed. For example, for www.example.com, the main domain name is example.com; and blog.example.com and shop.example.com are all subdomains of example.com. In addition, there may still be some links that do not need to be analyzed and judged. At this time, these links can be configured into a list in advance, and then removed according to the list, thereby ensuring that the links finally retained are all necessary links required to analyze whether the target web page has been maliciously tampered with.

[0059] After eliminating all features of the target web page, the first feature of the target web page under the link dimension can be generated based on the retained links. In a specific implementation, features can be extracted by traversing the link list and counting or classifying. Specifically, the first feature of the target web page under the link dimension can be determined from aspects such as the number of links, the number of different domain names pointed to by the links, the type of IP address of the links, the structure or classification of the links, etc. It should be noted that the specific method for determining the link features in the embodiment of the present disclosure is not specifically limited. In actual applications, it can be set according to actual conditions and needs, so as to achieve its functions.

[0060] S102: Based on the second request, extract a second feature corresponding to the second request under a preset dimension from the target webpage; the second request is obtained by adding identification information to the first request.

[0061] In the disclosed embodiment, special identification information may be added to the first request to distinguish it from the first request sent by a simulated normal user. Specifically, a special UA (User-Agent) may be carried in the first request to obtain a second request, and the second request may be used to simulate a search engine crawler, which is different from the simulated normal user request of the first request. Since web pages usually distinguish between normal user requests and search engine crawlers by identifying the UA value in the request, and only display tampered content to the request of the search engine crawler, the real status of the target web page can be obtained through the second request to achieve web page monitoring.

[0062] After receiving the second request, a second feature under a preset dimension can be extracted from the target web page based on the second request. The preset dimension and the method for extracting the second feature here are similar to the method for extracting the first feature under the preset dimension from the target web page based on the first request described in S101, and will not be repeated here.

[0063] S103: Monitoring the target webpage based on the first feature and the second feature.

[0064] In this step, if Figure 2 As shown, after extracting the first feature from the target web page based on the first request and extracting the second feature from the target web page based on the second request, the target web page can be monitored by comparing the first feature with the second feature. Specifically, monitoring the target web page based on the first feature and the second feature includes: calculating the cosine similarity between the first feature and the second feature under the structural dimension and the text dimension; extracting the difference set and the full set between the domain name lists corresponding to the first feature and the second feature under the link dimension, and taking the ratio between the difference set and the full set as the similarity between the first feature and the second feature under the link dimension; based on the cosine similarity between the first feature and the second feature under the structural dimension and the text dimension and the similarity between the first feature and the second feature under the link dimension, determining the target similarity between the target web page corresponding to the first request and the target web page corresponding to the second request, so as to monitor the target web page.

[0065] In a specific implementation, different comparison methods can be used for features under different preset dimensions. For example, for the first feature and the second feature under the structural dimension and the text dimension, the cosine similarity between the two can be calculated. For the first feature and the second feature under the link dimension, the similarity between the two can be defined by the proportion of the difference set to the whole set. Exemplarily, assuming that the link domain name list corresponding to the first feature is [a, b, c], and the link domain name list corresponding to the second feature is [b, c, d], the symmetric difference set of the two domain name lists can be calculated first, that is, the different parts of the two lists. In the above example, the difference set between the two lists is [a, d], and the size is 2. Then the two domain name lists can be combined to obtain the full set of the two domain name lists [a, b, c, d], which is 4. Then the proportion of the difference set to the whole set can be calculated, and the proportion is 0.5. It can be seen that in the above example, the similarity between the first feature and the second feature under the link dimension is 0.5. It should be noted that when calculating the similarity between the first feature and the second feature in the structural dimension and the text dimension, the similarity calculation method such as Euclidean distance can also be used. The present embodiment of the disclosure does not make any specific limitations again, and its function shall be realized. When calculating the similarity between the first feature and the second feature in the link dimension, the domain name list can also be used as a text set, and the TF-IDF (Term Frequency-Inverse Document Frequency) method can be used to vectorize the external links, and then the similarity between the vectors can be calculated. The present embodiment of the disclosure does not make any specific limitations on the method for determining the similarity between the first feature and the second feature in the link dimension. In actual applications, it can be set according to actual conditions, and its function shall be realized.

[0066] In the disclosed embodiment, in addition to comparing the first feature corresponding to the first request and the second feature of the second request to achieve webpage monitoring, the first feature corresponding to the first request at the first moment and the third feature corresponding to the first request at the second moment can also be compared to achieve webpage monitoring. Specifically, the method further includes: in response to a preset time interval, extracting a third feature corresponding to the first request under a preset dimension from the target webpage based on the first request; and monitoring the target webpage based on the first feature and the third feature.

[0067] In a specific implementation, a time interval can be preset, and after the time interval has passed, the first request can be resent to the target web page to extract the third feature under the preset dimension from the target web page based on the first request. The method for extracting the third feature is similar to the method for extracting the first feature and the second feature described above, and will not be repeated here. In practical applications, the first feature and the second feature, as well as the first feature and the third feature, can be compared simultaneously, thereby realizing the horizontal and vertical comparison of the target web page features, which can further improve the accuracy of web page monitoring and reduce the false alarm rate and the missed alarm rate. Alternatively, it can also be selectively compared between the first feature and the second feature, or between the first feature and the third feature. The embodiment of the present disclosure does not make specific restrictions on this, and its function can be realized. It should be noted that the preset time interval mentioned above is only used as one of the triggering conditions for obtaining the third feature in the embodiment of the present disclosure for example. In practical applications, it can also be triggered when a certain content or component appears in the target web page to obtain the third feature, etc. The embodiment of the present disclosure does not make specific restrictions on this.

[0068] According to a second aspect of the embodiment of the present disclosure, Figure 3 As shown, a webpage monitoring device 300 is provided, comprising:

[0069] The first extraction module 301 is used to extract, based on the first request, a first feature corresponding to the first request under a preset dimension from the target webpage; the preset dimension includes a structure dimension, a text dimension, and a link dimension;

[0070] A second extraction module 302 is used to extract a second feature corresponding to the second request under a preset dimension from the target webpage based on the second request; the second request is obtained by adding identification information to the first request;

[0071] The monitoring module 303 is used to monitor the target web page based on the first feature and the second feature.

[0072] Optionally, the first extraction module 301 is specifically used for:

[0073] Determine the level of each hypertext node in the target webpage in the structure tree of the target webpage and the total number of nodes under the corresponding level;

[0074] Determine the node weight of each hypertext node based on the level of each hypertext node in the structure tree and the total number of nodes under the corresponding level;

[0075] Based on the node weight of each hypertext node and the label value corresponding to the node type of each hypertext node, a first feature corresponding to the first request under the structural dimension in the target webpage is determined.

[0076] Optionally, the first extraction module 301 is specifically used for:

[0077] Extracting text information from a node of a preset node type in a structure tree of a target web page;

[0078] The text information is converted into a vector representation to obtain a first feature corresponding to the first request under a text dimension in the target webpage.

[0079] Optionally, the first extraction module 301 is specifically used for:

[0080] Get the full links corresponding to the target web page and extract the domain names corresponding to each link;

[0081] Based on the domain name corresponding to the full link, the main domain name corresponding to the target web page, the subdomain name associated with the main domain name, and the domain names in the preset list are eliminated to obtain the first feature corresponding to the first request under the link dimension in the target web page.

[0082] Optionally, the monitoring module 303 is specifically used for:

[0083] Calculate the cosine similarity between the first feature and the second feature in the structure dimension and the text dimension;

[0084] Extracting the difference set and the full set of domain name lists corresponding to the first feature and the second feature under the link dimension, and taking the ratio between the difference set and the full set as the similarity between the first feature and the second feature under the link dimension;

[0085] Based on the cosine similarity between the first feature and the second feature in the structure dimension and the text dimension and the similarity between the first feature and the second feature in the link dimension, the target similarity between the target web page corresponding to the first request and the target web page corresponding to the second request is determined to monitor the target web page.

[0086] Optionally, the second extraction module 302 is further used for:

[0087] In response to a preset time interval, extracting a third feature corresponding to the first request under a preset dimension from the target webpage based on the first request;

[0088] Based on the first feature and the third feature, the target web page is monitored.

[0089] According to a third aspect of an embodiment of the present disclosure, an electronic device for a target web page is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention.

[0090] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present disclosure is implemented.

[0091] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and the computer program implements the method of any of the above embodiments when executed by a processor.

[0092] Figure 4 An exemplary system architecture 400 to which the webpage monitoring method or webpage monitoring apparatus implemented by the present disclosure can be applied is shown.

[0093] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0094] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0095] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0096] The server 405 may be a server that provides various services, such as a backend management server (for example only) that provides support for shopping websites browsed by users using the terminal devices 401, 402, and 403. The backend management server may process the received webpage monitoring request and feed back the processing result (for example only) to the terminal device.

[0097] It should be noted that the webpage monitoring method provided in the embodiment of the present invention is generally executed by the server 405, and accordingly, the webpage monitoring device is generally set in the server 405. The webpage monitoring method provided in the embodiment of the present invention can also be executed by the terminal devices 401, 402, 403, and accordingly, the webpage monitoring device can be set in the terminal devices 401, 402, 403.

[0098] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0099] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0100] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0101] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0102] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0103] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, a processor includes a first extraction module, a second extraction module, and a monitoring module, wherein the names of these modules do not constitute a limitation on the modules themselves in certain cases. For example, the first extraction module may also be described as "based on the first request, extracting a first feature module corresponding to the first request under a preset dimension from a target web page".

[0106] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements the following method: based on a first request, extracting a first feature corresponding to the first request under a preset dimension from a target web page; the preset dimension includes a structure dimension, a text dimension, and a link dimension; based on a second request, extracting a second feature corresponding to the second request under a preset dimension from the target web page; the second request is obtained by adding identification information to the first request; based on the first feature and the second feature, monitoring the target web page.

[0107] Finally, it should be noted that the above embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.< / nav> < / footer>

Claims

1. A webpage monitoring method, characterized in that: include: Based on the first request, extracting a first feature corresponding to the first request under a preset dimension from the target webpage; The preset dimensions include structure dimension, text dimension and link dimension; Based on the second request, extracting a second feature corresponding to the second request under a preset dimension from the target webpage; the second request is obtained by adding identification information to the first request; The target webpage is monitored based on the first feature and the second feature.

2. The method according to claim 1, characterized in that Extracting a first feature corresponding to the first request in a structural dimension from the target webpage includes: Determine the level of each hypertext node in the target webpage in the structure tree of the target webpage and the total number of nodes at the corresponding level; Determine the node weight of each hypertext node based on the level of each hypertext node in the structure tree and the total number of nodes at the corresponding level; Based on the node weight of each hypertext node and the label value corresponding to the node type of each hypertext node, a first feature corresponding to the first request under the structural dimension in the target webpage is determined.

3. The method according to claim 1, characterized in that Extracting a first feature corresponding to the first request in a text dimension from the target webpage includes: Extracting text information from a node of a preset node type in the structure tree of the target web page; The text information is converted into a vector representation to obtain a first feature corresponding to the first request under a text dimension in the target webpage.

4. The method according to claim 1, characterized in that Extracting a first feature corresponding to the first request under a link dimension from the target webpage includes: Obtain the full link corresponding to the target web page, and extract the domain name corresponding to each link; Based on the domain name corresponding to the full link, the main domain name corresponding to the target web page, the subdomain name associated with the main domain name, and the domain names in the preset list are eliminated to obtain a first feature corresponding to the first request under the link dimension in the target web page.

5. The method according to claim 1, characterized in that Monitoring the target webpage based on the first feature and the second feature includes: Calculate the cosine similarity between the first feature and the second feature in the structure dimension and the text dimension; Extracting a difference set and a full set of domain name lists corresponding to the first feature and the second feature under the link dimension, and taking a ratio between the difference set and the full set as the similarity between the first feature and the second feature under the link dimension; Based on the cosine similarity between the first feature and the second feature in the structure dimension and the text dimension and the similarity between the first feature and the second feature in the link dimension, the target similarity between the target web page corresponding to the first request and the target web page corresponding to the second request is determined to monitor the target web page.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to a preset time interval, extracting a third feature corresponding to the first request under a preset dimension from the target webpage based on the first request; The target webpage is monitored based on the first feature and the third feature.

7. A webpage monitoring device, characterized in that: include: A first extraction module, configured to extract, based on the first request, a first feature corresponding to the first request under a preset dimension from a target webpage; the preset dimension includes a structure dimension, a text dimension, and a link dimension; A second extraction module is used to extract a second feature corresponding to the second request under a preset dimension from the target webpage based on the second request; the second request is obtained by adding identification information to the first request; A monitoring module is used to monitor the target webpage based on the first feature and the second feature.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images