A page recognition method, device, equipment and storage medium

Abstract

The present application discloses a page identification method, the method comprising: obtaining first hierarchical structure information including the hierarchy of any page of a website relative to the homepage of the website; wherein the first hierarchical structure information includes at least one node in any page; the at least one node is a block element node that meets a specified condition contained in any page; obtaining first region information of the first node; wherein the first node is any node in the at least one node; the first region information is the region information of the block element contained in the first node when the page where the first node is located is displayed; based on the first hierarchical structure information and the first region information, identifying the type of the page where the first node is located. The present application also discloses a page identification device, equipment and computer-readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a page recognition method, device, equipment and computer-readable storage medium. Background Art

[0002] The diversity of website types and the high frequency of dynamic data updates on websites have led to an explosive growth in the amount of data processing for website data collection services. In related technologies, it is usually necessary to first determine the channel pages on each website and use the channel pages as a benchmark for high-speed data collection. In actual applications, in order to determine whether the current web page is a channel page, it is usually implemented by manually configuring page recognition conditions or relying on a large amount of sample data with labels to identify page features. However, the method of manually configuring page recognition conditions is inefficient, and when the website changes, the page recognition conditions need to be modified; the method of relying on a large amount of sample data with labels to identify page features is too dependent on sample data, and the labeling of sample data also requires a lot of manpower costs. Summary of the invention

[0003] The present application provides a page recognition method, which overcomes the disadvantage of low efficiency caused by manually setting page recognition conditions during the page recognition process, and can also quickly and efficiently classify pages without relying on sample data carrying labels.

[0004] The page identification method provided by this application is implemented as follows:

[0005] The present application provides a page identification method, the method comprising:

[0006] Acquire first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website; wherein the first hierarchical structure information includes at least one node in any page; the at least one node is a block element node contained in any page and meeting a specified condition;

[0007] Acquire first region information of a first node; wherein the first node is any node of the at least one node; the first region information is region information of block elements contained in the first node when a page where the first node is located is displayed;

[0008] Based on the first hierarchical structure information and the first region information, the type of the page where the first node is located is identified.

[0009] In some implementations, the identifying the type of the page where the first node is located based on the first hierarchical structure information and the first region information includes:

[0010] Acquire first quantity information from the first hierarchical structure information; wherein the first quantity information represents quantity information of second nodes; the second node includes each child node of the first node; the second node is a node in a page corresponding to a link of the first node;

[0011] Determine a characteristic parameter of the first node based on the first quantity information and the first area information; wherein the characteristic parameter of the first node is used to determine the type of the first node;

[0012] Based on the characteristic parameters of the first node, the type of the page where the first node is located is identified.

[0013] In some implementations, determining the characteristic parameter of the first node based on the first quantity information and the first area information includes:

[0014] Acquire second region information; wherein the second region information is region information of block elements included in the second node when the page where the second node is located is displayed;

[0015] Determining initial characteristic parameters of the first node based on the second area information, the first quantity information, and the first area information;

[0016] Based on the initial characteristic parameters of the first node, characteristic parameters of the first node are determined.

[0017] In some implementations, determining the initial characteristic parameter of the first node based on the second area information, the first quantity information, and the first area information includes:

[0018] Acquire third area information and second quantity information; wherein the third area information is area information corresponding to the block elements contained in the third node when the page where the third node is located is displayed; the second quantity information is the number of the third nodes; the third node is a child node of the second node; the third node is a node in the link page corresponding to the second node;

[0019] acquiring a characteristic parameter of the second node based on the third area information, the second quantity information, and the second area information;

[0020] Obtaining a reference characteristic value based on the characteristic parameter of the second node and the first quantity information;

[0021] The first region information and the benchmark feature value are processed to determine initial feature parameters of the first node.

[0022] In some implementations, determining the characteristic parameter of the first node based on the initial characteristic parameter of the first node includes:

[0023] determining a correction parameter based on the second area information and the first quantity information;

[0024] Based on the correction parameter, the initial characteristic parameter of the first node is corrected to determine the characteristic parameter of the first node.

[0025] In some implementations, determining the correction parameter based on the second region information and the first quantity information includes:

[0026] Based on the first quantity information, obtaining a standard deviation of the second area information; wherein the standard deviation of the second area information is a standard deviation of the area information included in each piece of the second area information;

[0027] Based on the standard deviation, the correction parameter is determined.

[0028] In some implementations, before obtaining the first hierarchical structure information including the hierarchy of any page of the website relative to the website homepage, the method further includes:

[0029] Load and display the page of the current website, and in the case where any node of the page includes the block element, if the block element meets the specified condition, obtain the hierarchical information of the page relative to the homepage of the website; wherein the hierarchical information of any node of the page represents the hierarchical information of any node of the page relative to the homepage of the website;

[0030] The first hierarchical structure information is determined based on the hierarchical information of the page relative to the homepage of the website.

[0031] In some implementations, determining the first hierarchical structure information based on the hierarchical information of the page relative to the homepage of the website includes:

[0032] In a case where the address information of the page matches the address information of the current website, the first hierarchical structure information is determined based on the hierarchical information of the page relative to the homepage of the website.

[0033] In some embodiments, the block element satisfies the specified conditions, including: the area information of the block element when displayed satisfies the specified conditions, including: the position information of the area information of the block element matches the specified position information, and the area information of the area information of the block element satisfies the specified area range.

[0034] The present application also provides a page recognition device, which includes: an acquisition module and a processing module; wherein:

[0035] The acquisition module is used to acquire first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website, and first area information of the first node; wherein the first hierarchical structure information includes at least one node in the any page; the at least one node is a node of a block element that meets a specified condition contained in the any page; the first node is any node in the at least one node; the first area information is the area information of the block element contained in the first node when the page where the first node is located is displayed;

[0036] The processing module identifies the type of the page where the first node is located based on the first hierarchical structure information and the first region information.

[0037] The present application also provides a page recognition device, which includes the page recognition apparatus as described above.

[0038] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the page recognition method described in any one of the above items is implemented.

[0039] From the above, it can be seen that the page recognition method provided by the present application can obtain the first level information including the level information of at least one node relative to the current website homepage, and can also obtain the area information of the first node when the page where the first node is located is displayed, that is, the first area information, and then determine the type of the page where the first node is located based on the first level information and the first area information. Therefore, the page recognition method provided by the present application improves the problem of inefficiency caused by manually setting the page recognition conditions, and does not need to rely on massive sample data, reducing the amount of calculation, and the operation of determining the category of the page when the page is displayed also improves the accuracy of page feature recognition, thereby achieving efficient and accurate determination of the page type. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the flow of the first page recognition method provided in this application;

[0041] Figure 2 A flow chart of the second page identification method provided by this application;

[0042] Figure 3 A flow chart for identifying page types in a target website provided by this application;

[0043] Figure 4A schematic diagram of the flow of the third page identification method provided in this application;

[0044] Figure 5 A schematic diagram of the page structure displayed for any page of the current website;

[0045] Figure 6 A schematic diagram of the structure of the second-level structure information provided for this application;

[0046] Figure 7 A schematic diagram of the process of judging each node in the second-level structure information provided by this application;

[0047] Figure 8 A schematic diagram of the overall process of identifying the type of page where the first node is located provided by this application;

[0048] Fig. 9 A schematic diagram of the structure of a page recognition device provided by the present application;

[0049] Fig.10 A schematic diagram of the structure of a page recognition device provided in this application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0051] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The page recognition method provided in the embodiments of the present application relates to the field of information technology, and in particular to a page recognition method, system, device and computer-readable storage medium.

[0053] At present, the data collection volume corresponding to the data collection business of various types of websites shows an explosive growth trend with the diversification of website data, the high speed of website changes, and the high frequency of data updates.

[0054] The premise of efficient data collection is to be able to identify the type of the web page and collect data for specific types of web pages. In related technologies, the basic configuration conditions for determining the type of web page are usually manually configured by data collectors. For example, when data collectors need to collect data from a news website, they need to manually search for channel pages including "People's Livelihood Online", "Government News", "People's Livelihood Voice", "Technology Online", "Mother and Baby Channel", etc., and determine the configuration conditions based on the search results.

[0055] However, in actual data collection, especially for the revision or addition of each channel page in the dynamic changes of the website, there is no feasible and targeted automated solution to realize the search of channel pages. This requires data collectors to manually traverse the website in patrol mode at regular intervals, which consumes a lot of manpower costs and is inefficient.

[0056] In order to improve the above-mentioned inefficiency of purely relying on data collectors to manually traverse websites, the related technology also adopts the method of manually configuring page identification conditions, which is also a common technical solution in the industry, that is, the technicians write the collection program framework, and perform xpath search configuration according to the characteristics of each website, for example, write the path of the list node, title node, and text node, and let the collection program framework automatically collect data from the corresponding website. However, such a manually configured solution still needs to rely on the technicians to manually configure the search conditions, and such an operation still consumes a lot of time and labor costs.

[0057] In order to improve the defects of manually configured page recognition conditions, the relevant technology also proposes a method of searching each page of a website through a neural network. The neural network is trained through a machine learning algorithm and a large amount of sample data with labels.

[0058] Through the neural network, nodes in HyperText Markup Language (HTML) can be identified. Specifically, each page of the website is crawled out by a crawler, and the HTML source code of each page is identified by the neural network, so as to obtain the characteristics of the page and determine whether the page is a channel page.

[0059] Through the neural network, the screenshot of the web page after being loaded and displayed can also be identified and detected, so as to identify the characteristic information of the web page and further determine whether the web page is a channel page.

[0060] The process of data search through neural networks has improved the inefficiency of manually configuring page recognition conditions to a certain extent. However, the training process of neural networks needs to rely on a large amount of sample data with labels, and the labeling of these data samples also requires a lot of manpower and time costs. In addition, the training process of neural networks with higher recognition rates also requires the help of massive sample data. Therefore, the feasibility of page recognition solutions based on neural networks is relatively low.

[0061] On the other hand, the recognition result of HTML source code can only obtain the tree structure information between each node, but cannot obtain the actual visual information, and the reliability of its recognition basis is insufficient; for the neural network for web page screenshot recognition and detection, the sample data required for its training is millions of labeled sample data. Therefore, although the recognition accuracy of the neural network for page rendering effect screenshots is improved compared with the recognition rate of HTML source code, the amount of labeled sample data required is too large, which makes the training cost of the neural network too high.

[0062] Based on the above problems, an embodiment of the present application provides a page recognition method, which improves the inefficiency caused by manually setting page recognition conditions during the page recognition process, and can also quickly and efficiently determine the page type without relying on sample data carrying labels.

[0063] like Figure 1 The flowchart of the page recognition method provided by the embodiment of the present application is shown. The page recognition method can be implemented by a processor in a page recognition device.

[0064] It should be noted that the above-mentioned processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0065] like Figure 1 As shown, the page identification method provided by the embodiment of the present application may include the following steps:

[0066] Step 101: Obtain first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website.

[0067] The first level structure information includes at least one node in any page; the at least one node represents a block element node in any page that meets a specified condition.

[0068] In one implementation, a block element node may represent multiple single block element nodes.

[0069] In one implementation, a block element node may represent a node in which at least one block element is nested.

[0070] In one implementation, the specified condition may indicate that the number of block element nodes is greater than a preset threshold.

[0071] In one implementation, the specified condition may indicate that the intervals between each block element node are relatively small.

[0072] In one embodiment, the level information of any page of the website relative to the current website homepage may be the first number information. Exemplarily, the first number information may be the jump number information from the website homepage to any page set when building the website.

[0073] In one embodiment, the hierarchical information of any page of the website relative to the current website homepage may be obtained by obtaining the first numbering information and then obtaining the second numbering information of each node of any page of the website relative to the first node of the page where it is located. In other words, the first hierarchical structure information may include a combination of the first numbering information and the second numbering information.

[0074] In one implementation, the first hierarchical structure information may include only one level when the current website includes only one page.

[0075] In one implementation, since the first hierarchical structure information substantially includes the hierarchical information of at least one node in any page relative to the homepage of the website, the first hierarchical structure information may also include attribute information of each node.

[0076] In one implementation, the attribute information of each node may include at least one of the following:

[0077] Whether each node contains nested sub-nodes, the type of sub-nodes, the history of sub-node editing, the time information of sub-node editing, the number of times sub-node editing, the data information carried by the sub-node, and the link jump information corresponding to the sub-node.

[0078] Step 102: Obtain first area information of the first node.

[0079] The first node is any node among the at least one node; the first region information is region information of block elements included in the first node when the page where the first node is located is displayed.

[0080] In one implementation, the page where the first node is located is displayed, which may be a normal display state when the page where the first node is located is loaded by the browser.

[0081] In one implementation, the page where the first node is located is displayed, which may be a loading display when the page where the first node is located is loaded by a browser capable of executing an automated testing function.

[0082] In one embodiment, the browser capable of performing automated testing functions may be a simulated browser, such as a browser configured with Selenium, such as Chrome or Firefox, to automatically browse web pages and complete corresponding actions according to instructions. Selenium is an automated network testing tool that can be run directly in a browser, just like a real user operating it.

[0083] In one implementation, the first node may include at least one block element.

[0084] In one implementation, the first node may represent a list node, such as a news list, an article list, a post list, etc. Exemplarily, the above lists are usually composed of highly similar and repeated block elements arranged in parallel vertically, and occupy the main position of the entire page.

[0085] In one implementation, the region information of the block element included in the first node may represent region information of a starting position of the block element relative to the first node.

[0086] In one implementation, the region information of the block element included in the first node may represent region information of the block element relative to the starting position of the first node of the current page.

[0087] In one implementation, the region information of the block element included in the first node may represent region information of the block element relative to a geometric center point of the current page.

[0088] In one implementation, the first area information may be obtained by detecting node features when the page where the first node is located is displayed. Exemplarily, the node features may include node coordinates, node coverage area, and the like.

[0089] In one implementation, the first region information may be obtained by identifying page features when the page where the first node is located is displayed.

[0090] In one implementation, the first region information may represent page coordinate position information corresponding to data carried by the first node after being displayed when the page where the first node is located is displayed.

[0091] In one implementation, the first region information may represent page area information after the data carried by the first node is displayed when the page where the first node is located is displayed.

[0092] Step 103: Based on the first hierarchical structure information and the first region information, identify the type of the page where the first node is located.

[0093] In one implementation, the type of the page where the first node is located may indicate the type of data carried by the page where the first node is located, for example, whether the page displays text information or picture information.

[0094] In one implementation, the type of the page where the first node is located may represent the type of the link target contained in the page where the first node is located.

[0095] In one implementation, the type of the page where the first node is located may indicate whether the page where the first node is located is a channel page. The channel page may indicate a page including list information, and the list information may be a news list, an article list, a post list, and the like.

[0096] In one implementation, based on the first hierarchical structure information and the first region information, identifying the type of the page where the first node is located may be achieved in the following manner:

[0097] The hierarchical information of the page where the first node is located is obtained from the first hierarchical structure information, and the type of the page where the first node is located is identified based on the hierarchical information of the page where the first node is located and the first region information.

[0098] In one implementation, based on the first hierarchical structure information and the first region information, identifying the type of the page where the first node is located may be achieved in the following manner:

[0099] The hierarchical information of the first node is obtained from the first hierarchical structure information, and the type of the page where the first node is located is identified based on the hierarchical information of the first node and the first region information.

[0100] In one implementation, based on the first hierarchical structure information and the first region information, identifying the type of the page where the first node is located may be achieved in the following manner:

[0101] The hierarchical information of the first node and the hierarchical information of the page where the first node is located are obtained from the first hierarchical structure information, and the type of the page where the first node is located is identified based on the hierarchical information of the first node, the hierarchical information of the page where the first node is located and the first area information.

[0102] In one implementation, based on the first hierarchical structure information and the first region information, identifying the type of the page where the first node is located may be achieved in the following manner:

[0103] The hierarchical information of the first node and the hierarchical information of the child nodes of the first node are obtained from the first hierarchical structure information, and the type of the page where the first node is located is identified based on the hierarchical information of the first node, the hierarchical information of the child nodes of the first node and the first area information.

[0104] From the above, it can be seen that the page recognition method provided by the embodiment of the present application can obtain the first level information including the level of any page of the website relative to the current website homepage, and can also obtain the area information of the first node when the page where the first node is located is displayed, and then determine the type of the page where the first node is located based on the first level information and the first area information. Therefore, the page recognition method provided by the embodiment of the present application improves the problem of inefficiency caused by manually setting the page recognition conditions, and does not need to rely on massive sample data, reducing the amount of calculation, and the operation of determining the category of the page when the page is displayed also improves the accuracy of page feature recognition, thereby realizing efficient and rapid determination of the page type.

[0105] Based on the above embodiments, the present application provides a page recognition method, such as Figure 2 As shown, the page identification method may include the following steps:

[0106] Step 201: Obtain first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website.

[0107] The first level structure information includes at least one node in any page; the at least one node is a block element node that meets specified conditions and is contained in any page.

[0108] Step 202: Obtain first area information of the first node.

[0109] The first node is any node among the at least one node; the first region information is region information of block elements included in the first node when the page where the first node is located is displayed.

[0110] Step 203: Obtain first quantity information from the first hierarchical structure information.

[0111] The first quantity information indicates the quantity information of the second node; the second node includes the child nodes of the first node; and the second node is a node in the page corresponding to the link of the first node.

[0112] In a real-time manner, the first node carries link information, and accordingly, the page corresponding to the link of the first node may represent the page to which the address jumps and is rendered after the first node is selected; exemplarily, the page may include multiple nodes, among which the second node may be included.

[0113] In one implementation, the first node corresponds to a linked page, which may include a node including at least one block element.

[0114] In one implementation, the first node corresponds to a linked page, which may include multiple nodes each including only one block element.

[0115] In one implementation, the first quantity information may be 0.

[0116] In one implementation, the first quantity information may be an integer greater than or equal to 1.

[0117] Step 204: Determine a characteristic parameter of the first node based on the first quantity information and the first area information.

[0118] The characteristic parameter of the first node is used to determine the type of the first node.

[0119] In one implementation, the characteristic parameter of the first node may indicate the type of data carried by the first node, for example, the data carried by the first node is a picture or text information.

[0120] In one implementation, the characteristic parameter of the first node may represent the size of the amount of data carried by the first node. For example, the amount of data carried by the first node is 10 KB.

[0121] Exemplarily, step 204 may be implemented through steps A1 to A3:

[0122] Step A1: Obtain second area information.

[0123] The second area information is the area information corresponding to the block element included in the second node when the page where the second node is located is displayed.

[0124] In the embodiment of the present application, the method for obtaining the second region information is similar to the method for obtaining the first region information, and will not be described in detail here.

[0125] Step A2: determining initial characteristic parameters of the first node based on the second area information, the first quantity information and the first area information.

[0126] The initial characteristic parameter of the first node is a characteristic set of the first region information and each second region information.

[0127] In one implementation, the initial characteristic parameter of the first node may be a parameter representing the influence of the first quantity information and the second area information on the first area information.

[0128] In one implementation, the initial characteristic parameter of the first node may represent a parameter of the influence of the region information in the second region information that is greater than the first value on the first region information, wherein the first value is a positive number.

[0129] In one implementation, the initial characteristic parameter of the first node may be a parameter indicating the degree of influence of the first quantity information on the first region information.

[0130] In one embodiment, the initial characteristic parameters of the first node may represent the basic characteristic parameters of the first node. For example, the characteristics of the pages generated after the first node with the first region information is selected, and illustratively, the characteristics and quantity information of these pages may reflect the type and quantity of links carried by the first node. For a node carrying channel navigation content, the type of links carried by it may be the same or similar, and the number may be multiple.

[0131] Exemplarily, step A2 may also be implemented through steps B1 to B4:

[0132] Step B1, obtaining third area information and second quantity information.

[0133] Among them, the third area information is the area information corresponding to the block elements contained in the third node when the page where the third node is located is displayed; the second quantity information is the number of third nodes; the third node is the child node of the second node; the third node is the node in the link page corresponding to the second node.

[0134] In one implementation, the second quantity information may be acquired from the first hierarchical structure information based on the second node after the second node is determined.

[0135] In one implementation, the second quantity information may represent the total quantity of all child nodes of the second node.

[0136] In one implementation, the second quantity information may represent the quantity of some sub-nodes in the second node.

[0137] In one implementation, the third region information may be obtained when the link corresponding to the second node is selected and the link page is loaded and displayed. The method for obtaining the third region information is similar to the method for obtaining the first region information, and will not be described in detail here.

[0138] Step B2: Acquire characteristic parameters of the second node based on the third area information, the second quantity information and the second area information.

[0139] In the embodiment of the present application, the characteristic parameters of the second node are similar to the characteristic parameters of the first node and are not described again here.

[0140] In an embodiment of the present application, obtaining characteristic parameters of a current node, such as a first node or a second node, can be determined based on relevant parameters of child nodes of the current node, such as the child node of the first node, i.e., the second node, and the child node of the second node, i.e., the third node.

[0141] In the embodiment of the present application, the method for obtaining the characteristic parameters of the second node is similar to the method for obtaining the initial characteristic parameters of the first node, and will not be repeated here.

[0142] Step B3: Obtain a reference characteristic value based on the characteristic parameter of the second node and the first quantity information.

[0143] The reference feature value is used to represent the feature set of each child node of the first node.

[0144] In one implementation, the characteristic parameter of the second node may represent the sum of characteristic parameters of each child node of the first node, namely, the second node, and the characteristic parameter of each child node may be different.

[0145] In one implementation, the reference characteristic value may be obtained by weighted calculation of characteristic parameters of each second node.

[0146] In one implementation, the reference characteristic value may be obtained by processing the characteristic parameter of each second node based on the first quantity information and a characteristic parameter balancing algorithm.

[0147] In one implementation, the benchmark characteristic value may be obtained by averaging characteristic parameters of all second nodes based on the first quantity information.

[0148] In one implementation, the reference characteristic value may be data obtained by smoothing the characteristic parameters of each second node and capable of reflecting the overall characteristics of all child nodes of the first node.

[0149] Step B4: Process the first region information and the reference characteristic value to determine the initial characteristic parameters of the first node.

[0150] In one implementation, the initial characteristic parameter of the first node may be obtained by processing the first region information and the reference characteristic value through a characteristic parameter correlation algorithm.

[0151] In one implementation, the initial characteristic parameter of the first node may be obtained by weighting the first region information and the reference characteristic value according to the numerical value of the first region information and the numerical value of the reference characteristic value.

[0152] In one implementation, the initial characteristic parameter of the first node may be obtained by superimposing the first region information and the reference characteristic value.

[0153] Exemplarily, the initial characteristic parameter of the first node may be determined by formula (1):

[0154]

[0155] In formula (1), P is the initial characteristic parameter of the first node, is the first area information, is the benchmark eigenvalue obtained by accumulating and adding the characteristic information of the second node, is the feature information of the i-th second node, N is the level information of the first node in the first hierarchical structure, and M is the quantity information of the second node, that is, the first quantity information. Wherein, M and N are both integers greater than or equal to 2, and i is an integer greater than or equal to 1. Exemplarily, formula (1) can be a calculation step of the memory forgetting algorithm, and accordingly, It can be the memory amount of the i-th second node.

[0156] Step A3: Determine the characteristic parameters of the first node based on the initial characteristic parameters of the first node.

[0157] The characteristic parameter of the first node is used to determine the type of the first node.

[0158] In one implementation, the type of the first node may indicate whether the first node is a node carrying designated information.

[0159] In one implementation, the type of the first node may indicate whether the first node includes at least one link target.

[0160] In one implementation, the type of the first node may represent the type of at least one link target included in the first node.

[0161] In one implementation, the type of the first node may indicate that at least one link target included in the first node is of a channel navigation type.

[0162] In one implementation, the characteristic parameter of the first node may represent the probability of a node in the first node carrying designated information.

[0163] In one implementation, the characteristic parameter of the first node may indicate a characteristic of at least one link target included in the first node, such as whether the link target is an invalid link or whether the link target is a picture link.

[0164] In one implementation, the characteristic parameter of the first node may represent a characteristic of at least one link target included in the first node.

[0165] In one embodiment, the characteristic parameters of the first node may indicate the type of page data corresponding to the link target of at least one channel navigation type contained in the first node, for example, whether the page data characteristics corresponding to the link targets of these channel navigation types are mainly pictures or mainly text information.

[0166] In one implementation, the characteristic parameter of the first node may be obtained by correcting the initial characteristic parameter of the first node using a node correction algorithm.

[0167] In one implementation, the characteristic parameter of the first node may be obtained by weighted calculation of the initial characteristic parameter of the first node.

[0168] In one implementation, the characteristic parameter of the first node may be obtained by modifying the initial characteristic parameter of the first node according to the level information of the first node.

[0169] In one implementation, the characteristic parameters of the first node may be obtained by modifying the initial characteristic parameters of the first node based on the overall structure of the website, that is, the first hierarchical structure information.

[0170] Exemplarily, step A3 may be implemented by steps D1 and D2:

[0171] Step D1: Determine a correction parameter based on the second area information and the quantity information of the second nodes.

[0172] In one implementation, the correction parameter may be matrix data.

[0173] In one implementation, the correction parameter may be a specific number.

[0174] In one implementation, the correction parameter may be obtained by performing statistical calculations on the second region information through the quantity information of the second nodes.

[0175] In one implementation, the correction parameter may be obtained by statistically averaging the second region information through the quantity information of the second nodes.

[0176] Exemplarily, step D1 may be implemented by steps E1 and E2:

[0177] Step E1: Based on the quantity information of the second node, obtain the standard deviation of the second area information.

[0178] The standard deviation of the second region information is the standard deviation of the area information included in each second region information.

[0179] In the embodiment of the present application, the standard deviation of the second region may represent a penalty term in the forgetting memory algorithm.

[0180] Step E2: Determine a correction parameter based on the standard deviation.

[0181] In one implementation, the correction parameter may be obtained by weighting the standard deviation.

[0182] In one implementation, the correction parameter may be obtained by obtaining an empirical parameter and performing a weighted calculation on the standard deviation based on the empirical parameter.

[0183] In one implementation, the correction parameter may be obtained by compensating the standard deviation, taking the inverse of the compensation result, and then weighting it.

[0184] In one implementation, the correction parameter may be used to represent compensation for a penalty term in a forgetting memory algorithm.

[0185] Exemplarily, the correction parameter in the embodiment of the present application can be determined by formula (2):

[0186]

[0187] In formula (2), β represents the correction parameter, c represents the empirical parameter, which is generally taken as 0.31, M represents the first quantity information; δ represents the standard deviation of the second area information, that is, The standard deviation of , where N is the Nth level in the first level structure information, Represents the area information of the first second node, the second second node, and the Mth second node of the first node, and N and M are both integers greater than or equal to 2. It should be noted that for each node, β needs to be calculated separately. In addition, the number of levels in the first hierarchical structure information starts from the terminal node, that is, the level of the terminal node is the first level.

[0188] In formula (2), δ can be calculated by formula (3):

[0189]

[0190] In formula (3), for The average value of . is the area information of the i-th second node.

[0191] Step D2: Based on the correction parameter, correct the initial characteristic parameter of the first node to determine the characteristic parameter of the first node.

[0192] In one implementation, the characteristic parameter of the first node may be realized by weighting the initial characteristic parameter of the first node using the correction parameter.

[0193] Exemplarily, the characteristic parameter of the first node may be determined by formula (4):

[0194]

[0195] In formula (4), β represents the correction parameter, M is the number of child nodes of the second node, i.e., the first node. is the first area information, It is the benchmark feature value obtained by cumulatively adding the feature information of the second node.

[0196] It should be noted that formula (4) is an iterative calculation process, that is, based on the feature information of the child nodes of the current node and the region information of the current node, the feature parameters of the current node can be obtained by calculating the correction parameters. For example, It can be obtained through the second area information, the area information and quantity information of the child nodes of the second node through formula (4), which will not be described in detail here.

[0197] Exemplarily, when the first node is a terminal node in the first hierarchical structure information, the values ​​of N and β are both 1, and the values ​​of δ and M are both 0.

[0198] Combining equations (1) to (4), we can obtain the total calculation formula (5) of the characteristic parameters of the first node:

[0199]

[0200] Exemplarily, the above formula (1) to formula (5) can be the memory amount of the first node obtained by the memory forgetting algorithm When the first node is a terminal node, that is, N=1,

[0201] Exemplarily, the operations performed by the above formulas (1) to (5) can be implemented by a specific recognition module.

[0202] For example, after the memory capacity of any node of the current website is calculated, the nodes can be sorted according to the memory capacity, and during actual data collection, data can be collected only from the nodes with the highest ranking.

[0203] Step 205: Based on the characteristic parameters of the first node, identify the type of the page where the first node is located.

[0204] In one implementation, the type of the page where the first node is located can be identified by the number of first nodes of the first type in the page. Exemplarily, the first type can indicate that the first node is a node that includes a link target.

[0205] In one embodiment, the type of the page where the first node is located can be identified by statistical analysis of the types of each first node in the page. For example, in the page, there are N1 first nodes of the first type, N2 first nodes of the second type, and Nn first nodes of the nth node. Then, the type of the page can be determined according to the weight of the number of nodes of each type in the total number of nodes. Wherein, N1, N2, Nn and n are all integers greater than or equal to 0.

[0206] In one implementation, the type of the page where the first node is located may be determined by the type of any first node in the page. Exemplarily, if the first node belongs to the first type, then the page belongs to the first type of page.

[0207] In one embodiment, as long as any first node contains a large number of link targets, such as 8, and each link target is a navigation link, it can be determined that the first node belongs to a channel navigation node. At this time, it can be determined that the type of the page where the first node is located is a channel page.

[0208] Illustratively, the flowchart of the embodiment of the present application for identifying the page type in a target website is as follows: Figure 3 As shown. Figure 3 In the example, the target website is the current website. The process of identifying the page type in the target website may include the following steps:

[0209] Step 301: simulate a browser to open the homepage of a target website.

[0210] Exemplarily, the simulated browser may automatically open the homepage of the target website, or may open the homepage of the target website when a trigger condition is met.

[0211] Step 302: Identify the block elements after the web page is rendered.

[0212] Exemplarily, identifying the block element may be by acquiring area information, number information, etc. of the node where the block element is located.

[0213] Step 303: recursively open the subpage according to the webpage link.

[0214] Exemplarily, opening the subpage may be achieved by simulating the browser's selection and loading of a link contained in the node where the block element is located.

[0215] Step 304: All links identified as “channel pages” are summarized and output.

[0216] Exemplarily, the parameters in formula (1) to formula (5) in the aforementioned embodiment can be first obtained, and the type of the node where the block element is located can be determined through the obtained parameters (1) to formula (5), and the type of the page where the node is located can be determined according to the type of the node where the block element is located, and the links of all channel pages in the target website can be summarized and output.

[0217] From the above, it can be seen that the page recognition method provided by the embodiment of the present application, after obtaining the first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website, and the first area information of the first node, can also obtain the quantity information of each child node of the first node, that is, the first quantity information, and then determine the characteristic parameters of the first node according to the first quantity information and the first area information, and then determine the type of the page where the first node is located according to the characteristic parameters of the first node. Therefore, the page recognition method provided by the embodiment of the present application, when determining the page type, does not need to rely on manually set page recognition conditions, nor does it need to rely on a large amount of sample data carrying labels, thereby reducing labor costs, time costs and computing costs; when rendering the page, the type of the current node is determined based on the parameter information of the current node and child nodes after rendering, thereby realizing fast and accurate classification of pages.

[0218] Based on the above embodiments, the present application provides a page recognition method, such as Figure 4 As shown, the page identification method may include the following steps:

[0219] Step 401, load and display the page of the current website, and when any node of the page includes a block element, if the block element meets a specified condition, obtain the hierarchical information of the page relative to the homepage of the website.

[0220] The level information of the page relative to the homepage of the website includes the level information of any node of the page relative to the homepage of the website.

[0221] In one implementation, the above page may be any page of the current website.

[0222] In one implementation, step 401 may be performed on the basis of obtaining the second hierarchical structure information, wherein the second hierarchical structure information includes hierarchical information of any node of the current website, and the hierarchical information of any node may be hierarchical information of any node in any page of the current website, that is, hierarchical information of any node contained in the current website relative to the homepage of the website.

[0223] In one implementation, the level information of any node represents the level information of each node in the current website page relative to the website homepage.

[0224] In one implementation, the hierarchical information of any node may also include the hierarchical information of the page where any node is located relative to the homepage of the current website.

[0225] In one implementation, each node of the current website and the hierarchical information of each node relative to the website homepage may be obtained by counting the nodes in the HTML of the current website.

[0226] In one implementation, each node of the current website and the hierarchical information of each node relative to the website homepage may be obtained by counting a certain type of node in the HTML of the current website.

[0227] In one embodiment, each node of the current website can be obtained by counting the Document Object Model (DOM) in HTML. DOM is a collection of nodes or information fragments organized in a hierarchical structure. This hierarchical structure allows developers to navigate in the tree to find specific information. Analyzing this structure usually requires loading the entire document and constructing the hierarchical structure. Since it is based on the information hierarchy, DOM is considered to be tree-based or object-based.

[0228] In one embodiment, loading and displaying each page of the current website can be achieved by simulating a browser, which usually uses the Python programming language to control a browser on a computer, such as Chrome or Firefox, to automatically browse web pages and perform corresponding actions according to instructions, such as clicking, scrolling, switching pages, etc.

[0229] Exemplarily, the block element satisfies the specified condition, including: the position information of the region information of the block element matches the specified position information, and the area information of the region information of the block element satisfies the specified area range.

[0230] In one implementation, the region information of the block element may represent region information when the block element is displayed.

[0231] In one implementation, the block element being displayed may mean that when a page where the block element is located is displayed, the block element is displayed accordingly.

[0232] In one implementation, the position information of the region information matches the specified position information, which may indicate that the coordinate information of the block element in the page is within the specified coordinate region.

[0233] In one implementation, the area information of the region information satisfies a specified area range, which may include that a display area of ​​the block element on the page is within a specified area interval.

[0234] In one implementation, the level information of each node may represent the level information of the node in the second level structure information.

[0235] For example, when each page of the current website is displayed and the node of the page includes a block element, the page structure diagram of each page can be as follows: Figure 5 As shown, in Figure 5 In the example, the page includes three nodes #1, #2 and #3, and #2 may include three child nodes #2-1, #2-2 and #2-3; and #3 may include five child nodes #3-1, #3-2, #3-3, #3-4 and #3-5, among which #3-1 may include two child nodes #3-1-1 and #3-1-2; #3-2 may include two child nodes #3-2-1 and #3-2-2; #3-3 may include two child nodes #3-3-1 and #3-3-2; #3-4 may include two child nodes #3-4-1 and #3-4-2; and #3-5 may include two child nodes #3-5-1 and #3-5-2. For example, Figure 5 The corresponding schematic diagram of the second-level structure information is as follows Figure 6 shown. Figure 6 The tree structure clearly shows the current website Figure 5 The relative relationship between the nodes in the web page shown.

[0236] For example, Figure 6 The judgment flow diagram of each node in the second level structure information shown in FIG. Figure 7 As shown. Figure 7 In the example, the judgment of any node, that is, the judgment of any node containing block elements, is divided into three steps: whether the position is close to the middle, whether the visible area exceeds the threshold, and whether the current node has child nodes. The following will be explained in detail.

[0237] exist Figure 7 The first operation to be performed is: input the root node and get the current node.

[0238] Secondly, it is determined that the data type is a block element. Exemplarily, the data type may represent the data type contained in the current node.

[0239] If the data type in the current node is not a block element, the processing of the current node is terminated. This step can exclude most of the cluttered items on the web page, such as pictures, separators, anchor tags, styles, scripts, etc.

[0240] If the data type is a block element, determine whether the position is close to the middle.

[0241] Exemplarily, it can be determined whether the position of the block element contained in the current node is close to the middle area of ​​the page. Exemplarily, assuming that the coordinates of the four vertices of the block element are (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4), the visible horizontal axis range of the entire page is X, and the visible vertical axis range is Y, then the threshold is set to R1=20%, that is, (-10%, 10%), and all the above coordinate points are traversed respectively, for example, Xn, if R1*X<=Xn<=(1-R1)*X, it means that Xn is located in the middle part of the visible page, and other coordinate points are traversed similarly; if the position of the node where the block element is located is not close to the middle area of ​​the page, the current node processing is terminated.

[0242] Exemplarily, if the location of the node where the block element is located is close to the middle area of ​​the page, it is determined whether the visible area exceeds the threshold. Exemplarily, the visible area can represent the area of ​​the block element contained in the current node displayed on the page. The visible area can be a manifestation of the first area information. Accordingly, it can be determined whether the area of ​​the node where the block element is located on the page exceeds the threshold. Assuming that the visible area of ​​the entire page is A, for example, when the browser resolution is adjusted to a conventional resolution such as 1024*800, the real-time visible area of ​​the page in the browser is not the entire area after scrolling up and down. The threshold R2 is set to 30% of the entire page area. The area of ​​the current node is An. If An>=A*R2, it indicates that the visible area of ​​the current node exceeds the threshold; otherwise, the current node processing is terminated.

[0243] Determine whether the visible area of ​​the block element exceeds a threshold. Exemplarily, if the visible area exceeds the threshold, determine whether the current node has a child node, if so, obtain the level information of the current node; if not, end the processing of the current node.

[0244] After the current node is processed, all child nodes of the node are traversed according to the above process. After all child nodes of the node are traversed, other nodes in the second hierarchical structure information can be traversed in the above manner, so as to obtain the hierarchical information of each child node in the current node that meets the specified conditions.

[0245] Step 402: Determine first hierarchical structure information based on the hierarchical information of the page relative to the website homepage.

[0246] Exemplarily, the hierarchical information of any page or any node in the second hierarchical structure information is statistically summarized, and any node is organized and managed according to the hierarchical information, so that the first hierarchical structure information can be determined.

[0247] Exemplarily, each node of the current website is traversed, so that the first level structure information of the tree structure can be obtained.

[0248] In one implementation, the first hierarchical structure information includes the hierarchical structure of any first node and may also include the first node itself.

[0249] Exemplarily, step 402 may also be implemented in the following manner:

[0250] When the address information included in the page matches the address information of the current website, the first hierarchical structure information is determined based on the hierarchical information of the page relative to the homepage of the website.

[0251] In one implementation, the match between the address information contained in the page and the address information of the current website may be determined by:

[0252] Get the link of the page to which any node of the page jumps after being selected, and the number of the links may be M. Exemplarily, the links may be divided into several forms such as complete links, relative links, and invalid links. Exemplarily, relative links usually take the form of / / path, and for relative links, the domain name of the current website may be added thereto to obtain the corresponding complete link.

[0253] Exemplarily, after obtaining each link, each link can be analyzed, and the domain name contained in each link can be compared with the domain name of the website homepage. If the two match, the connection is placed in the link pool P2, wherein the link pool P2 can correspond to the second hierarchical structure information, that is, the second hierarchical structure information is generated based on the link pool P2; conversely, if the two do not match, the link is discarded, and the matching judgment operation of the next link is continued, so as to traverse each node in the current website, thereby obtaining the link pool P2 containing each node of the current website, and generating the corresponding second hierarchical structure information.

[0254] Exemplarily, each link in the link pool P2 is continuously obtained and rendered, and the following is executed according to the rendering result: Figure 7 The process shown will satisfy Figure 7The link corresponding to the rendering result of the process shown is put into the link pool P1. Each link in the link pool P2 is traversed in the above manner to obtain a link pool including all the links corresponding to the block nodes in the current website that meet the specified conditions. Exemplarily, the link pool can be P1, and the first hierarchical structure information can be generated based on the link pool P1.

[0255] In this way, the page recognition method provided in the embodiment of the present application, on the premise of obtaining the second-level structure information containing each node in the current website, traverses each node in the second-level structure information according to the specified conditions based on the visual information after rendering of each node in the second-level structure information, thereby eliminating useless nodes, effectively reducing the amount of calculation generated by subsequent node type judgment, and speeding up the speed of node type recognition.

[0256] Step 403: Obtain first area information of the first node.

[0257] The first node refers to any node in the at least one node; the first region information refers to region information of a block element included in the first node when the page where the first node is located is displayed.

[0258] Step 404: Determine the type of the page where the first node is located based on the first hierarchical structure information and the first region information.

[0259] For example, Figure 8 A schematic diagram of the overall process of identifying the type of page where the first node is located in an embodiment of the present application.

[0260] Step 801, recursively start from the root node and remove irrelevant nodes to obtain the first level structure information.

[0261] For example, recursively starting from the root node downward, removing irrelevant nodes, and obtaining the first-level structure information can be performed by Figure 7 This is achieved by using the flowchart shown in FIG.

[0262] Step 802: recursively start from the terminal structure of the first-level structural information and calculate the memory amount of the first node.

[0263] Exemplarily, the memory amount of the first node can be obtained by the memory forgetting algorithm represented by equations (1) to (5):

[0264] Exemplarily, recursively starting from the terminal structure of the first-level structural information, the memory amount of the first node is calculated, which can be obtained by obtaining each parameter in formula (1)-formula (5) and calculating each parameter through formula (1)-formula (5).

[0265] Step 803: Identify the type of the page where the first node is located according to the memory amount.

[0266] Exemplarily, the type of the page where the first node is located can be identified based on the relationship between the memory amount and the preset memory amount threshold. When the memory amount is greater than the preset memory amount threshold, it can be determined that the page where the first node is located is a channel page; otherwise, it is not a channel page.

[0267] From the above, it can be seen that the page recognition method provided in the embodiment of the present application, after obtaining the second-level structure information including the hierarchical information of each node in the current website, judges the block elements in the intuitive page visual effect after any page is loaded and displayed according to specified conditions, thereby eliminating useless nodes in the second-level structure information. On this basis, the judgment of the node type in the current website based on the first-level structure information obtained based on the judgment result is more efficient, and the accuracy of the judgment based on the rendered page visual effect is also higher.

[0268] Based on the above embodiments, the present application embodiment provides a page recognition device 9, such as Fig. 9 As shown, the page identification device 9 includes an acquisition module 901 and a processing module 902, wherein:

[0269] Acquisition module 901 is used to acquire first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website, and first area information of the first node; wherein the first hierarchical structure information includes at least one node in any page; at least one node represents a node of a block element that meets specified conditions contained in any page; the first node is any node in the at least one node; the first area information is the area information of the block element contained in the first node when the page where the first node is located is displayed.

[0270] The processing module 902 is used to identify the type of the page where the first node is located based on the first hierarchical structure information and the first region information.

[0271] In some embodiments, the acquisition module 901 is used to obtain first quantity information from the first hierarchical structure information; wherein the first quantity information represents the quantity information of the second node; the second node includes the child node of the first node; the second node is a node in the page corresponding to the link of the first node.

[0272] Processing module 902 is used to determine characteristic parameters of the first node based on the first quantity information and the first area information; wherein the characteristic parameters of the first node are used to determine the type of the first node; and based on the characteristic parameters of the first node, identify the type of the page where the first node is located.

[0273] In some implementations, the acquisition module 901 is used to acquire second region information; wherein the second region information is region information of block elements included in the second node when the page where the second node is located is displayed.

[0274] The processing module 902 is used to determine the initial characteristic parameters of the first node based on the second area information, the first quantity information and the first area information; and determine the characteristic parameters of the first node based on the initial characteristic parameters of the first node.

[0275] In some embodiments, the acquisition module 901 is used to obtain third area information and second quantity information; wherein the third area information is the area information corresponding to the block elements contained in the third node when the page where the third node is located is displayed; the second quantity information is the number of the third nodes; the third node is a child node of the second node; the third node is a node in the link page corresponding to the second node.

[0276] The processing module 902 is used to obtain a characteristic parameter of the second node based on the third area information, the second quantity information and the second area information.

[0277] The processing module 902 is used to obtain a reference characteristic value based on the characteristic parameter of the second node and the first quantity information.

[0278] The processing module 902 is further used to process the first region information and the reference characteristic value to determine the initial characteristic parameter of the first node.

[0279] In some implementations, the processing module 902 is used to determine a correction parameter based on the second region information and the first quantity information; based on the correction parameter, correct the initial characteristic parameter of the first node to determine the characteristic parameter of the first node.

[0280] In some embodiments, the processing module 902 is used to obtain the standard deviation of the second area information based on the first quantity information; determine the correction parameter based on the standard deviation; the standard deviation of the second area information is the standard deviation of the area information included in each second area information.

[0281] In some embodiments, the processing module 902 is used to load and display the page of the current website, and when any node of the page includes a block element, if the block element meets specified conditions, obtain the hierarchical information of the page relative to the homepage of the website; wherein the hierarchical information of the page relative to the homepage of the website includes any node of the page.

[0282] Based on the hierarchical information of the page relative to the homepage of the website, the first hierarchical structure information is determined.

[0283] In some implementations, the processing module 902 is used to determine the first hierarchical structure information based on the hierarchical information of the page relative to the homepage of the website when the address information corresponding to the page matches the address information of the current website.

[0284] In some implementations, the block element satisfies a specified condition, including: position information of the region information of the block element matches the specified position information, and area information of the region information of the block element satisfies a specified area range.

[0285] The acquisition module 901 and the processing module 902 may be at least one of an application-specific integrated circuit ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present application does not specifically limit it.

[0286] As can be seen from the above, the page identification device 9 provided in the embodiment of the present application can obtain the first hierarchical information including the hierarchical information of at least one node relative to the current website homepage, and can also obtain the area information of the first node when the page where the first node is located is displayed, and then determine the type of the page where the first node is located based on the first hierarchical information and the first area information. Therefore, the page identification device 9 provided in the embodiment of the present application improves the problem of inefficiency caused by manually setting the page identification conditions, and does not need to rely on massive sample data, reducing the amount of calculation, and the operation of determining the category of the page when the page is displayed also improves the accuracy of page feature recognition, thereby achieving efficient and accurate determination of the page type.

[0287] Based on the foregoing embodiments, an embodiment of the present application provides a page identification device 10. Fig.10 The schematic diagram of the structure of the page recognition device 10 provided in the embodiment of the present application is as follows. The page recognition device 10 comprises the page recognition device 9 as described in the above embodiment.

[0288] Based on the foregoing embodiments, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the page recognition method as described in any of the foregoing embodiments is implemented.

[0289] In some embodiments, the functions or modules included in the device or system provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0290] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0291] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0292] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0293] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0294] It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0295] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0296] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0297] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0298] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0299] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0300] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0301] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A page recognition method, characterized in that: The method comprises: Acquire first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website; wherein the first hierarchical structure information includes at least one node in any page; the at least one node is a block element node contained in any page and meeting a specified condition; Acquire first region information of a first node; wherein the first node is any node of the at least one node; the first region information is region information corresponding to a block element contained in the first node when a page where the first node is located is displayed; Based on the first hierarchical structure information and the first region information, the type of the page where the first node is located is identified.

2. The method according to claim 1, characterized in that The identifying, based on the first hierarchical structure information and the first region information, the type of the page where the first node is located includes: Acquire first quantity information from the first hierarchical structure information; wherein the first quantity information is quantity information of second nodes; the second nodes include child nodes of the first nodes; the second nodes are nodes in the page corresponding to the link of the first node; Determine a characteristic parameter of the first node based on the first quantity information and the first area information; wherein the characteristic parameter of the first node is used to determine the type of the first node; Based on the characteristic parameters of the first node, the type of the page where the first node is located is identified.

3. The method according to claim 2, characterized in that The determining, based on the first quantity information and the first area information, characteristic parameters of the first node includes: Acquire second region information; wherein the second region information is region information corresponding to the block element included in the second node when the page where the second node is located is displayed; Determining initial characteristic parameters of the first node based on the second area information, the first quantity information, and the first area information; Based on the initial characteristic parameters of the first node, characteristic parameters of the first node are determined.

4. The method according to claim 3, characterized in that The determining, based on the second area information, the first quantity information and the first area information, an initial characteristic parameter of the first node includes: Acquire third area information and second quantity information; wherein the third area information is area information corresponding to the block elements contained in the third node when the page where the third node is located is displayed; the second quantity information is the number of the third nodes; the third node is a child node of the second node; the third node is a node in the link page corresponding to the second node; acquiring a characteristic parameter of the second node based on the third area information, the second quantity information, and the second area information; Obtaining a reference characteristic value based on the characteristic parameter of the second node and the first quantity information; The first region information and the benchmark feature value are processed to determine initial feature parameters of the first node.

5. The method according to claim 3, characterized in that: The determining the characteristic parameter of the first node based on the initial characteristic parameter of the first node includes: determining a correction parameter based on the second area information and the first quantity information; Based on the correction parameter, the initial characteristic parameter of the first node is corrected to determine the characteristic parameter of the first node.

6. The method according to claim 5, characterized in that The determining of the correction parameter based on the second area information and the first quantity information includes: Based on the first quantity information, obtaining a standard deviation of the second area information; wherein the standard deviation of the second area information is a standard deviation of the area information included in each piece of the second area information; Based on the standard deviation, the correction parameter is determined.

7. The method according to claim 1, characterized in that Before obtaining the first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website, the method further includes: Load and display a page of the current website, and in the case where any node of the page includes the block element, if the block element satisfies the specified condition, obtain the hierarchical information of the page relative to the homepage of the website; wherein the hierarchical information of the page relative to the homepage of the website includes any node of the page; The first hierarchical structure information is determined based on the hierarchical information of the page relative to the homepage of the website.

8. The method according to claim 7, characterized in that The determining the first hierarchical structure information based on the hierarchical information of the page relative to the homepage of the website includes: In a case where the address information included in the page matches the address information of the current website, first hierarchical structure information is determined based on the hierarchical information of the page relative to the homepage of the website.

9. The method according to any one of claims 1 to 8, characterized in that: The block element satisfies the specified condition, including: the position information of the region information of the block element matches the specified position information, and the area information of the region information of the block element satisfies the specified area range.

10. A page recognition device, characterized in that: The device comprises: an acquisition module and a processing module; wherein: The acquisition module is used to acquire first hierarchical structure information including the hierarchy of any page of the website relative to the homepage of the website, and first area information of the first node; wherein the first hierarchical structure information includes at least one node in the any page; the at least one node is a node of a block element that meets a specified condition contained in the any page; the first node is any node in the at least one node; the first area information is the area information of the block element contained in the first node when the page where the first node is located is displayed; The processing module identifies the type of the page where the first node is located based on the first hierarchical structure information and the first region information.

11. A page recognition device, characterized in that: The page recognition device comprises the page recognition apparatus as claimed in claim 10.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the page recognition method according to any one of claims 1 to 9 is implemented.

Images