A web page information processing method, system, electronic device and storage medium

Abstract

The embodiment of the present application provides a web page information processing method, system, electronic device and storage medium, in the field of information processing technology. The web page information processing method includes: obtaining weight data of each web page in multiple web pages and weight data of the root page; traversing each web page in multiple web pages, and performing the following processing on the weight data of each traversed web page: taking the currently traversed web page as the target web page, obtaining the link data of the target web page, the link data includes the external link web page linked out by the target web page; assigning the weight data of the target web page to the external link web page or the root web page according to the link data; clearing the weight data of the target web page; assigning the weight data of the root page to each web page in multiple web pages; clearing the weight data of the root page; crawling with a web crawler according to the weight data of the assigned web page. The web page information processing method can achieve the technical effect of improving the scoring stability of the web page system.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a web page information processing method, system, electronic device and storage medium. Background Art

[0002] At present, with the rapid development of the Internet and the popularization of personal computers, how to quickly and accurately retrieve the information you need from the massive information database is an important problem faced by the majority of netizens. Search engines have become widely used information retrieval tools. With the growth of information volume, various duplicate, invalid, and false web pages are flooding the Internet. Therefore, it is very important to filter out junk web pages and judge the importance of pages, especially for web crawlers that search engines rely on.

[0003] In the prior art, web crawlers generally crawl web pages through the PageRank algorithm or the Online Page Importance Computation (OPIC) algorithm. However, the PageRank algorithm performs iterative calculations on all downloaded web pages, and each calculation consumes a lot of resources and requires offline calculations. It is not suitable for online use, and new web pages have relatively few imported links, which is unfair to new web pages. The OPIC algorithm determines the importance of a web page, i.e., the web page score, based on the amount of cash the web page link has. After multiple crawls through the OPIC algorithm, the score calculated by cash will also decrease, resulting in the quality of the web page score being much lower than the score quality of the initial system, causing instability in the scoring system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a web page information processing method, system, electronic device and storage medium, which can achieve the technical effect of improving the scoring stability of the web page system.

[0005] In a first aspect, an embodiment of the present application provides a web page information processing method, which is applied to a web page system, wherein the web page system includes a plurality of web pages and a root web page, and the method includes:

[0006] Obtaining weight data of each web page in the plurality of web pages and weight data of the root web page;

[0007] Each web page in the plurality of web pages is traversed, and the following processing is performed on the weight data of each traversed web page:

[0008] Taking the currently traversed web page as the target web page, obtaining the link data of the target web page, wherein the link data includes the external link web pages linked to by the target web page;

[0009] Allocating the weight data of the target web page to the external link web page or the root web page according to the outbound link data;

[0010] Clearing the weight data of the target web page;

[0011] assigning the weight data of the root web page to each web page of the plurality of web pages;

[0012] Clearing the weight data of the root web page;

[0013] The web crawler is used to crawl the web page according to the weight data of the web page after the allocation.

[0014] In the above implementation process, the web page information processing method adds a root web page connected to all web pages in the web page system, so that when encountering a leaf node (the web page has no external links to other web pages), the weight data that should have disappeared can flow into the root web page, so that the weight data owned by the entire web page system remains constant, ensuring that the subsequent proportion of web page scoring is the same as the initial web page scoring method (the overall score of the web page system will not be reduced due to the reduction of weight data); thereby, the web page information processing method can achieve the technical effect of improving the scoring stability of the web page system.

[0015] Furthermore, the step of allocating the weight data of the target web page to the external link web page or the root web page according to the outbound link data includes:

[0016] Determining whether the number of the external link web pages is zero;

[0017] If the number of the external link web pages is zero, the weight data of the target web page is assigned to the root web page;

[0018] If the number of the external link web pages is not zero, the weight data of the target web page is allocated to the external link web pages according to the outbound link data.

[0019] In the above implementation process, if the number of external link web pages of the target web page is zero, it means that the target web page is a leaf node, and its weight data needs to be recovered to the root web page; if the number of external link web pages of the target web page is not zero, its weight data is allocated to the external link web pages to complete this round of allocation process.

[0020] Furthermore, the step of allocating the weight data of the target web page to the external link web page according to the outbound link data includes:

[0021] The weight data of the target web page is evenly distributed to the external link web pages according to the outbound link data.

[0022] Furthermore, the step of allocating the weight data of the root web page to each of the plurality of web pages comprises:

[0023] The weight data of the root web page is evenly distributed to each web page in the plurality of web pages.

[0024] In the above implementation process, the weight data of the root web page is evenly distributed to all web pages in the current web page system.

[0025] Furthermore, the step of allocating the weight data of the root web page to each of the plurality of web pages comprises:

[0026] Obtaining historical total weight data of the plurality of web pages;

[0027] Obtaining a ratio parameter between the historical weight data of each web page in the plurality of web pages and the historical total weight data;

[0028] The weight data of the root web page is distributed to each of the plurality of web pages according to the ratio parameter and a preset distribution formula.

[0029] Furthermore, the preset allocation formula is:

[0030]

[0031] Among them, current_cash is the weight data assigned to the web page, R_cash is the weight data of the root web page, current_score is the historical weight data of the web page, total_score is the historical total weight data, and the ratio of current_score and total_score is the ratio parameter.

[0032] Furthermore, the preset allocation formula is:

[0033]

[0034] Wherein, current_cash is the weight data assigned to the web page, R_cash is the weight data of the root web page, count is the total number of the plurality of web pages, current_score is the historical weight data of the web page, and total_score is the historical total weight data. is the ratio parameter.

[0035] In a second aspect, an embodiment of the present application provides a web page information processing system, which is applied to a web page system, wherein the web page system includes a plurality of web pages and a root web page, and the web page information processing system includes:

[0036] An acquisition module, used to acquire weight data of each web page in the plurality of web pages and weight data of the root web page;

[0037] The traversal module is used to traverse each of the multiple web pages, and perform the following processing on the weight data of each traversed web page: take the currently traversed web page as the target web page, obtain the link data of the target web page, and the link data includes the external link web pages linked from the target web page; assign the weight data of the target web page to the external link web page or the root web page according to the link data; clear the weight data of the target web page; assign the weight data of the root web page to each of the multiple web pages; and clear the weight data of the root web page;

[0038] The web crawler module is used to perform web crawling according to the weight data of the web pages after allocation.

[0039] Furthermore, the traversal module includes:

[0040] A judging unit, judging whether the number of the external link web pages is zero;

[0041] A first allocating unit, configured to allocate the weight data of the target web page to the root web page if the number of the external link web pages is zero;

[0042] The second allocating unit is used to allocate the weight data of the target web page to the external link web page according to the outbound link data if the number of the external link web pages is not zero.

[0043] Furthermore, the first allocating unit is specifically configured to evenly allocate the weight data of the target web page to the external link web pages according to the outbound link data.

[0044] Furthermore, the traversal module also includes:

[0045] The first root web page distribution unit is used to evenly distribute the weight data of the root web page to each web page among the plurality of web pages.

[0046] Furthermore, the traversal module also includes:

[0047] A total weight acquisition unit, used to acquire historical total weight data of the plurality of web pages;

[0048] a ratio acquisition unit, used to acquire a ratio parameter between the historical weight data of each web page in the plurality of web pages and the historical total weight data;

[0049] The second root web page allocation unit is used to allocate the weight data of the root web page to each web page of the plurality of web pages according to the ratio parameter and a preset allocation formula.

[0050] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0053] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0054] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of the structure of a web page system provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of a flow chart of a web page information processing method provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of a process for allocating weight data of a target web page provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of a process for allocating weight data of a root web page provided in an embodiment of the present application;

[0060] Figure 5 A structural block diagram of a web page information processing system provided in an embodiment of the present application;

[0061] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0063] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0064] The embodiments of the present application provide a web page information processing method, system, electronic device and storage medium, which can be applied to the information retrieval process of the network, such as the process of crawling web pages by a web crawler; the web page information processing method adds a root web page connected to all web pages in the web page system, so that when encountering a leaf node (a web page with no external links to other web pages), the weight data that should have disappeared can flow into the root web page, so that the weight data owned by the entire web page system remains constant, ensuring that the proportion of subsequent web page scoring is the same as the initial web page scoring method (the overall score of the web page system will not be reduced due to the reduction of weight data); thus, the web page information processing method can achieve the technical effect of improving the scoring stability of the web page system.

[0065] See also Figure 1 , Figure 1 A structural diagram of a web page system provided for an embodiment of the present application, wherein the web page system includes multiple web pages and a root web page; as an example, the web page system includes web page A, web page B, web page C, and web page D, and the root web page is web page R; web page D has no external linked web pages except the root web page, that is, it is a leaf node.

[0066] Exemplarily, the number of web pages in the web page system is only an example and not a limitation, and the specific number can be modified according to actual needs.

[0067] For example, Figure 1As shown in the figure, the traditional OPIC algorithm does not include the root web page R. The allocation principle of the traditional OPIC algorithm is specifically described as follows: Web page A, Web page B, Web page C, and Web page D represent four web pages. The direction of the solid arrow represents that the web page has a URL that links to other web pages, that is, external links (outbound data). For example, the external link of web page A is web page B, that is, web page B is an external link web page linked out by web page A; similarly, web page B has two external links, namely web page A and web page C, that is, web page A and web page C are external link web pages linked out by web page B; web page C has three external links, namely web page A, web page B, web page D, and so on.

[0068] In the traditional OPIC algorithm allocation scheme, the initial cash value (weight data) of each web page is set to 1, and the total cash value of the current web page system is 4. During each round of allocation, the current web page will divide the cash value it has equally among its external link web pages; after the allocation is completed, the cash value of the current web page is reset to zero, and each web page will add up the cash value allocated to its page in this round to get a new cash value. Therefore, the score value is calculated based on the cash value, and the importance of this batch of web pages is determined based on the score, and the order of crawling is determined. In other words, the OPIC algorithm will convert the cash owned into web page scores, and web page links with larger scores will be downloaded first.

[0069] In some embodiments, according to the allocation scheme of the above-mentioned traditional OPIC algorithm, Figure 1 The specific allocation process of the example is as follows: Web page C has 3 external links, that is, the cash of Web page C = 1, which is divided into three parts, and 1 / 3 cash is given to each external linking web page; Web page A obtains 1 / 3 of the cash from Web page C and 1 / 2 from Web page B. After one round, the cash value of Web page A is 5 / 6;

[0070] Thus, after one round of allocation, the cash value of webpage A is 5 / 6, the cash value of webpage B is 4 / 3, the cash value of webpage C is 1 / 2, and the cash value of webpage D is 1 / 3. Since webpage D has no external links, its cash value will be reset to zero after the allocation process is completed, so the total cash value in the entire webpage system is reduced by 1, and the total cash value of the current webpage system is 3. Therefore, the total cash value in the entire system will decrease after multiple rounds of cycles, resulting in the quality of webpage scores being much lower than the score quality of the initial system, causing instability in the scoring system.

[0071] See also Figure 2 , Figure 2 A schematic diagram of a web page information processing method provided in an embodiment of the present application, wherein the web page information processing method is applied to Figure 1The web page system shown includes a plurality of web pages and a root web page, and the web page information processing method includes:

[0072] S100: Obtaining weight data of each web page in a plurality of web pages and weight data of a root web page.

[0073] Exemplarily, when initializing the entire web page system, initial weight data (i.e., cash value, which will not be described below) will be configured for each web page; in each round of weight data allocation process, it is necessary to obtain the weight data (or initial weight data) of each web page allocated in the previous round, and then redistribute the weight data for this round.

[0074] In some implementations, when the web page system is initialized, the cash value assigned to the root web page is zero.

[0075] S200: traverse each web page among the multiple web pages, and perform the following processing on the weight data of each traversed web page:

[0076] S210: taking the currently traversed webpage as the target webpage, obtaining the link-out data of the target webpage, the link-out data including the external link webpages linked out by the target webpage;

[0077] S220: Allocating weight data of the target web page to external link web pages or root web pages according to the outbound link data;

[0078] S230: Clearing the weight data of the target web page.

[0079] Exemplarily, during a round of allocation, each web page will be traversed to realize the redistribution of weight data of each web page; during the traversal, if the target web page has external link web pages (except the root web page), the weight data of the target web page will be allocated to the external link web pages; if the target web page has no external link web pages (except the root web page), the weight data of the target web page will be allocated to the root web page; then the weight data of the target web page will be cleared.

[0080] It should be noted that in each round of allocation, the weight data obtained through allocation will no longer be allocated in this round to avoid repeated allocation.

[0081] S240: Distribute the weight data of the root web page to each web page among the plurality of web pages;

[0082] S250: Clear the weight data of the root web page.

[0083] Exemplarily, in each round of allocation, the weight data of the root web page is allocated to each web page, and then the weight data of the root web page is cleared to zero, thereby ensuring that the total weight data of the entire web page system remains constant.

[0084] S300: Crawling the web based on the weight data of the assigned web pages.

[0085] In some implementation scenarios, the web page information processing method adds a root web page connected to all web pages in the web page system, so that when encountering a leaf node (a web page with no external links to other web pages), the weight data that should have disappeared can flow into the root web page, so that the weight data owned by the entire web page system remains constant, ensuring that the subsequent proportion of web page scoring is the same as the initial web page scoring method (the overall score of the web page system will not be reduced due to the reduction of weight data); thereby, the web page information processing method can achieve the technical effect of improving the scoring stability of the web page system.

[0086] See also Figure 3 , Figure 3 A schematic diagram of a process for allocating weight data of a target web page provided in an embodiment of the present application.

[0087] Exemplarily, S220: the step of allocating the weight data of the target web page to the external link web page or the root web page according to the outbound link data includes:

[0088] S221: Determine whether the number of external link web pages is zero;

[0089] S222: If the number of external link web pages is zero, assign the weight data of the target web page to the root web page;

[0090] S223: If the number of external link web pages is not zero, the weight data of the target web page is distributed to the external link web pages according to the outbound link data.

[0091] Exemplarily, if the number of external link web pages of the target web page is zero, it means that the target web page is a leaf node and its weight data needs to be recovered to the root web page; if the number of external link web pages of the target web page is not zero, its weight data is distributed to the external link web pages to complete this round of distribution process.

[0092] Optionally, S223: the step of allocating the weight data of the target web page to the external link web page according to the outbound link data includes:

[0093] According to the outbound link data, the weight data of the target web page is evenly distributed to the external link web pages.

[0094] Optionally, S240: the step of distributing the weight data of the root web page to each of the plurality of web pages comprises:

[0095] The weight data of the root web page is evenly distributed to each web page among the multiple web pages.

[0096] Exemplarily, the weight data of the root web page is evenly distributed to all web pages in the current web page system.

[0097] See also Figure 4 , Figure 4 A schematic diagram of a process for allocating weight data of a root web page provided in an embodiment of the present application.

[0098] Exemplarily, S240: the step of allocating the weight data of the root web page to each of the plurality of web pages includes:

[0099] S241: Obtaining historical total weight data of multiple web pages;

[0100] S242: Obtaining a ratio parameter between the historical weight data and the historical total weight data of each web page in the plurality of web pages;

[0101] S243: Distribute the weight data of the root web page to each of the plurality of web pages according to the ratio parameter and a preset distribution formula.

[0102] Optionally, preset the allocation formula:

[0103]

[0104] Among them, current_cash is the weight data assigned to the web page, R_cash is the weight data of the root web page, current_score is the historical weight data of the web page, total_score is the historical total weight data, and the ratio of current_score and total_score is the ratio parameter.

[0105] Exemplarily, the strategy of the preset allocation formula is to give priority to updating web pages with higher weight data, while ignoring web pages with lower weight data; that is, the weight data of the root web page is mainly allocated to web pages with higher scores in the current web page system. The higher the ratio of the current web page score to the total score, the higher the weight data obtained from the root web page allocation.

[0106] Optionally, preset the allocation formula:

[0107]

[0108] Among them, current_cash is the weight data assigned to the web page, R_cash is the weight data of the root web page, count is the total number of web pages of multiple web pages, current_score is the historical weight data of the web page, and total_score is the historical total weight data. is the ratio parameter.

[0109] Exemplarily, the strategy of the preset allocation formula is to increase the proportion of weight data obtained from low-score pages (including new web pages) and maintain the balance of weight data of each web page; at this time, in order to ensure that the total amount of allocated weight data is equal to the amount of R_cash, the weight data obtained each time should be divided into count-1 parts.

[0110] In some implementation scenarios, the web page information processing method provided in the embodiments of the present application is a general crawler improved algorithm based on adaptive online web page importance calculation, which is improved on the basis of the above-mentioned traditional OPIC algorithm; the specific improvement scheme is to add a root web page (root web page) connected to all pages (web pages). When encountering a leaf node, the cash value that should have disappeared will flow into the root web page, so that the amount of cash owned by the entire web page system is constant, ensuring that the proportion of subsequent scoring is the same as the initial scoring method, and the overall score of the web page system will not be reduced due to the reduction of cash.

[0111] For example, the cash value obtained from the root webpage will be redistributed in the next round of selecting webpages to be crawled according to the specified crawling strategy to avoid the loss of cash. Figures 1 to 4 When the cash value is transferred, since web page D does not contain any externally linked web pages, a link to the root web page R will be opened, and web page D will transfer all the cash values ​​it currently has to the root web page R. This ensures that the total amount of cash in the current web page system remains the same and is constant at 4. This solves the problem of the loss of cash value in the traditional OPIC algorithm, which causes the overall web page score to drop.

[0112] For example, the cash value held by the root webpage R will select different allocation strategies according to specific business needs in the next round of crawling, as shown in the following formula:

[0113] current_cash=R_cash*N;

[0114] Among them, current_cash is the cash value allocated to the current web page; R_cash is the cash value held by the root web page in the previous round; N is the target allocation ratio. Different strategies can be selected according to business needs. The specific plan is shown in the following formula:

[0115]

[0116] Among them, count is the total number of web pages stored in the current web page system; current_score is the cash value of the current web page; total_score is the sum of the cash values ​​of all web pages in the current web page system.

[0117] Exemplarily, the first allocation strategy only keeps the cash value in the web page system constant, and all web pages have equal crawling priority; the approach is to evenly distribute the cash value of the root web page to all web pages in the current system: in this case, N takes the value of 1 / count.

[0118] Exemplarily, the second allocation strategy is to increase the probability of crawling web pages with higher scores; this strategy is to give priority to updating important web pages and ignore web pages with lower scores. The approach is to allocate the cash of node R mainly to web pages with higher scores in the current system. At this time, N is the ratio of the current web page score to the total score, that is, N = current_score / total_score.

[0119] Exemplarily, the third allocation strategy is to expand the crawling range and increase the crawling probability of new web pages or web pages with lower scores; contrary to the second allocation strategy, (1-current_score / total_score) is used to increase the proportion of cash value obtained from low scores (including new web pages). At this time, in order to ensure that the total amount of allocated cash is equal to the amount of R_cash, the cash obtained each time should be divided into count-1 parts, that is, N=(1 / (count-1))*(1-current_score / total_score).

[0120] For example, by adding a root web page to store the cash value lost in the traditional OPIC algorithm, the consistency of the system score is guaranteed, and the overall web page score will not decrease over time, ensuring the stability and reliability of the web page scoring system. At the same time, a cash allocation strategy held by the root web page is proposed, and different strategies can be selected according to the business scenario, which improves the flexibility of the crawler system.

[0121] See also Figure 5 , Figure 5 The structural block diagram of the web page information processing system provided in the embodiment of the present application is applied to a web page system, the web page system includes multiple web pages and a root web page, and the web page information processing system includes:

[0122] The acquisition module 100 is used to acquire the weight data of each web page in the plurality of web pages and the weight data of the root web page;

[0123] The traversal module 200 is used to traverse each web page in the plurality of web pages, and perform the following processing on the weight data of each traversed web page: taking the currently traversed web page as the target web page, obtaining the link-out data of the target web page, the link-out data including the external link web pages linked out by the target web page; assigning the weight data of the target web page to the external link web page or the root web page according to the link-out data; clearing the weight data of the target web page; assigning the weight data of the root web page to each web page in the plurality of web pages; and clearing the weight data of the root web page;

[0124] The web crawler module 300 is used to perform web crawling according to the weight data of the assigned web pages.

[0125] Exemplarily, the traversal module 200 includes:

[0126] A judging unit, judging whether the number of external link web pages is zero;

[0127] A first allocation unit is used to allocate the weight data of the target web page to the root web page if the number of the external link web pages is zero;

[0128] The second allocating unit is used to allocate the weight data of the target web page to the external link web page according to the outbound link data if the number of the external link web pages is not zero.

[0129] Exemplarily, the first allocating unit is specifically configured to evenly allocate the weight data of the target web page to the external link web pages according to the outbound link data.

[0130] Exemplarily, the traversal module 200 further includes:

[0131] The first root web page distribution unit is used to evenly distribute the weight data of the root web page to each web page among the multiple web pages.

[0132] Exemplarily, the traversal module 200 further includes:

[0133] A total weight acquisition unit, used to acquire historical total weight data of multiple web pages;

[0134] A ratio acquisition unit, used to acquire a ratio parameter between the historical weight data and the historical total weight data of each web page in the plurality of web pages;

[0135] The second root web page allocation unit is used to allocate the weight data of the root web page to each web page of the plurality of web pages according to the ratio parameter and a preset allocation formula.

[0136] It should be understood that Figure 5 The web page information processing system shown is Figures 1 to 4 The method embodiments shown correspond to each other and will not be described again here to avoid repetition.

[0137] This application also provides an electronic device, see Figure 6 , Figure 6 A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 510, a communication interface 520, a memory 530, and at least one communication bus 540. The communication bus 540 is used to realize direct connection and communication between these components. The communication interface 520 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 510 may be an integrated circuit chip with signal processing capabilities.

[0138] The processor 510 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 510 can also be any conventional processor, etc.

[0139] The memory 530 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 530 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 510, the electronic device can execute the above-mentioned Figures 1 to 4 The method embodiment involves various steps.

[0140] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0141] The memory 530, storage controller, processor 510, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 540. The processor 510 is used to execute executable modules stored in the memory 530, such as software function modules or computer programs included in the electronic device.

[0142] The input and output unit is used to provide users with the task creation and to create a start optional time period or preset execution time for the task to realize the interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0143] Understandably, Figure 6 The structure shown is for illustration only, and the electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown. Figure 6 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0144] An embodiment of the present application further provides a storage medium having instructions stored thereon. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described here.

[0145] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0146] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0148] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0149] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0151] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A web page information processing method, characterized in that: Applied to a web page system, the web page system includes a plurality of web pages and a root web page, the method includes: Obtaining weight data of each web page in the plurality of web pages and weight data of the root web page; Each web page in the plurality of web pages is traversed, and the following processing is performed on the weight data of each traversed web page: Taking the currently traversed web page as the target web page, obtaining the link data of the target web page, wherein the link data includes the external link web pages linked to by the target web page; Allocating the weight data of the target web page to the external link web page or the root web page according to the outbound link data; Clearing the weight data of the target web page; assigning the weight data of the root web page to each web page of the plurality of web pages; Clearing the weight data of the root web page; Performing web crawling according to the weight data of the web page after the allocation; The step of allocating the weight data of the target web page to the external link web page or the root web page according to the outbound link data includes: Determining whether the number of the external link web pages is zero; If the number of the external link web pages is zero, the weight data of the target web page is assigned to the root web page; If the number of the external link web pages is not zero, the weight data of the target web page is allocated to the external link web pages according to the outbound link data.

2. The webpage information processing method according to claim 1, characterized in that: The step of allocating the weight data of the target web page to the external link web page according to the outbound link data includes: The weight data of the target web page is evenly distributed to the external link web pages according to the outbound link data.

3. The webpage information processing method according to claim 1, characterized in that: The step of allocating the weight data of the root web page to each of the plurality of web pages comprises: The weight data of the root web page is evenly distributed to each web page in the plurality of web pages.

4. The webpage information processing method according to claim 1, characterized in that: The step of allocating the weight data of the root web page to each of the plurality of web pages comprises: Obtaining historical total weight data of the plurality of web pages; Obtaining a ratio parameter between the historical weight data of each web page in the plurality of web pages and the historical total weight data; The weight data of the root web page is distributed to each of the plurality of web pages according to the ratio parameter and a preset distribution formula.

5. The webpage information processing method according to claim 4, characterized in that: The preset allocation formula is: Among them, current_cash is the weight data assigned to the web page, R_cash is the weight data of the root web page, current_score is the historical weight data of the web page, total_score is the historical total weight data, and the ratio of current_score and total_score is the ratio parameter.

6. The webpage information processing method according to claim 4, characterized in that: The preset allocation formula is: Wherein, current_cash is the weight data assigned to the web page, R_cash is the weight data of the root web page, count is the total number of the plurality of web pages, current_score is the historical weight data of the web page, and total_score is the historical total weight data. is the ratio parameter.

7. A web page information processing system, characterized in that: Applied to a web page system, the web page system includes a plurality of web pages and a root web page, and the web page information processing system includes: An acquisition module, used to acquire weight data of each web page in the plurality of web pages and weight data of the root web page; The traversal module is used to traverse each of the multiple web pages, and perform the following processing on the weight data of each traversed web page: take the currently traversed web page as the target web page, obtain the link data of the target web page, and the link data includes the external link web pages linked from the target web page; assign the weight data of the target web page to the external link web page or the root web page according to the link data; clear the weight data of the target web page; assign the weight data of the root web page to each of the multiple web pages; and clear the weight data of the root web page; A web crawler module, used for performing web crawling according to the weight data of the web pages after the allocation; The traversal module includes: A judging unit, judging whether the number of the external link web pages is zero; A first allocating unit, configured to allocate the weight data of the target web page to the root web page if the number of the external link web pages is zero; The second allocating unit is used to allocate the weight data of the target web page to the external link web page according to the outbound link data if the number of the external link web pages is not zero.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the web page information processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is enabled to execute the web page information processing method according to any one of claims 1 to 6.

Images