A paging data collection method and system

Abstract

The present invention discloses a paging data collection method and system, including: respectively obtaining the dynamic webpage source code and the static webpage source code of the webpage page; comparing the similarity of the dynamic webpage source code and the static webpage source code to determine the dynamic and static state of the page; based on the determined dynamic and static state of the page, respectively based on the corresponding next page identification rules, obtaining all the paging addresses; updating all the paging addresses into the list page queue for data collection. Beneficial effects of the present invention: by distinguishing between dynamic pages and static pages, the complete webpage source code can be obtained more efficiently and accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a paging data acquisition method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous improvement of various network infrastructure and basic resources and the full launch of new infrastructure, various innovative services such as online news, online shopping, online office, and online government affairs have also developed vigorously, and the number of websites has continued to increase; these websites display a variety of news, information, policies, etc., and contain a lot of valuable value, such as public voice, policies, industry dynamics, economic trends, etc.

[0004] Today's network data is like a gold mine. The integration and analysis of web resources will play a more extensive guiding and decision-making role in the operation and development of enterprises and governments. In order to meet the needs of governments and enterprises for massive web data, professional distributed Internet data collection technology is needed to achieve universal web data collection.

[0005] At this stage, different websites have different styles and their development levels are also uneven, which leads to a variety of website page structures and implementation methods. The number of static web pages and dynamic web pages has a corresponding proportion. When obtaining these web page data, it is usually found that the data is not displayed on a list page, but the list page is divided into many pages and displayed in a certain order.

[0006] In the prior art, when performing data collection, the next page is often identified by manually clicking a next page button or configuring the Xpath of the next page, which consumes a lot of manual work.

[0007] In addition, early websites were usually made of static pages. The content was written when the page was generated and would not change. Crawlers could directly obtain the web page content, so the data collection methods involved were mostly designed for static web pages. The content of dynamic web pages was spliced ​​from requests from databases or interfaces. Crawlers could not directly obtain the web page content and had to imitate browser access and wait for several seconds before they could obtain the web page content. This resulted in difficulties such as high technical barriers and slow access speeds when acquiring data for dynamic web pages. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes a paging data collection method and system, which obtains the dynamic page source code and static page source code of the list page during the collection process, judges the dynamic and static nature of the page by comparing the similarity of the two page source codes, and distinguishes between dynamic and static to formulate a series of rules for identifying the next page and perform corresponding processing, and obtains all page URLs (Uniform Resource Locators) of the list page through rule matching and intelligent parsing methods.

[0009] In order to achieve the above purpose, in some embodiments, the following technical solutions are adopted:

[0010] A paging data collection method, comprising:

[0011] Respectively obtain the dynamic webpage source code and the static webpage source code of the webpage;

[0012] Compare the similarity between the source code of the dynamic web page and the source code of the static web page to determine whether the page is dynamic or static;

[0013] Based on the determined dynamic and static state of the page, all paging addresses are obtained based on the corresponding next page identification rules;

[0014] Update all paging addresses to the list page queue for data collection.

[0015] In other embodiments, the following technical solutions are adopted:

[0016] A paging data collection system, comprising:

[0017] The page dynamic and static judgment module is used to obtain the dynamic webpage source code and the static webpage source code of the webpage respectively; compare the similarity between the dynamic webpage source code and the static webpage source code to judge the dynamic and static of the page;

[0018] A paging address recognition module is used to obtain all paging addresses based on the determined dynamic and static state of the page and the corresponding next page recognition rules;

[0019] The data collection module is used to update all paging addresses into the list page queue for data collection.

[0020] In other embodiments, the following technical solutions are adopted:

[0021] A terminal device comprises a processor and a memory, wherein the processor is used to implement various instructions; the memory is used to store a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executing the above-mentioned paging data acquisition method.

[0022] In other embodiments, the following technical solutions are adopted:

[0023] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned paging data acquisition method.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The present invention can obtain the complete web page source code more efficiently and accurately by distinguishing between dynamic pages and static pages. If the static page method is used to obtain the dynamic web page source code, many database and interface interaction data will not be obtained, resulting in the inability to obtain the complete web page source code for the dynamic page. On the contrary, if the dynamic page method is used to obtain the static web page source code, the request time will be lengthened, resulting in a decrease in the overall efficiency of the crawler.

[0026] (2) The paging data collection method of the present invention can obtain all page URLs of the list page through rule matching and intelligent parsing methods, thereby facilitating the collection of paging data, reducing workload, improving data collection efficiency, and being able to collect more complete data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a paging data collection method in an embodiment of the present invention;

[0028] Figure 2 This is the next page identification process of a dynamic web page in an embodiment of the present invention;

[0029] Figure 3 This is the next page identification process of a static web page in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] Embodiment 1

[0033] First, the static web pages and dynamic web pages involved in this embodiment are described:

[0034] For static web pages, the content is already written when the page is generated and will not change, so crawlers can directly obtain the web page content; when accessing a static page in a browser, the content remains completely unchanged; static web pages do not need to connect to a database during opening, so the page browsing speed is fast and the opening speed is faster than dynamic pages.

[0035] Dynamic web pages are spliced ​​together from databases or interface requests. Crawlers cannot directly obtain web page content and need to imitate browser access and wait for several seconds before obtaining web page content. Dynamic pages contain interactive behaviors such as database requests, program requests, and interface requests. Therefore, various interactive actions must be performed during the opening of dynamic web pages, resulting in slow page loading. This results in difficulties such as high technical barriers and slow access speeds when acquiring data for dynamic web pages.

[0036] Based on the above technical problems, in one or more implementations, a paging data collection method is disclosed, such as Figure 1 As shown, the specific steps include:

[0037] Step S101: respectively obtaining a dynamic webpage source code and a static webpage source code of a webpage;

[0038] In this embodiment, the webpage source code of the dynamic webpage is obtained by rendering the dynamic webpage through the Handless Chromedriver in the Selenium tool; the specific process is as follows:

[0039] (1) Initialize the Handless Chromedriver in the Selenium tool;

[0040] (2) Access the web URL through Chromedriver simulated browser;

[0041] (3) The program sleeps for a set time, waiting for the web page to load;

[0042] (4) The web page content is loaded through Chromedriver, and the dynamic source code of the web page is obtained.

[0043] In this embodiment, the webpage source code of the static page is obtained by requesting the page URL through HttpClient; the specific process is as follows:

[0044] (1) Initialize the HttpClient object and the HttpGet object;

[0045] (2) Request the page URL via HttpGet;

[0046] (3) Parse the HttpEntity object returned by the HttpGet request page to obtain the static source code of the web page.

[0047] In this embodiment, after determining whether a page is static or dynamic, the web page source code of the following page number can be accurately obtained based on whether it is dynamic or static, without having to obtain both dynamic and static web page source codes.

[0048] Step S102: comparing the similarity between the dynamic web page source code and the static web page source code to determine whether the page is dynamic or static;

[0049] Specifically, the method for comparing the similarity between the above-mentioned dynamic web page source code and the static web page source code is as follows:

[0050] (1) Delete stop words and space characters in both source codes;

[0051] (2) Use the k-shingle algorithm to split the dynamic source code and the static source code into word sets respectively; put the word sets split from the dynamic source code into Set1 for deduplication, and obtain the word set seta; split the static source code into word sets, and put them into Set2 for deduplication, and obtain the word set setb;

[0052] (3) Put the word set seta and the word set setb into set3 to remove duplicates, and obtain the word set setc;

[0053] (4) The number of identical words in the dynamic source code and the static source code is the sum of the lengths of the word sets seta and setb minus the length of the word set setc. The similarity value of the two source codes is the ratio of the number of identical words to the length of the word set setc.

[0054] After obtaining the similarity value t between the two, make a judgment:

[0055] If the similarity satisfies t>a, the webpage is considered to be a static page;

[0056] If the similarity satisfies b<t<a, then the webpage is considered to be a dynamic page;

[0057] If the similarity satisfies t<b, the character lengths of the dynamic webpage source code and the static webpage source code are determined. If only the length of the dynamic webpage source code is less than L, the webpage is determined to be a static webpage; if only the length of the static webpage source code is less than L, the webpage is determined to be a dynamic webpage; if both lengths are less than L, the page is determined to be an abnormal page. Among them, a, b, and L are all set values.

[0058] As an optional implementation, if the similarity value is greater than 90%, the web page is considered to be a static page; if the similarity is greater than 20% and less than 90%, the web page is considered to be a dynamic page; when the similarity is less than 20%, the character length of the dynamic web page source code or the static web page source code is judged, and if the length is less than 50, it is considered that the acquisition of such web page source code has failed. For example, if the length of the dynamic web page source code is less than 50, the web page is determined to be a static page; if the length of the static web page source code is less than 50, the web page is determined to be a dynamic page; if the length of both the dynamic web page source code and the static web page source code is less than 50, the page is added to the abnormal page queue without performing paging identification and the following parsing.

[0059] Step S103: based on the determined dynamic and static status of the page, all paging addresses are obtained based on the corresponding next page identification rules;

[0060] Specifically, combined Figure 2 If it is determined to be a dynamic page, the identification rules for the next page include:

[0061] If the text of the button label or the a label is the next page or text or characters related to it (such as: "next page" or ">", etc.), simulate clicking the corresponding button or a label or ">" symbol to get the URL of the next page;

[0062] If the title attribute of the button tag or a tag contains the next page or text related to it (for example: "next page"), simulate clicking the corresponding button or a tag or the ">" symbol to get the URL of the next page;

[0063] If it is a digital page turn (such as 1, 2, 3... the last page), the URL string of each a tag in the page is parsed and compared with the URL of the current page. If the strings are the same in a set proportion (such as 95% of the strings are the same), it is considered to be a pagination URL. The last page is identified to obtain the total number of pages, and the same part of the URL is extracted. The page numbers are concatenated according to the total number of pages to obtain all the pagination URLs.

[0064] Combination Figure 3 If it is determined to be a static page, the identification rules for the next page include:

[0065] If the text of the a tag is the next page or text or characters related to it (such as "next page" or ">", then parse the href attribute of the corresponding tag to get the URL of the next page;

[0066] If the title attribute of the a tag contains the next page or text related to it (for example: "next page"), the href attribute of the corresponding tag is parsed to obtain the URL of the next page;

[0067] If it is a digital page turn, the URL string of each a tag in the page is parsed and compared with the URL of the current page. If the strings meet the set ratio (for example, 95% of the strings are the same), it is considered to be a pagination URL; the last page is identified to obtain the total number of pages, and the same part of the URL is extracted. The previous page number is concatenated according to the total number of pages to obtain all the pagination URLs.

[0068] In the above process, if a page URL obtained contains a set script character (such as a "JavaScript" character), the page is defined as a dynamic page and re-identified according to the dynamic page identification rules.

[0069] Step S104: update all paging addresses into the list page queue for data collection.

[0070] It should be noted that the processes of obtaining web page source code and parsing, extracting or collecting page data involved in this embodiment are all legal applications of data in compliance with laws and regulations and with the consent of users.

[0071] It should also be noted that the a tag, button tag, JavaScript characters, etc. involved in this embodiment are all technical terms well known to those skilled in the art when performing programming, and will not be described in detail in this embodiment.

[0072] Embodiment 2

[0073] In one or more embodiments, a paging data collection system is disclosed, comprising:

[0074] The page dynamic and static judgment module is used to obtain the dynamic webpage source code and the static webpage source code of the webpage respectively; compare the similarity between the dynamic webpage source code and the static webpage source code to judge the dynamic and static of the page;

[0075] A paging address recognition module is used to obtain all paging addresses based on the determined dynamic and static state of the page and the corresponding next page recognition rules;

[0076] The data collection module is used to update all paging addresses into the list page queue for data collection.

[0077] The paging data collection system of this embodiment can be deployed in a distributed manner on multiple nodes, and each node can work together to complete the same task.

[0078] It should be noted that the specific implementation process of the above modules has been described in detail in Example 1 and will not be repeated here.

[0079] Embodiment 3

[0080] In one or more embodiments, a terminal device is disclosed, including a server, the server including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the paging data collection method in Embodiment 1 when executing the program. For the sake of brevity, it will not be described in detail here.

[0081] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0082] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0083] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0084] Embodiment 4

[0085] In one or more embodiments, a computer-readable storage medium is disclosed, in which a plurality of instructions are stored. The instructions are suitable for being loaded by a processor of a terminal device and executing the paging data acquisition method described in the first embodiment.

[0086] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A paging data collection method, characterized in that: include: Respectively obtain the dynamic webpage source code and the static webpage source code of the webpage; Compare the similarity between the source code of the dynamic web page and the source code of the static web page to determine whether the page is dynamic or static; Based on the determined dynamic and static state of the page, all paging addresses are obtained based on the corresponding next page identification rules; Specifically, if it is determined to be a static page, the identification rules for the next page include: if the text of the a tag is the next page or text or characters related to it, then parse the href attribute of the corresponding tag to obtain the URL of the next page; if the title attribute of the a tag contains the next page or text related to it, then parse the href attribute of the corresponding tag to obtain the URL of the next page; if it is a digital page turn, then parse the URL string of each a tag in the page and compare it with the URL of the current page. If the string is the same as the set ratio, it is considered to be a paging URL; identify the last page to obtain the total number of pages, extract the same part of the URL, and cyclically concatenate the previous page numbers according to the total number of pages to obtain all the paging URLs; if the URL of a certain page contains the set script characters, then define the page as a dynamic page and re-identify it according to the identification rules of dynamic pages; If it is determined to be a dynamic page, the identification rules for the next page include: If the text of the button label or a label is the next page or text or characters related to it, simulate clicking the corresponding text or characters to get the URL of the next page; if the title attribute of the button label or a label contains the next page or text related to it, simulate clicking the corresponding text or characters to get the URL of the next page; if it is a digital page turn, parse the URL string of each a label in the page and compare it with the URL of the current page. If the string is the same as the set ratio, it is considered to be a paging URL; identify the last page to get the total number of pages, extract the same part of the URL, and cyclically splice the previous page number according to the total number of pages to get all the paging URLs; Update all paging addresses to the list page queue for data collection.

2. A paging data collection method as claimed in claim 1, characterized in that: Access the web page URL by simulating a browser to obtain the dynamic web page source code of the web page; By requesting the page URL and parsing the returned object, the static web page source code of the web page is obtained.

3. A paging data collection method as claimed in claim 1, characterized in that: Compare the similarities between the above dynamic web page source code and the static web page source code, including: Deleting stop words and space characters in the dynamic web page source code and the static web page source code; The dynamic source code is divided into word sets, and put into Set1 for deduplication, to obtain the word set seta; the static source code is divided into word sets, and put into Set2 for deduplication, to obtain the word set setb; Put the word set seta and the word set setb into set3 to remove duplicates, and get the word set setc; The number of identical words in the dynamic source code and the static source code is the sum of the lengths of the word sets seta and setb minus the length of the word set setc, and the similarity value of the two source codes is the ratio of the number of identical words to the length of the word set setc.

4. A paging data collection method as claimed in claim 1, characterized in that: The process of determining whether a page is dynamic or static specifically includes: If the similarity satisfies t>a, the webpage is considered to be a static page; If the similarity satisfies b<t<a, then the webpage is considered to be a dynamic page; If the similarity satisfies t<b, the character lengths of the dynamic web page source code and the static web page source code are determined. If only the length of the dynamic web page source code is less than L, the web page is determined to be a static web page; if only the length of the static web page source code is less than L, the web page is determined to be a dynamic web page; if both lengths are less than L, the page is determined to be an abnormal page.

5. A paging data collection system, characterized in that: include: A page dynamic and static judgment module is used to obtain the dynamic web page source code and the static web page source code of the web page respectively; Compare the similarity between the source code of the dynamic web page and the source code of the static web page to determine whether the page is dynamic or static; A paging address recognition module is used to obtain all paging addresses based on the determined dynamic and static state of the page and the corresponding next page recognition rules; Specifically, if it is determined to be a static page, the identification rules for the next page include: if the text of the a tag is the next page or text or characters related to it, then parse the href attribute of the corresponding tag to obtain the URL of the next page; if the title attribute of the a tag contains the next page or text related to it, then parse the href attribute of the corresponding tag to obtain the URL of the next page; if it is a digital page turn, then parse the URL string of each a tag in the page and compare it with the URL of the current page. If the string is the same as the set ratio, it is considered to be a paging URL; identify the last page to obtain the total number of pages, extract the same part of the URL, and cyclically concatenate the previous page numbers according to the total number of pages to obtain all the paging URLs; if the URL of a certain page contains the set script characters, then define the page as a dynamic page and re-identify it according to the identification rules of dynamic pages; If it is determined to be a dynamic page, the identification rules for the next page include: If the text of the button label or a label is the next page or text or characters related to it, simulate clicking the corresponding text or characters to get the URL of the next page; if the title attribute of the button label or a label contains the next page or text related to it, simulate clicking the corresponding text or characters to get the URL of the next page; if it is a digital page turn, parse the URL string of each a label in the page and compare it with the URL of the current page. If the string is the same as the set ratio, it is considered to be a paging URL; identify the last page to get the total number of pages, extract the same part of the URL, and cyclically splice the previous page number according to the total number of pages to get all the paging URLs; The data collection module is used to update all paging addresses into the list page queue for data collection.

6. A terminal device, comprising a processor and a memory, wherein the processor is used to implement each instruction; and the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the paging data acquisition method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instruction is suitable for being loaded by a processor of a terminal device and executing the paging data acquisition method according to any one of claims 1 to 4.

Images