A collection rate optimization analysis method and system for Internet data collection

Abstract

The present invention belongs to the field of data collection, and provides a method and system for optimizing and analyzing the collection rate of Internet data collection. The method includes obtaining all clickable column page URLs of a website; obtaining the column URLs of the column page URLs and each list URL of the list page HTML according to all clickable column page URLs of the website; obtaining the column collection rate according to the column URLs of the column page URLs; judging whether to perform paging collection rate according to each list URL of the list page HTML, if so, obtaining the paging collection rate, otherwise, obtaining the list collection rate.

Description

Technical Field

[0001] The present invention belongs to the field of data collection, and in particular relates to a collection rate optimization analysis method and system for Internet data collection. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Nowadays, with the rapid development of Internet technology, there are more and more information resources on the Internet. Researchers and enterprises tend to obtain data resources from the Internet. It is well known that data resources on the Internet basically exist in the form of web pages, and the existing web pages have column queues, and there are list pages in the corresponding column lists, and there are multiple paginated lists under the list pages.

[0004] Existing Internet data collection technologies have problems such as missing and incomplete data collection. Summary of the invention

[0005] In order to solve the problem of how to realize the intelligent configuration of web page rules, further improve the accuracy and parsing performance of the web page structure analysis model, and realize the full automation and intelligence of the collection process, the present invention provides a collection rate optimization analysis method and system for Internet data collection.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides a collection rate optimization analysis method for Internet data collection.

[0008] A collection rate optimization analysis method for Internet data collection, comprising:

[0009] Get all clickable column page URLs of the website;

[0010] According to all clickable column page URLs of the website, get the column URL of the column page URL and each list URL of the list page HTML;

[0011] According to the column URL of the column page URL, get the column collection rate;

[0012] According to the list URLs of the list page HTML, determine whether to perform paging collection. If so, get the paging collection rate; otherwise, get the list collection rate.

[0013] The second aspect of the present invention provides a collection rate optimization and analysis system for Internet data collection.

[0014] A collection rate optimization and analysis system for Internet data collection, comprising:

[0015] The column page URL acquisition module is configured to: acquire all clickable column page URLs of the website;

[0016] The column URL and list URL acquisition module is configured to: obtain the column URL of the column page URL and each list URL of the list page HTML according to all clickable column page URLs of the website;

[0017] The column collection rate optimization module is configured to: obtain the column collection rate according to the column URL of the column page URL;

[0018] The paging and list collection rate optimization module is configured as follows: according to each list URL of the list page HTML, determine whether to perform paging collection rate, if so, get the paging collection rate, otherwise, get the list collection rate.

[0019] A third aspect of the present invention provides a computer-readable storage medium.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the collection rate optimization analysis method for Internet data collection as described in the first aspect above.

[0021] A fourth aspect of the present invention provides a computer device.

[0022] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the collection rate optimization and analysis method for Internet data collection as described in the first aspect above are implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention is carried out from two links of website column collection completeness test and website list collection completeness test according to the collection process. The present invention directly obtains the required column page and list page by collecting all the websites in the whole network, without manual sorting of web page URLs, thus saving labor costs, greatly reducing intermediate workload and improving collection efficiency.

[0025] The present invention performs a collection rate test based on the collection results of the collection platform list, and can obtain the paging status success rate and the paging success rate, which is of great significance in further analyzing the advantages and disadvantages of the current version of the collection platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 It is a flow chart of a collection rate optimization and analysis method for Internet data collection shown in the first embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0031] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0032] Embodiment 1

[0033] The present embodiment provides a collection rate optimization and analysis method for Internet data collection. The present embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In the present embodiment, the method includes the following steps:

[0034] Get all clickable column page URLs of the website;

[0035] According to all clickable column page URLs of the website, get the column URL of the column page URL and each list URL of the list page HTML;

[0036] According to the column URL of the column page URL, get the column collection rate;

[0037] According to the list URLs of the list page HTML, determine whether to perform paging collection. If so, get the paging collection rate; otherwise, get the list collection rate.

[0038] In order to optimize the full collection rate of the entire network and improve the target results, the full collection rate of the current version of the collection platform is tested and verified according to the full collection rate test plan. In order to better compare the full collection rate of the current version of the collection platform and check the problems of the existing full collection platform, this embodiment conducts tests from two aspects - website column full collection rate test and website list full collection rate test.

[0039] like Figure 1 As shown, the method of this embodiment includes the following steps:

[0040] Step 1: Test data preparation. Manually determine the key policy test website required, and obtain the HTML source code and the URL corresponding to the test website.

[0041] Step 2: Get all clickable column page URLs of the website based on the policy test website URL and website source code obtained in step 1.

[0042] Step 2 also includes the following steps:

[0043] Step 2.1: Extract the source code of the policy test website and collect <href>The column URL and column name contained in the tag.

[0044] Step 2.2: Save the collected column URL and column name in the fields task_url and column_name.

[0045] Step 2.3: Set the fields website_url, website_name, depth, and type to store the website url, website name, website level, and web page type. A depth of 1 indicates that the columns included in the web page are the first-level content of the website, and 2 indicates that the columns included in the web page are the second-level content of the website; when type is set to 0, it means that the web page is a column page, and 1 indicates that the web page is a list page.

[0046] Step 3: Based on the current version of the full network collection function, test the collection rate of the website column URLs.

[0047] Step 3 also includes the following steps:

[0048] Step 3.1: Based on all the clickable column page URLs contained in a website obtained in step 2, select some column URLs for testing and save them in the field urls_result.

[0049] Step 3.2: Call the collection platform column parsing interface to obtain the URL list collected by the platform.

[0050] Step 3.3: Compare the result set in urls_result with the url list content obtained in step 3.2 to check the number of url lists included in the result set and obtain the coverage rate of all columns in the entire network.

[0051] Step 4: Get the clickable list page URL corresponding to the list page in the policy website, including the URL corresponding to the list page pagination.

[0052] Step 4 also includes the following steps:

[0053] Step 4.1: Collect the list page to obtain the URL and HTML source code of the policy website list page.

[0054] Step 4.2: Collect the source code of the list page according to step 2.2 <href>Tag content, collects the URL and list name of each clickable list corresponding to each page in the list page.

[0055] Step 4.3: For paginated list collection, get the total number of pages and save the number in the page_sum field.

[0056] Step 5: Based on the current version of the full network collection function, test the collection rate of the website list URL.

[0057] Step 5 also includes the following steps:

[0058] Step 5.1: Based on all the clickable column page URLs contained in the list page obtained in step 4, select some column URLs for testing and save them in urls_result. For the list page with pages, select the list URLs in different pages and save them in urls_result, in order to prepare for testing the paging status success rate and paging success rate.

[0059] Step 5.2: Call the collection platform list parsing interface, return the interface processing result, and obtain the web page list URL table.

[0060] Step 5.3: Compare the result set in urls_result with the url list content obtained in step 5.2 to check the number of url lists included in the result set and obtain the full coverage rate of the entire network list.

[0061] Step 6: Next, compare and check the number of result sets contained in the URL list in step 5.3 to obtain the paging status success rate and paging success rate.

[0062] Embodiment 2

[0063] This embodiment provides a collection rate optimization and analysis system for Internet data collection.

[0064] A collection rate optimization and analysis system for Internet data collection, comprising:

[0065] The column page URL acquisition module is configured to: acquire all clickable column page URLs of the website;

[0066] The column URL and list URL acquisition module is configured to: obtain the column URL of the column page URL and each list URL of the list page HTML according to all clickable column page URLs of the website;

[0067] The column collection rate optimization module is configured to: obtain the column collection rate according to the column URL of the column page URL;

[0068] The paging and list collection rate optimization module is configured as follows: according to each list URL of the list page HTML, determine whether to perform paging collection rate, if so, get the paging collection rate, otherwise, get the list collection rate.

[0069] It should be noted that the above-mentioned column page URL acquisition module, column URL and list URL acquisition module, column full rate optimization module and paging and list full rate optimization module are the same as the examples and application scenarios implemented by the steps in the first embodiment, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0070] Embodiment 3

[0071] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the collection rate optimization analysis method for Internet data collection as described in the above-mentioned embodiment 1 are implemented.

[0072] Embodiment 4

[0073] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the collection rate optimization and analysis method for Internet data collection as described in the above-mentioned embodiment 1 are implemented.

[0074] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0075] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0078] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / href> < / href>

Claims

1. A collection rate optimization analysis method for Internet data collection, characterized in that: include: Get all clickable column page URLs of the website; According to all clickable column page URLs of the website, get the column URL of the column page URL and each list URL of the list page HTML; According to the column URL of the column page URL, get the column collection rate; According to each list URL of the list page HTML, determine whether to perform paging collection. If so, get the paging collection rate, otherwise, get the list collection rate; The process of obtaining the column collection rate according to the column URL of the column page URL includes: According to all clickable column page URLs, select some column URLs for testing and save them in the field urls_result; Call the collection platform column parsing interface to obtain the URL list collected by the platform; Compare the result set in urls_result with the content of the url list collected by the platform, check the number of url lists included in the result set, and obtain the collection rate of all columns in the entire network; The process of obtaining the paging collection rate includes: According to the list page source code <href> Tag content, collects the URL and list name of each clickable list corresponding to each page in the list page;< / href> For paging list collection, get the total number of pages, and save the number in the page_sum field; Compare the list web page paging result set in page_sum with the contents of the url list web page paging collected by the platform, check the number of url list web page paging included in the result set, and obtain the full collection rate of the whole network paging; The obtained list of completeness includes: Get all clickable column page URLs contained in the list page, select some column URLs for testing, and save them in urls_result; Call the collection platform list parsing interface, return the interface processing result, and obtain the web page list URL table; Compare the result set in urls_result with the contents of the url list obtained in the url table of the web page list to check the number of url lists included in the result set and obtain the full coverage rate of the entire web list.

2. The method for optimizing the collection rate of Internet data collection according to claim 1 is characterized in that: The process of obtaining the URLs of all clickable column pages of the website includes: obtaining the HTML source code of the test website and the URL corresponding to the website; and obtaining the URLs of all clickable column pages of the website according to the HTML source code of the test website and the URL corresponding to the website.

3. The method for optimizing the collection rate of Internet data collection according to claim 2 is characterized in that: The process of obtaining all clickable column page URLs of the website according to the HTML source code in the test website and the URL corresponding to the website includes: Get the HTML source code of the test website. According to the html source code collection in the test website <href> The column URL and column name contained in the tag;< / href> Save the collected column URL and column name in the fields task_url and column_name; Set the fields website_url, website_name, depth, type to save the website url, website name, website depth and web page type respectively; Get all clickable column page URLs of the website.

4. The method for optimizing the collection rate of Internet data collection according to claim 3 is characterized in that: According to the completeness of the list, compare and check the number of result sets included in the URL list to obtain the paging status success rate and paging success rate.

5. A collection rate optimization and analysis system for Internet data collection, characterized in that: include: The column page URL acquisition module is configured to: acquire all clickable column page URLs of the website; The column URL and list URL acquisition module is configured to: obtain the column URL of the column page URL and each list URL of the list page HTML according to all clickable column page URLs of the website; The column collection rate optimization module is configured to: obtain the column collection rate according to the column URL of the column page URL; The paging and list collection rate optimization module is configured to: determine whether to perform paging collection rate according to each list URL of the list page HTML, if so, obtain the paging collection rate, otherwise, obtain the list collection rate; The process of obtaining the column collection rate according to the column URL of the column page URL includes: According to all clickable column page URLs, select some column URLs for testing and save them in the field urls_result; Call the collection platform column parsing interface to obtain the URL list collected by the platform; Compare the result set in urls_result with the content of the url list collected by the platform, check the number of url lists included in the result set, and obtain the collection rate of all columns in the entire network; The process of obtaining the paging collection rate includes: According to the list page source code <href> Tag content, collects the URL and list name of each clickable list corresponding to each page in the list page;< / href> For paging list collection, get the total number of pages, and save the number in the page_sum field; Compare the list web page paging result set in page_sum with the content of the url list web page paging collected by the platform, check the number of url list web page paging included in the result set, and obtain the full collection rate of the whole network paging; The obtained list of completeness includes: Get all clickable column page URLs contained in the list page, select some column URLs for testing, and save them in urls_result; Call the collection platform list parsing interface, return the interface processing result, and obtain the web page list URL table; Compare the result set in urls_result with the contents of the url list obtained in the url table of the web page list to check the number of url lists included in the result set and obtain the full coverage rate of the entire web list.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the collection rate optimization and analysis method for Internet data collection as described in any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the collection rate optimization and analysis method for Internet data collection as described in any one of claims 1-4 are implemented.

Images