Method and device for obtaining the total number of data records
A data recording and acquisition method technology, applied in the field of data processing, can solve the problems of increasing the risk of denying access to the target website, it is difficult to effectively avoid the instability of crawling, and the total number of data records cannot be obtained, so as to reduce the number of crawling failures , reduce the number of crawls, and reduce the effect of instability
Active Publication Date: 2022-02-08
数贸科技(北京)有限公司
View PDF5 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, this crawling method requires frequent and continuous access to the target website, which not only increases the risk of being denied access by the target website, but also has low efficiency in obtaining the total number of data records
In addition, this crawling method is highly unstable, and any request failure may result in the inability to obtain an accurate total number of data records
Although concurrency and fault-tolerant mechanisms can be used in the crawling process, such as multi-threading, failure retry and other strategies, it is still difficult to effectively avoid the instability of crawling
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0027] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0028] figure 1 A schematic flow diagram showing a method for obtaining the total number of data records according to an embodiment of the present invention, as shown in figure 1 As shown, the method includes the following steps:
[0029] Step S100, determine the page number to be captured according to the preset start page number and the preset end page number.
[0030] In order to carry out risk warning and strategic decision-m...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a method, device, computing device and computer storage medium for acquiring the total number of data records, wherein the method includes: step 1, determining the page number to be captured according to the preset start page number and the preset end page number; step 2. According to the page number to be captured, capture the page information corresponding to the page number to be captured; step 3, judge whether the page information is the target page information according to the page characteristics of the page information; if so, execute step 4; if not, execute Step 5; Step 4, calculate the total number of data records according to the target page information and the page data record threshold, and the method ends; Step 5, update the preset start page number or preset end page number according to the page characteristics of the page information ; Then jump to step 1. According to the technical solution provided by the present invention, the number of captures is effectively reduced, the efficiency of obtaining the total number of data records is improved, and the way of obtaining the total number of data records is optimized.
Description
technical field [0001] The present invention relates to the technical field of data processing, in particular to a method, device, computing device and computer storage medium for acquiring the total number of data records. Background technique [0002] In technical fields such as e-commerce, digital trade, Internet finance, etc., web crawler technology is often used to capture the number of data records disclosed by various platforms in the same industry to obtain the total number of data records, and then analyze the total number of data records to obtain To achieve industry analysis, risk warning, strategic decision-making and other purposes. Wherein, the total number of data records may be the total number of transaction data records or the total number of comment data records. [0003] Some target websites do not directly display the total number of data records, but display data records page by page in a dynamic and automatic loading manner, so the total number of dat...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/951G06F16/955G06F16/958
Inventor 王苗淼薛继东
Owner 数贸科技(北京)有限公司