A data deduplication method in a post crawling process and a terminal

Abstract

The application discloses a data deduplication method in a crawling post process. The method receives page data through a current link, acquires multiple groups of key fields in the data at the same time, combines the multiple groups of key fields, and generates a first key value corresponding to the page data. When another first key value same as the first key value of the current data exists in an intranet, it indicates that the current data is repeated data, which is excluded to realize the first deduplication. Then, current link information is acquired again, the current link information is combined with preset key fields to generate a second key value corresponding to the page data. When another second key value same as the current second key value does not exist in the intranet, it is judged as new data, otherwise, it is judged as repeated data to realize the second deduplication. The received data is deduplicated twice by comparing the first key value and the second key value, which greatly reduces the possibility of crawling the same data from the extranet and reduces the repetition rate of the data in the data crawling process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data processing, in particular to a data deduplication method in a post crawling process and a terminal. BACKGROUND

[0002] Currently, the update of posts or news on the Internet is usually achieved by updating the posts or news according to the unique ID corresponding to the internal posts or news. Since the ID has uniqueness, the uniqueness of the posts and news can be guaranteed, so the existing posts and news are not likely to be duplicated.

[0003] However, the data collected through the Internet is not comprehensive compared with the data on the external network, and even the same data content on the external network may have different URL (Uniform Resource Locator) parameters, so the problem of data duplication is likely to occur. In some actual scenarios, for example, two data resources are the same post or news on the external network, but in the post or news crawling process, the system may judge the two resources as different posts or news because the URL parameters corresponding to the two data resources are different. Thus, the duplication of posts and news occurs, which reduces the user experience. Moreover, when the posts or news are operated, the duplication of posts or news will cause repetitive work, which affects the operation efficiency. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a data deduplication method in a post crawling process and a terminal, which reduces the duplication rate of data in the data crawling process.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0006] A data deduplication method in a post crawling process, comprising the steps of:

[0007] receiving page data according to a current link;

[0008] obtaining a plurality of groups of key fields in the page data, combining the plurality of groups of key fields, and generating a first key value corresponding to the page data;

[0009] judging whether the first key value already exists, if not, obtaining current link information according to the current link;

[0010] combining the current link information with the preset key fields, and generating a second key value corresponding to the page data;

[0011] judging whether the second key value already exists, if not, marking the page data as new data, and if yes, marking the page data as duplicate data.

[0012] To solve the above technical problems, another technical solution adopted by the present application is:

[0013] A data deduplication terminal in a crawling post process comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements each step of the above-mentioned data deduplication method in a crawling post process.

[0014] The present application has the beneficial effects that: the current link receives page data at the same time, obtains multiple groups of key fields in the data, and combines the multiple groups of key fields to generate a first key value corresponding to the page data. When there is another first key value in the intranet that is the same as the first key value of the current data, it indicates that the current data is duplicate data, which is excluded to realize the first deduplication. Then, the current link information is obtained, and the current link information is combined with the preset key field to generate a second key value corresponding to the page data. When there is no other second key value in the intranet that is the same as the current second key value, it is judged as new data, otherwise it is judged as duplicate data to realize the second deduplication. The received data is deduplicated twice by comparing the first key value and the second key value, which greatly reduces the possibility of crawling the same data from the extranet, and reduces the data repetition rate in the data crawling process. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A step flowchart of a data deduplication method in a crawling post process according to an embodiment of the present application;

[0016] Figure 2 Another step flowchart of a data deduplication method in a crawling post process according to an embodiment of the present application;

[0017] Figure 3 A structure diagram of a data deduplication terminal in a crawling post process according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] To explain the technical content, purposes and effects of the present application in detail, the following will be described in conjunction with the embodiments and the accompanying drawings.

[0019] Please refer to Figure 1 A data deduplication method in a crawling post process comprises the following steps:

[0020] Receiving page data according to the current link;

[0021] Obtaining multiple groups of key fields in the page data, combining the multiple groups of key fields to generate a first key value corresponding to the page data;

[0022] determining whether the first key value exists, if not, obtaining current link information according to the current link;

[0023] combining the current link information with the preset key field to generate a second key value corresponding to the page data;

[0024] determining whether the second key value exists, if not, marking the page data as new data, and if yes, marking the page data as repeated data.

[0025] As can be seen from the above description, the present application has the beneficial effects that: the current link receives page data while obtaining multiple groups of key fields in the data, and the multiple groups of key fields are combined to generate a first key value corresponding to the page data, when another first key value identical to the first key value of the current data exists in the intranet, it indicates that the current data is repeated data, which is excluded to realize the first deduplication; then the current link information is obtained, the current link information is combined with the preset key field to generate a second key value corresponding to the page data, when another second key value identical to the second key value does not exist in the intranet, it is determined as new data, otherwise it is determined as repeated data, realizing the second deduplication; the received data is deduplicated twice by comparing the first key value and the second key value, greatly reducing the possibility of crawling the same data from the extranet, and reducing the repetition rate of the data in the data crawling process.

[0026] Further, the obtaining of the current link information according to the current link comprises:

[0027] setting a pre-cut position, and cutting the current link according to the pre-cut position;

[0028] retaining the data before the pre-cut position to obtain the current link information.

[0029] As can be seen from the above description, most of the URL addresses are dynamic web page URL addresses at present, when the post data is updated, the parameters after the question mark in the URL link will also change, such as time parameters, resulting in that the system judges the same data as two data; and by setting the pre-cut position, the data before the pre-cut position in the current link is retained, and the data after the pre-cut position is deleted, so that the interference parameters in the current link can be deleted, avoiding the influence of the interference parameters when generating the second key value, and improving the deduplication precision.

[0030] Further, the combining of the multiple groups of key fields to generate the first key value corresponding to the page data comprises:

[0031] combining the multiple groups of key fields into new data and obtaining the first key value through encryption calculation;

[0032] The combining the current link information with the preset key field and generating a second key value corresponding to the page data comprises:

[0033] The current link information is combined with the preset key field into new data, and the second key value is obtained by encryption calculation.

[0034] The first key value and the second key value are hash values.

[0035] As can be seen from the above description, by generating the corresponding hash value through encryption calculation in the process of generating the first key value and the second key value by the hash algorithm, the key field of different lengths can be combined to generate a fixed length data value, so that the judgment of the first key value and the second key value is more simple and accurate, and the generated first key value and second key value have uniqueness, which improves the accuracy of the post duplication.

[0036] Further, the page data received according to the current link comprises:

[0037] The real-time data in the page is obtained through kafka as the page data.

[0038] As can be seen from the above description, the real-time data is obtained through kafka, which ensures high throughput of data acquisition, and also has high availability and other characteristics, ensuring the efficiency of data acquisition.

[0039] Further, the key field comprises a post title, a post content, a user ID and a publishing time.

[0040] The obtaining of the multiple groups of key fields in the page data comprises:

[0041] The page data is processed through flink, and the post title, the post content, the user ID and the publishing time are obtained.

[0042] As can be seen from the above description, the post title, the post content, the user ID and the publishing time corresponding to the real-time data are extracted through flink, so that the related information in the post is accurately obtained, and the accuracy of the obtained post data is improved.

[0043] Further, the combining the current link information with the preset key field and generating a second key value corresponding to the page data comprises:

[0044] The current link information is combined with the user ID to generate a second key value corresponding to the page data.

[0045] From the above description, by combining the current link information with the user ID, the calculation of the second key value is simplified, and the data can be quickly checked and judged according to the current link information and the user ID.

[0046] Further, the judging whether the first key value exists includes:

[0047] Caching the page data corresponding to the first key value by taking the first key value as a key in redis;

[0048] Judging whether the page data is cached successfully, if not, the first key value exists.

[0049] From the above description, by using the redis database to cache data, the first data deduplication is performed by using the unique key-value characteristics of the redis database, thereby improving the accuracy and efficiency of data deduplication.

[0050] Further, judging whether the second key value exists, if not, the page data is new data, otherwise, it is repeated data includes:

[0051] Storing the page data in the database;

[0052] Judging whether the second key value exists, if not, storing the data.

[0053] From the above description, by storing data in the database with a unique primary key type, the second data deduplication is performed by using the primary key unique characteristics of the database, thereby improving the deduplication rate of repeated data.

[0054] Further, the judging whether the second key value exists, if yes, marking the page data as repeated data includes:

[0055] Judging whether the second key value exists, if yes, covering the data corresponding to the second key value in the database according to the second key value.

[0056] From the above description, by comparing whether the second key value exists, when it exists, the corresponding data is updated, the data in the database can be updated in time, thereby improving the accuracy of storing data in the database.

[0057] Please refer to Figure 3 The application also provides a data deduplication terminal in a post crawling process, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements each step of the above-mentioned data deduplication method in the post crawling process when executing the computer program.

[0058] The data deduplication method in the post crawling process and the terminal described above are suitable for the data deduplication operation scene in the migration of a large amount of data, and the following will be described through a specific embodiment.

[0059] Embodiment one

[0060] Please refer to Figure 1 A data deduplication method in a post crawling process, comprising the steps of:

[0061] S1, receiving page data according to a current link;

[0062] The real-time data in the page corresponding to the current connection is obtained through kafka (an open source big data distributed event stream platform with high throughput, high availability, etc.) as the page data; the data is received in real time through kafka, which guarantees the high throughput of the received data; and the real-time data is processed through flink (an open source big data stream processing framework with high performance, high availability, etc.), which guarantees the efficiency of data acquisition, that is, improves the efficiency of data crawling;

[0063] S2, obtaining a plurality of groups of key fields in the page data, combining the plurality of groups of key fields, and generating a first key value corresponding to the page data;

[0064] The plurality of groups of key fields are combined into new data and the first key value is obtained through encryption calculation; specifically, the key fields include the title of the post, the content of the post, the user ID and the publishing time; and the page data is processed through flink, and the title of the post, the content of the post, the user ID and the publishing time are obtained; when flink consumes data for data processing, the post data is cached through a redis (an open source high-performance cache database with key-value characteristics, that is, there can be only one unique key, which is an important basis for deduplication) database; the cache time of the post data can be set according to specific needs, such as setting the expiration time of the cache to one month; the key of the redis database cache is the first key value, specifically, the data of the title of the post, the content of the post, the user ID and the publishing time are combined into new data, and the first key value is calculated by encrypting the new data;

[0065] S3, judging whether the first key value exists, if not, obtaining current link information according to the current link;

[0066] cache the page data corresponding to the first key value through redis with the first key value as a key; determine whether the page data is cached successfully, if not, the first key value already exists; when the redis database performs a set (cache) operation on data with the first key value as a key, if the data set is successful, it indicates that the data corresponding to the first key value is new data, and if the set fails, it is repeated data;

[0067] S4, combine the current link information with the preset key field and generate a second key value corresponding to the page data;

[0068] Combine the current link information with the user ID and generate a second key value corresponding to the page data; specifically, combine the current link information with the preset key field into new data and obtain the second key value through encryption calculation;

[0069] The preset key field can be the same as the key field in the first key value, or other types of key fields can be obtained from the page data as the preset key field, such as the current connection, and the preset key field can be a combination of multiple key fields;

[0070] S5, determine whether the second key value already exists, if not, mark the page data as new data; otherwise, mark the page data as repeated data;

[0071] Store the page data through the database; determine whether the second key value already exists, if not, store the data; use the second key value as the primary key of the database to perform data storage operation, which can ensure the primary key covering operation when storing and realize the second deduplication; wherein, the encryption algorithm in steps S2 and S4 is SM3 encryption algorithm of national cryptography (national cryptography bureau approved domestic cryptography algorithm, i.e. commercial cryptography);

[0072] In an optional embodiment, the second key value can be used as the primary key to store the page data in a database with a unique primary key type, because the primary key is unique, so automatic deduplication can be realized when storing data.

[0073] Embodiment two

[0074] The difference between this embodiment and embodiment one is that how to obtain the current link information in step S3 is limited;

[0075] Please refer to Figure 2 , the current link information includes:

[0076] Setting a pre-cut position, cutting the current link according to the pre-cut position; retaining data before the pre-cut position to obtain the current link information;

[0077] Meanwhile, step S5 further includes: judging whether the second key value exists, if yes, covering the data corresponding to the second key value in the database with the page data according to the second key value;

[0078] After the deduplication through the first key value, a large amount of repeated data can be excluded; but in some scenarios, there are still repeated data; for example, a version of a post is published in the external network, but the post is updated again within a few hours; and with the update of the post, some parameters in the URL link corresponding to the post after the question mark will also change; at this time, although the content of the post has changed, it is equivalent to the same data in the external network; if only once deduplication is performed, the updated post cannot be updated as repeated data, but the updated data will be cached as new data; therefore, by setting the pre-cut position to the data bit after the question mark in the URL, the parameters before the question mark in the URL corresponding to the post can be obtained, and the parameters after the question mark in the URL are discarded, which ensures that the system will not save the updated data as new data after the post is updated because the URL corresponding to the post changes, so that the deduplication rate reaches more than 95%; and the old data can be invalidated, and only the modified new data is displayed, thereby improving the timeliness and accuracy of the post content finally crawled.

[0079] Embodiment three

[0080] Please refer to Figure 3 A data deduplication terminal in a post crawling process, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements each step of the data deduplication method in a post crawling process as described in embodiment one or two when executing the computer program.

[0081] In summary, the application provides a data deduplication method in a crawling post process, which obtains multiple groups of key fields in the data, such as the title of the post, the content of the post, the user ID and the publishing time, while receiving the page data through the current link, and generates a first key value corresponding to the page data by combining the four groups of key fields of the title of the post, the content of the post, the user ID and the publishing time and through encryption calculation. When there is another first key value same as the first key value of the current data in the intranet, it indicates that the current data is duplicate data, which is excluded to realize the first deduplication. Then the current link is cut through setting a pre-cut position to obtain the current link information, and the current link information is combined with the preset key fields and through encryption calculation to generate a second key value corresponding to the page data. When there is no another second key value same as the current second key value in the intranet, it is judged as new data, otherwise it is judged as duplicate data, and the old data in the database is updated as new data to realize the second deduplication. The received data is deduplicated twice by comparing the first key value and the second key value, which greatly reduces the possibility of crawling the same data from the external network, and reduces the data repetition rate in the data crawling process.

[0082] The above description is only an embodiment of the application, and does not limit the patent scope of the application. Any equivalent transformation or direct or indirect application in the related technical field based on the content of the specification and drawings is also included in the patent protection scope of the application.

Claims

1. A data deduplication method in a crawling post process, characterized in that, The method comprises the steps of: receiving page data according to a current link, specifically comprising: obtaining real-time data in the page through kafka as the page data; obtaining multiple sets of key fields in the page data, combining the multiple sets of key fields, and generating a first key value corresponding to the page data; judging whether the first key value exists, if not, obtaining current link information according to the current link, the judgment of whether the first key value exists specifically comprising: caching the page data corresponding to the first key value in redis with the first key value as the key; judging whether the page data is cached successfully, if not, the first key value exists; when the redis database caches data with the first key value as the key, if the data is cached successfully, it means that the data corresponding to the first key value is new data, if the caching fails, it is repeated data; the obtaining of the current link information comprises: setting a pre-cut position, cutting the current link according to the pre-cut position; retaining the data before the pre-cut position to obtain the current link information; the pre-cut position is set to the data bit after the question mark in the URL link, the parameters before the question mark in the URL link are retained, and the parameters after the question mark in the URL link are discarded; combining the current link information with the preset key fields to generate a second key value corresponding to the page data, generating a corresponding hash value through SM3 encryption calculation in the process of generating the first key value and the second key value by using a hash algorithm, and combining different length key fields to generate a fixed length data value; judging whether the second key value exists, if not, marking the page data as new data; if yes, marking the page data as repeated data; the judgment of whether the second key value exists, if yes, marking the page data as repeated data comprises: judging whether the second key value exists, if yes, covering the data corresponding to the second key value in the database with the page data according to the second key value.

2. The method of claim 1, wherein, The key fields comprise a post title, post content, user ID and publishing time; The obtaining of the multiple sets of key fields in the page data comprises: processing the page data through flink, and obtaining the post title, post content, user ID and publishing time.

3. The method of claim 1, wherein, The combining of the current link information with the preset key fields to generate a second key value corresponding to the page data comprises: combining the current link information with the user ID to generate a second key value corresponding to the page data.

4. The method of claim 1, wherein, judging whether the second key value exists, if not, the page data is new data; otherwise, it is repeated data, comprising: storing the page data through a database; judging whether the second key value exists, if not, storing the page data.

5. A data deduplication terminal in a crawling post process, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor executes the computer program to realize each step in the data deduplication method in the process of crawling the post according to any one of claims 1-4.

Images