Network media push data matching method and system

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a network media push data matching method and system, and the method comprises the steps: 1, analyzing a label of a grabbed article, and carrying out the duplicate checking of a title; 2, the title is processed; 3, searching the titles, performing similarity calculation on the titles and the searched titles, and if a search article with the similarity higher than a first threshold value exists, capturing the search article and storing the search article in a capturing library; 4, performing word segmentation on the title, searching each segmented word, counting the occurrence frequency of the same title in a search result, calculating a score according to a segmented word hit, and capturing and storing a search article of which the score is higher than a second threshold value into a capture library; the innovative pushed data matching method provided by the invention can realize quick matching and automatic release of the pushed manuscripts, and the working efficiency and the timeliness of news pushing are greatly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of data processing, and in particular relates to a network media push data matching method and a system thereof. Background Art

[0002] With the rapid development of science and technology, online media has gradually replaced paper media and become the mainstream channel for information dissemination. Compared with traditional paper media, online media has the advantages of strong real-time, high interactivity, and fast transmission speed. It can transmit information more quickly and cover a wider audience; online media has become the main way for people to obtain information.

[0003] With the continuous advancement and in-depth development of integrated media projects in various places, the demand for monitoring hot spots and timely news push is increasing. At present, editors of online media mainly use the method of manually observing mobile phone push messages, manually sharing links and then obtaining manuscript content to create and publish manuscripts; it increases the workload of editors, and requires a lot of time and energy to observe mobile phone push messages, which is inefficient; and it is impossible to monitor the hot news of the day in real time, and it is easy to miss important information. In addition, due to the uncertainty of manual operation, news push is often not timely and cannot meet users' demand for real-time news. Summary of the invention

[0004] The purpose of the present invention is to provide a network media push data matching method and system thereof, which captures push data through a data capture client and then determines the validity of the data through data matching, thereby reducing the workload of editors and improving the promotion of real-time news.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A network media push data matching method comprises the following steps:

[0007] Step 1: parse and capture the data structure of the article, and check the data structure for duplication; wherein the data structure includes a title subtitle;

[0008] Step 2: Process the subtitle, wherein the processing means first filtering out characters without substantial meaning, and then determining whether the subtitle belongs to a special topic article;

[0009] Step 3: Search the titles and subtitles of non-special topic articles, and calculate the similarity between the titles and subtitles of the captured articles and the searched titles. If there are searched articles with similarity higher than the first threshold, they will be captured and stored in the capture library; if there are no search results with similarity higher than the first threshold, a secondary match will be performed;

[0010] The similarity F=aY+bT, where Y is the matching degree, calculated by Levenshtein distance; T is the time coefficient, t is time, unit is min; a and b are weight coefficients, and a+b=1, a≥b;

[0011] Step 4: Secondary matching includes first segmenting the title subtitle, searching for each segmentation, counting the number of times the same title appears in the search results, that is, counting the number of hits for multiple segments of the same title, calculating the score based on the segmentation hits, and crawling the search articles with scores higher than the second threshold and storing them in the crawling library;

[0012] The score n is the number of hit segmented words, m is the total number of segmented words, T is the time coefficient, t is time, in minutes; Z is the source coefficient, which is 1 if the sources are the same and 0 if the sources are different; c, d, e are weight coefficients, and c+d+e=1, e>c>d;

[0013] Step 5: Cluster analysis of articles in the crawled library. Cluster analysis refers to calculating the similarity between two articles in the crawled library, and associating two articles with similarity higher than a third threshold.

[0014] Furthermore, the duplication check in step 1 is implemented using the MD5 hash algorithm.

[0015] Furthermore, the determination of whether it is a special topic article in step 2 refers to matching the title subtitle with the keywords in the special keyword library to determine whether it matches the special keywords. If it does, it belongs to a special topic. Articles on special topics directly use the latest series of articles on this topic published within a certain time range, and the certain time range refers to within 12 to 24 hours.

[0016] Furthermore, in step 5, cluster analysis is performed only on articles captured within 8 to 15 minutes.

[0017] The present invention also provides a system for implementing the network media push data matching method, comprising: a data capture module, used to capture articles and send them to a data preprocessing module;

[0018] The data preprocessing module is used to parse the data structure of the captured article and check the data structure for duplication; then the subtitle is processed, and the processing means first filtering out characters without substantial meaning, and then judging whether it belongs to a special topic article;

[0019] The data matching module is used to match the title subtitle of non-special topic articles with the searched articles, obtain accurate and effective target articles and store them in the crawling library;

[0020] Matching includes first match and second match. If the target article is matched for the first time, there is no need to perform second match. Otherwise, a second match is performed.

[0021] The first match refers to searching for the title subtitle of non-special topic articles, and calculating the similarity between the title subtitle of the captured article and the searched title, and capturing the searched articles with similarity higher than the first threshold and storing them in the capture library. The similarity F=aY+bT, where Y is the matching degree, calculated by the Levenshtein distance; T is the time coefficient, t is time, unit is min; a and b are weight coefficients, and a+b=1, a≥b;

[0022] Secondary matching includes first segmenting the title subtitle, searching for each segmentation, counting the number of times the same title appears in the search results, that is, counting the number of hits for multiple segments of the same title, calculating the score based on the segmentation hits, and crawling the search articles with scores higher than the second threshold and storing them in the crawling library;

[0023] The score n is the number of hit segmented words, m is the total number of segmented words, T is the time coefficient, t is time, unit is min; Z is source coefficient, which is 1 if the sources are the same and 0 if the sources are different; c, d, e are weight coefficients, and c+d+e=1, e>c>d.

[0024] Furthermore, there is a data association module in the crawl library. The data association module is to calculate the similarity between the articles crawled into the crawl library and associate two articles whose similarity is higher than a third threshold.

[0025] The present invention proposes an innovative push data matching method, which can achieve rapid matching and automatically publish push articles, greatly improving work efficiency and the timeliness of news push. At the same time, a crawling library and a push library are also established to aggregate and summarize the hot news of the day in order to better meet business demand scenarios.

[0026] The data matcher of the present invention can better control hot spots, improve the timeliness and accuracy of news releases, provide users with better information services, and promote the development of integrated media projects to a new level. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0028] Example 1

[0029] like Figure 1As shown, a network media push data matching method provided in this embodiment includes the following steps:

[0030] (1) Analyze and capture the data structure of the article and check for duplication of the data structure.

[0031] The data structure includes platform platformCode, title subtitle, and type title.

[0032] The duplicate check is implemented using the MD5 hash algorithm, which uses the MD5 hash algorithm to generate a 128-bit (16-byte) hash value for the data structure, which is then compared with the hash value in the database to see if it is repeated, thereby implementing duplicate checking for the captured data and avoiding multiple captures of the same data.

[0033] (2) Processing the subtitle, wherein the processing is to first filter out characters without substantial meaning, and then determine whether it belongs to a special topic article.

[0034] The characters without substantial meaning do not refer to meaningless words or special characters. Meaningless words include but are not limited to “details”, “full text”, etc.; special characters include but are not limited to “|”, “>>”, etc.

[0035] The special topic articles refer to popular articles, such as learning about a cultural relic every day, learning about a Chinese medicine every day, and other articles that popularize knowledge and culture.

[0036] The determination of special articles refers to matching the title subtitle with the keywords in the special keyword library to determine whether they meet the special keywords. If they do, they belong to special topics. Articles on special topics directly adopt the latest series of articles on this topic published within a certain time range. The certain time range refers to within 12 to 24 hours.

[0037] (3) Search the titles and subtitles of non-special topic articles, and calculate the similarity between the titles and subtitles of the captured articles and the searched titles. Capture the searched articles whose similarity is higher than a first threshold and store them in the capture library; perform secondary matching on the searched articles whose similarity is lower than the first threshold.

[0038] The similarity F=aY+bT, where Y is the matching degree, calculated by Levenshtein distance; T is the time coefficient, t is time in minutes; a and b are weight coefficients, and a+b = 1. It is reasonable for the weight coefficients a and b to be equal, or b to be slightly greater than a.

[0039] (4) Secondary matching includes first segmenting the title subtitle, searching each segmented word, counting the number of times the same title appears in the search results, that is, counting the number of times the same title has multiple segmented words hit, calculating the score based on the segmented word hits, and crawling the search articles with scores higher than the second threshold and storing them in the crawl library.

[0040] The score n is the number of hit segmented words, m is the total number of segmented words, T is the time coefficient, t is time in minutes; Z is the source coefficient, which is 1 if the sources are the same and 0 if the sources are different; c, d, e are weight coefficients, and c+d+e=1.

[0041] This embodiment takes into account the real-time nature of the news. The weight coefficient e related to the time coefficient T in the weight coefficient is greater than the remaining weight coefficients. Considering that different sources may be the same news, the weight coefficient c of the word segmentation hit is greater than the weight coefficient d of the source coefficient, that is, e>c>d.

[0042] (5) Cluster analysis of articles in the crawled library. Cluster analysis refers to calculating the similarity between two articles in the crawled library. Two articles with similarity higher than a third threshold are related and are associated to facilitate editors to view and filter.

[0043] In order to reduce the slow data processing speed caused by a large amount of data, this embodiment only performs similarity calculation on articles captured within 8 to 15 minutes, and the similarity calculation adopts the calculation method in step 3.

[0044] The method of this embodiment can achieve rapid matching and automatically publish and push manuscripts, greatly improving work efficiency and the timeliness of news push; and according to the subsequent pre-setting of push rules, etc., an automatic push function can be realized to reduce the workload of editors.

[0045] Example 2

[0046] This embodiment provides a push data matching system that can implement Embodiment 1. The system includes a data capture module, a data preprocessing module and a data matching module.

[0047] The data capture module is used to capture articles and send the captured articles to the data preprocessing module. The data capture module can be provided with multiple terminals.

[0048] The data preprocessing module is used to parse the data structure of the captured article and check the data structure for duplication, wherein the data structure includes platform platformCode, title subtitle, and type title; the title subtitle is then processed, and the processing means first filtering out characters without substantial meaning, and then determining whether it belongs to a special topic article.

[0049] The duplicate check is implemented using the MD5 hash algorithm, which uses the MD5 hash algorithm to generate a 128-bit (16-byte) hash value for the data structure, which is then compared with the hash value in the database to see if it is repeated, thereby implementing duplicate checking for the captured data and avoiding multiple captures of the same data.

[0050] The characters without substantial meaning do not refer to meaningless words or special characters. Meaningless words include but are not limited to “details”, “full text”, etc.; special characters include but are not limited to “|”, “>>”, etc.

[0051] The special topic articles refer to popular articles, such as learning about a cultural relic every day, learning about a Chinese medicine every day, and other articles that popularize knowledge and culture.

[0052] The determination of special articles refers to matching the title subtitle with the keywords in the special keyword library to determine whether they meet the special keywords. If they do, they belong to special topics. Articles on special topics directly adopt the latest series of articles on this topic published within a certain time range. The certain time range refers to within 12 to 24 hours.

[0053] The data matching module is used to search the title and subtitle, perform a primary match, and a secondary match before entering the article into the crawl library, and finally match the articles in the crawl library to determine whether they are related, so as to achieve fast matching and automatically publish and push articles, which greatly improves work efficiency and the timeliness of news push.

[0054] The first matching searches for the titles and subtitles of non-special topic articles, and calculates the similarity between the titles and subtitles of the captured articles and the searched titles, captures the searched articles whose similarity is higher than a first threshold and stores them in the capture library; and performs a second matching on the searched articles whose similarity is lower than the first threshold.

[0055] The similarity F=aY+bT, where Y is the matching degree, calculated by Levenshtein distance; T is the time coefficient, t is time in minutes; a and b are weight coefficients, and a+b = 1. It is reasonable for the weight coefficients a and b to be equal, or b to be slightly greater than a.

[0056] The secondary matching includes first segmenting the title subtitle, searching each segmentation, counting the number of times the same title appears in the search results, that is, counting whether multiple segmentations of the same title are hit, calculating the score based on the segmentation hits, and crawling the search articles with scores higher than the second threshold and storing them in the crawl library.

[0057] The score n is the number of hit segmented words, m is the total number of segmented words, T is the time coefficient, t is time in minutes; Z is the source coefficient, which is 1 if the sources are the same and 0 if the sources are different; c, d, e are weight coefficients, and c+d+e=1.

[0058] This embodiment takes into account the real-time nature of the news. The weight coefficient e related to the time coefficient T in the weight coefficient is greater than the remaining weight coefficients. Considering that different sources may be the same news, the weight coefficient c of the word segmentation hit is greater than the weight coefficient d of the source coefficient, that is, e>c>d.

[0059] Secondary matching refers to calculating the similarity between the articles captured in the capture library. Two articles with a similarity higher than the third threshold are related and are associated to facilitate editors to view and filter.

[0060] Application Example 1

[0061] The data structure of the crawled article:

[0062] "platformCode":"A News",

[0063] "subtitle":"The spokesperson made a statement on XXXX>>Details",

[0064] "title":"Breaking News".

[0065] The result after MD5 encryption is: b8ebcfa8f83d2ce508964a433559e5ee; there is no duplication compared with the one in the database.

[0066] Remove “>>” and “Details” from the title and get the title: Press spokesperson makes a statement on XXXX.

[0067] It is judged not to be a special feature article.

[0068] The article with the title “Press Spokesperson Makes a Talk on XXXX” is searched and the similarity is calculated, as shown in Table 1.

[0069] Table 1: Search results and similarity calculation results

[0070]

[0071] In this embodiment, the first threshold is set to 80, and sequence numbers 1 and 4 are directly captured into the capture library.

[0072] Application Example 2

[0073] The data structure of the crawled article:

[0074] {"platformCode":"A_xin_wen",

[0075] "subtitle":"Q meets WE",

[0076] "title":"Exclusive Video"}.

[0077] The similarity calculation is shown in Table 2.

[0078]

[0079] The above results do not reach the default configuration threshold of 80, so a second match is performed.

[0080] After word splitting, the phrase obtained is [Q, W, E]; the matching result is: News 1 has a total of 4 words that are matched, and the matching score of News 1 is 100 points. It is determined that News 1 successfully matches the pushed title, and News 1 is the pushed news and enters the crawling library.

[0081] The above is only a preferred implementation of the present invention, but the protection scope of the present invention is not limited thereto, and any modification and replacement based on the technical solution and inventive concept provided by the present invention should be covered within the protection scope of the present invention.

Claims

1. A network media push data matching method, It is characterized in that The steps include: Step 1: parse and capture the data structure of the article, and check the data structure for duplication; wherein the data structure includes a title subtitle; Step 2: Process the subtitle, wherein the processing means first filtering out characters without substantial meaning, and then determining whether the subtitle belongs to a special topic article; Step 3: Search the titles and subtitles of non-special topic articles, and calculate the similarity between the titles and subtitles of the captured articles and the searched titles. If there are searched articles with similarity higher than the first threshold, they will be captured and stored in the capture library; if there are no search results with similarity higher than the first threshold, a secondary match will be performed; The similarity F=aY+bT, where Y is the matching degree, calculated by Levenshtein distance; T is the time coefficient, t is time, unit is min; a and b are weight coefficients, and a+b=1, a≥b; Step 4: Secondary matching includes first segmenting the title subtitle, searching for each segmentation, counting the number of times the same title appears in the search results, that is, counting the number of hits for multiple segments of the same title, calculating the score based on the segmentation hits, and crawling the search articles with scores higher than the second threshold and storing them in the crawling library; The score n is the number of hit segmented words, m is the total number of segmented words, T is the time coefficient, t is time, in minutes; Z is the source coefficient, which is 1 if the sources are the same and 0 if the sources are different; c, d, e are weight coefficients, and c+d+e=1, e>c>d; Step 5: Cluster analysis of articles in the crawled library. Cluster analysis refers to calculating the similarity between two articles in the crawled library, and associating two articles with similarity higher than a third threshold.

2. A network media push data matching method according to claim 1, It is characterized in that The duplicate checking described in step 1 is implemented using the MD5 hash algorithm.

3. A network media push data matching method according to claim 1, It is characterized in that The judgment as to whether it is a special topic article in step 2 refers to matching the title subtitle with the keywords in the special keyword library to determine whether it matches the special keywords. If it does, it belongs to a special topic. Articles on special topics directly use the latest series of articles on this topic published within a certain time range. The certain time range refers to within 12 to 24 hours.

4. A network media push data matching method according to claim 1, It is characterized in that In step 5, cluster analysis is performed only on articles captured within 8 to 15 minutes.

5. A system for implementing the network media push data matching method according to any one of claims 1 to 4, It is characterized in that include: Data crawling module, used to crawl articles and send them to the data preprocessing module; The data preprocessing module is used to parse the data structure of the captured article and check the data structure for duplication; then the subtitle is processed, and the processing means first filtering out characters without substantial meaning, and then judging whether it belongs to a special topic article; The data matching module is used to match the title subtitle of non-special topic articles with the searched articles, obtain accurate and effective target articles and store them in the crawling library; Matching includes first match and second match. If the target article is matched for the first time, there is no need to perform second match. Otherwise, a second match is performed. The first match refers to searching for the title subtitle of non-special topic articles, and calculating the similarity between the title subtitle of the captured article and the searched title, and capturing the searched articles with similarity higher than the first threshold and storing them in the capture library. The similarity F=aY+bT, where Y is the matching degree, calculated by the Levenshtein distance; T is the time coefficient, t is time, unit is min; a and b are weight coefficients, and a+b=1, a≥b; Secondary matching includes first segmenting the title subtitle, searching for each segmentation, counting the number of times the same title appears in the search results, that is, counting the number of hits for multiple segments of the same title, calculating the score based on the segmentation hits, and crawling the search articles with scores higher than the second threshold and storing them in the crawling library; The score n is the number of hit segmented words, m is the total number of segmented words, T is the time coefficient, t is time, unit is min; Z is source coefficient, which is 1 if the sources are the same and 0 if the sources are different; c, d, e are weight coefficients, and c+d+e=1, e>c>d.

6. A system for implementing a network media push data matching method according to claim 5, It is characterized in that There is also a data association module in the crawl library. The data association module is to calculate the similarity between the articles crawled into the crawl library and associate two articles whose similarity is higher than a third threshold.

Images