A complementary method of real-time and offline paper recommendation based on Hadoop ecosystem

Abstract

The present invention discloses a complementary method of real-time and offline recommendation of papers based on Hadoop ecology, which belongs to the field of big data recommendation. The present invention uses the data warehouse Hive in the Hadoop ecology to store paper data, segment paper tags, and store them in the data warehouse in a vectorized manner; introduces the offline computing engine MapReduce to calculate offline recommendation results for data; introduces the stream computing engine Spark Streaming to collect paper data in Kafka in real time, and after segmentation and vectorization, reads the paper vectorized data in the Hive data warehouse with Spark SQL to calculate the real-time recommendation results. The patent of the present invention combines the recommendation algorithm and the Hadoop ecology to not only effectively solve the problem of mass paper recommendation and improve the real-time nature of paper recommendation, but also improves the recommendation accuracy and optimizes the paper recommendation results by correcting the similarity calculation according to the paper classification number.

Description

Technical Field

[0001] The present invention belongs to the field of big data paper recommendation, and more specifically, relates to a complementary method of real-time paper recommendation and offline recommendation based on Hadoop ecology. Background Art

[0002] With the continuous development of science and technology, not only the number of scientific researchers is increasing, but also the number of academic papers published is growing rapidly. The explosive growth of paper data has led scientific researchers from the era of information scarcity to the era of information overload. As the most widely popular method to solve information overload, the recommendation system includes demography-based recommendation algorithms, content-based recommendation algorithms, and collaborative filtering recommendation algorithms. The demography-based recommendation algorithm (Demographic-based Recommendation) finds the relevance of users based on the basic information of system users, and then recommends other items that similar users like to the current user. The content-based recommendation algorithm (Content-basedRecommendation) finds the relevance of items based on the characteristics of the recommended items or content, such as the characteristics of the items--labels; the characteristics of text information--keywords, to find the relevance of items, and then recommends similar items to users based on the user's historical preferences. Collaborative filtering algorithms (Collaborative Filtering) include memory-based collaborative filtering (Memory-Based CF) and model-based collaborative filtering (Model-Based CF); memory-based collaborative filtering can be divided into user-based collaborative filtering and item-based collaborative filtering. Model-based collaborative filtering uses data mining and machine learning to build recommendation models, including singular value decomposition (SVD), latent semantic analysis (LSA), clustering, etc.

[0003] Traditional offline paper recommendation technology faces massive amounts of papers, takes a long time to calculate, and is prone to memory overflow. In addition, offline computing is difficult to meet real-time requirements, and the import of new papers will cause cold start and other problems.

[0004] After searching, a Chinese patent application for an expert recommendation method, device, equipment and storage medium based on paper data analysis, with publication number CN112100470A and publication date December 18, 2020, includes: obtaining keywords of the demand document; crawling the paper title, paper abstract, author and journal of the paper related to the keywords of the demand document; calculating the text similarity between the paper and the demand document based on the paper title and paper abstract; obtaining the contribution rate of each paper author to the paper based on the paper author information; obtaining the standardized composite impact factor of the paper based on the journal; and calculating the recommendation score of each paper author based on the text similarity, the contribution rate and the composite impact factor. However, the invention does not take into account that under large-scale paper data, the expert recommendation method not only takes a long time to calculate, but also has a memory overflow problem; at the same time, with the collection of new papers, new papers have problems such as cold start.

[0005] In addition, the patent publication number is CN110717093A, the publication date is January 21, 2020, and the name of the invention is a movie recommendation system based on Spark, including a data collection part, a data storage part, an offline recommendation part and a real-time recommendation part, wherein: the data collection part: real-time monitoring of user data changes through the Web end, Flume reads log updates from the Web end operation log, obtains the user's rating of the movie, sends the log information to the Kafka cluster in real time, uses Kafka as a cache component for streaming data to accept data collection requests from Flume, and pushes the data to the real-time recommendation part; and uses MongoDB, ES, and Redis as the data storage part, and as the data storage for raw data and offline and real-time data processing. The invention realizes data collection and processing through the user's rating of the movie; but does not save the intermediate temporary calculation results in the data warehouse Hive to save the intermediate temporary calculation time. In addition, it is different in the dimension and number of data tags, so it is difficult to apply the paper recommendation method of the present invention. Summary of the invention

[0006] 1. Problem to be solved

[0007] In view of the problems of poor real-time performance, memory overflow, cold start, etc. when using traditional offline recommendation technology to recommend massive papers in the prior art, the present invention provides a complementary method for real-time and offline recommendation of papers based on Hadoop, which can solve the cold start problems caused by large data volume, memory overflow and import of new papers.

[0008] 2. Technical solution

[0009] In order to solve the above problems, the technical solution adopted by the present invention is as follows:

[0010] A complementary method for real-time recommendation and offline recommendation based on Hadoop papers, comprising the following steps:

[0011] Step S1: Using MapReduce, an offline computing engine component of the Hadoop framework, it not only reads the paper vectorized label data in the data warehouse Hive, but also runs the offline recommendation algorithm for offline computing, thereby obtaining offline recommendation results;

[0012] Step S2: Use the distributed message queue Kafka to receive paper data in real time, then use the Spark framework's stream computing engine Spark Streaming and database engine Spark SQL to read the paper data in Kafka and Hive respectively, and then run the real-time recommendation algorithm for real-time computing to obtain real-time recommendation results;

[0013] Step S3: merge the real-time calculation recommendation results into the offline calculation recommendation results and push them.

[0014] Wherein, step S1 includes the following steps:

[0015] Step S11: Build the data warehouse Hive in the Hadoop ecosystem, and use the ODS layer of Hive to store the original data of the paper; format and ETL the semi-structured paper data in the ODS layer, convert it into structured paper data, and store it in the DWD layer;

[0016] Step S12: Extract the secondary classification number labels of papers in the DWD layer, construct the secondary classification number-paper weight matrix c_p, and store it in the DWS layer, where the weight w of the paper belonging to each secondary classification number pc The formula is:

[0017]

[0018] where f pc is the number of times paper p is marked as secondary classification code c, It is the cumulative number of times that n papers p are marked with secondary classification code c, that is, the importance of this secondary classification code c in the secondary classification code set in the paper collection. It is stipulated that the total number of secondary classification code labels for each paper is 1. For papers with multiple secondary classification code labels, the number of secondary classification code labels is the frequency in the secondary classification code labels of the paper. For example, if the paper has three secondary classification code labels, the number of each secondary classification code label is 1 / 3; the denominator is the cumulative number of secondary classification code label c. Among them, the ODS layer is the original data layer, which is used to store the original data and keep the original appearance without processing; the DWD layer is the detailed data layer, which is used to clean the ODS layer data, such as ETL; the DWS layer is the service data layer, which is used to summarize and organize the DWD layer data; the ADS layer is the data application layer, which is used to save the offline recommendation calculation results and the real-time calculation recommendation results;

[0019] Step S13: Extract the abstracts of the papers in the DWD layer and segment them to obtain the abstract tag elements, construct the abstract-paper weight matrix a_p, and store it in the DWS layer. For each element f of the a_p matrix pa , which represents the weight of the abstract tag a in the paper p, the formula is:

[0020]

[0021] in is the number of abstract tags a in paper p, N pa is the number of abstract tags of paper p, N is the number of papers, and N a is the number of papers containing abstract label a;

[0022] Step S14: Extract the titles of papers in the DWD layer and segment them to obtain the title tag elements, construct the title-paper weight matrix t_p, and store it in the DWS layer. For each element f of the t_p matrix pt , which represents the weight of title tag t in paper p. If the title of paper p contains title tag t after word segmentation, then:

[0023]

[0024] Otherwise set f pt =0, where N pt is the number of title tags in paper p, N is the number of papers, and N t is the number of papers containing title tag t;

[0025] Step S15: Extract keywords from papers in the DWD layer, construct a keyword-paper weight matrix k_p, and store it in the DWS layer. For each element f of the k_p matrix pk , which represents the weight of keyword tag k in paper p. If paper p contains keyword tag k, then:

[0026]

[0027] Otherwise set f pt =0, where N pk is the number of keywords in paper p, N is the number of papers, and N k is the number of papers containing keyword tag k;

[0028] Step S16: Extract the relevant weight matrix of the abstract, title, keyword, classification number, etc. of the DWS layer paper, f(p i ,p j ; w) is the paper similarity, take the first N1 recommendations with similarity greater than T1, then filter the recommendations with threshold greater than T2 or the first N2, and store them in the ADS layer. The formula is as follows:

[0029]

[0030] where w 1 is the paper title weight, w 2 is the paper keyword weight, w 3 is the paper abstract weight, 0≤w 2 ≤w 1 ≤w 3 ≤1,w 1 +w 2 +w 3 =1, r is the adjustment factor, that is, the weighted weight of the secondary classification number. Through experimental observation, 0.01≤r≤0.1 is used to prevent overfitting. is the similarity of the paper elements, and the formula is as follows:

[0031]

[0032] When l=1, O is the title set T; when l=2, O is the keyword set K; when l=3, O is the abstract set A.

[0033] Wherein, step S2 includes the following steps:

[0034] Step S21: After building the distributed message publishing and subscription system Kafka, new paper data is written into Kafka using the data stream method; Spark Streaming collects the paper data set S of the Kafka message queue within the rolling window T time period, and the paper data set S includes paper s;

[0035] Step S22: The weight w of each secondary classification number of the papers in the paper dataset S sc , the formula is:

[0036]

[0037] The numerator f sc is the number of times paper s is marked with secondary classification code c, the denominator is the cumulative number of secondary classification number labels c without considering the paper collection S. If the original paper collection does not contain classification number c, then w sc =1;

[0038] Step S23: Construct the abstract-paper weight matrix a_s. For each element f of the a_s matrix sa , which represents the weight of the abstract label a in the paper s, and the formula is:

[0039]

[0040] in is the number of papers s with abstract tags a, N sa is the number of abstract tags of paper s, N is the number of papers not included in the paper set S, and N a is the number of papers that do not contain the paper set S but contain the abstract label a;

[0041] Step S24: Construct the title-paper weight matrix t_s. For each element f of the t_s matrix st , which represents the weight of title tag t in paper s. If paper s contains title tag t, then:

[0042]

[0043] Otherwise set f st =0, where N st is the number of title tags of paper s, N is the number of title tags not including paper set S, N t is the number of papers containing title tag t;

[0044] Step S25: Construct the keyword-paper weight matrix k_s. For each element f of the k_s matrix sk , which represents the weight of keyword tag k in paper s. If paper s contains keyword tag k, then:

[0045]

[0046] Otherwise set f sk =0, where N sk is the number of keywords in paper s, N is the number of papers, and N k is the number of papers containing keyword tag k;

[0047] Step S26: f(p i ,s j; w) is the paper similarity, take the first N1 recommendations with similarity greater than T1, and then filter the recommendations with a threshold greater than T2 or the first N2 recommendations, where T1 represents the initial screening, constructed as a screening library, T2 represents precise screening, T1≤T2, and stored in the ADS layer. The formula is as follows:

[0048]

[0049] where w 1 is the paper title weight, w 2 is the paper keyword weight, w 3 is the paper abstract weight, 0≤w 2 ≤w 1 ≤w 3 ≤1,w 1 +w 2 +w 3 =1, r is the adjustment factor, which is the weighted weight of the secondary classification number, 0.01≤r≤0.1, is the title vector in paper i, is the keyword vector in paper i, is the abstract vector in paper i, For the paper i and j The difference between the secondary classification numbers, is the similarity of paper elements, and the formula is as follows:

[0050]

[0051] When l=1, O is the title set T; when l=2, O is the keyword set K; when l=3, O is the abstract set A.

[0052] In order to improve the effect of paper recommendation, the paper tag information, such as paper abstract, title, keyword, classification number, etc., is integrated into the traditional method, and the similarity calculation result is corrected by the classification number of the paper. At the same time, the paper data volume is too large, and the import of new paper data causes cold start and other problems. The present invention adopts the Hadoop ecosystem and designs a content recommendation algorithm based on MapReduce + data warehouse Hive to solve the problem of excessive data volume. In addition, in order to solve the cold start problem, Spark Streaming is introduced to process the newly added paper data.

[0053] The invention discloses a complementary method for real-time recommendation and offline recommendation of papers based on Hadoop, which comprises building a data warehouse Hive of Hadoop ecology, a parallel computing model MapReduce based on external memory, a distributed message queue Kafka, and a distributed stream computing engine Spark Streaming based on memory, constructing sparse vectorized temporary data according to the abstract, title, keyword and secondary classification number of the paper, using the data warehouse to store the sparse quantized temporary data, providing the offline computing model MapReduce with an offline recommendation algorithm based on secondary classification number correction to calculate the recommendation result, using Spark Streaming to read the paper in the distributed message queue Kafka in a stream mode, constructing sparse vectorized temporary data, and then using the real-time recommendation algorithm based on secondary classification correction to calculate the recommendation result.

[0054] 3. Beneficial effects

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) The present invention uses Hadoop big data technology to effectively handle the storage and calculation problems of massive paper data, avoiding insufficient memory during calculation; and introduces streaming processing technology to effectively solve the cold start problem caused by the import of new papers;

[0057] (2) The present invention uses data warehouse technology to effectively process the unstructured data of the paper. In addition, when calculating the recommendation algorithm, the hierarchical mode of the data warehouse is used to improve the efficiency of using temporary data and save the calculation time of temporary intermediate results. It is very suitable for paper recommendation in massive data.

[0058] (3) The present invention adopts a recommendation algorithm based on the correction of the secondary classification number, which improves the accuracy of paper recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. However, it should be noted that these drawings are designed only for explanation purposes and are not intended to limit the scope of the present invention.

[0060] Figure 1 This is a flowchart of real-time and offline recommendation of papers based on Hadoop ecology in the present invention;

[0061] Figure 2 This is a flowchart of the Hive hierarchical processing of the data warehouse of the present invention;

[0062] Figure 3 This is an offline calculation flow chart of step S1 of the present invention;

[0063] Figure 4This is a real-time calculation flow chart of step S2 of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments; moreover, the various embodiments are not relatively independent and can be combined with each other as needed to achieve better results. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] The present invention provides a complementary method for real-time paper recommendation and offline recommendation based on Hadoop, adopts the data warehouse in Hadoop to store massive paper data; and introduces Hadoop's offline computing engine MapReduce in the process of traditional offline computing to solve the problems of large data volume and memory overflow; at the same time, introduces the stream computing engine Spark Streaming and the distributed message queue Kafka to solve the cold start problem caused by the import of new papers; and adopts a correction formula based on the secondary classification number of the paper in the process of calculating similarity to improve the accuracy of paper similarity.

[0067] Example 1

[0068] The present invention provides a complementary method for real-time and offline paper recommendation based on Hadoop ecology. The specific embodiments described here are only used to explain the present invention, and the specific implementation method can be determined according to actual conditions.

[0069] A complementary method for real-time and offline paper recommendation based on Hadoop ecosystem, comprising the following steps:

[0070] 1. After building the data warehouse Hive of the Hadoop ecosystem, its ODS layer stores the original paper data of the paper, as shown in Table 1. Each paper contains title, author, keyword, abstract, classification number, institution, source journal, content, etc. The semi-structured data of the ODS layer is then converted into structured data and stored in the DWD layer by ETL. The ETL, for example, cleans papers with empty keywords, that is, the key of the paper with title5 is empty and is cleaned up, as shown in Table 2, where p1, p2, p3, and p4 are the IDs of the papers.

[0071] 2. Extract the DWD layer paper classification number, according to the rules of step S12, extract the DWD layer paper secondary classification number label, construct the secondary classification number-paper weight matrix c_p, and store it in the DWS layer, where the paper belongs to each secondary classification number weight w pc The formula is:

[0072]

[0073] where f pc is the number of times paper p is marked as secondary classification code c, is the cumulative number of times n papers p are marked with the secondary classification code c, that is, the importance of this secondary classification code c in the secondary classification code set in the paper collection. The total number of secondary classification code labels for each paper is specified to be 1. For papers with multiple secondary classification code labels, the number of secondary classification code labels is the frequency in the secondary classification code labels of the paper. The results are shown in Table 3. TQ, P6, TG, TU, and TM are all secondary classification code labels. They are classified according to the Chinese Library Classification System. The first letter represents the first-level classification code, and the first letter plus the second character represents the second-level classification code. The second-level classification code is used here because the first-level classification code range is too large, and the third-level classification code is too detailed. Considering comprehensively, the second-level classification code is the best. Then construct the second-level classification code-paper weight matrix c_p and store it in the DWS layer. Its sparse matrix storage form is shown in Table 4, where w p1TQ , w p4TQ The calculation is as follows:

[0074]

[0075] This indicates that the TQ secondary classification number is more representative in p4 than in p1.

[0076] Then extract the abstracts of the papers in the DWD layer and segment them to obtain the paper abstract matrix, as shown in Table 5. Then reconstruct the abstract-paper weight matrix a_p and store it in the DWS layer. Its sparse matrix storage form is shown in Table 6.

[0077] For each element f of the a_p matrix pa , which represents the weight of the abstract tag a in the paper p, the formula is:

[0078]

[0079] in is the number of abstract tags a in paper p, N pa is the number of abstract tags of paper p, N is the number of papers, and N a is the number of papers containing abstract label a.

[0080] In this embodiment, f p1a1, f p1a2 The calculation is as follows:

[0081]

[0082] f p1a1 <f p1a2 , indicating that a2 is more representative than a1 in p1. Even though a1 appears more often in p1 than a2 does, a1 appears more often than a2 in all papers, and a2 is more scarce and more distinctive.

[0083] Then extract the titles of the papers in the DWD layer and segment them to get the paper title matrix, as shown in Table 7. Then construct the title-paper weight matrix t_p and store it in the DWS layer. For each element f of the t_p matrix pt , if the title of paper p contains title tag t after word segmentation, then:

[0084]

[0085] Otherwise set f pt =0, where N pt is the number of title tags in paper p, N is the number of papers, and N t is the number of papers containing title tag t. Its sparse matrix storage form is shown in Table 8. p1t1 , f p1t2 The calculation is as follows:

[0086]

[0087] It means that t2 is more representative than t1 in p1. In all the papers, t1 appears more times than t2, so t2 is more scarce and more distinctive.

[0088] Then extract the keywords of the papers in the DWD layer and segment them to obtain the paper keyword matrix, as shown in Table 9. Then construct the keyword-paper weight matrix k_p and store it in the DWS layer. For each element f of the k_p matrix pk , if paper p contains keyword tag k, then:

[0089]

[0090] Otherwise set f pt =0, where N pk is the number of keywords in paper p, N is the number of papers, and N k is the number of papers containing keyword tag k.

[0091] Its sparse matrix storage form is shown in Table 10, where f p1k1 , f p1k2 The calculation is as follows:

[0092]

[0093] It means that k2 is more representative than k1 in p1. In all the papers, k1 appears more often than k2, so k2 is more scarce and more distinctive.

[0094] 3. Extract the relevant weight matrix of the abstract, title, keyword, classification number, etc. of the DWS layer paper, f(p i ,p j ; w) is the paper similarity, take the first N1 recommendations with similarity greater than T1, and then filter the recommendations with threshold greater than T2 or the first N2. In this embodiment, the threshold T1 is 0.1, N1 is 50, T2 is 0.3, and N2 is 15. The results are stored in the ADS layer, and its f(p 1 ,p 2 ; w) is calculated as follows:

[0095]

[0096] where w 1 is the paper title weight, w 2 is the paper keyword weight, w 3 is the weight of the paper abstract, the cumulative sum is 1, r is the adjustment factor, that is, the weighted weight of the secondary classification number. In this embodiment, r is taken as 0.05, w 1 is 0.3, w 2 is 0.55, w 3 The weight is 0.15, because the keywords are the condensed information of the paper, so they have the highest weight. The title implies the main direction of the paper, so the weight is second. is the similarity of the paper elements, such as The formula is as follows:

[0097]

[0098] The calculation is as follows:

[0099]

[0100] The other results are calculated in the same way, and f(p 1 ,p 2 ; w)

[0101]

[0102] If the initial screening is satisfied, the similarity value is greater than the threshold of 0.1 and is in the top 50, it will be retained. When the similarity is greater than the threshold of 0.3 or the top 15 are retained, it will be retained and recommended. The threshold can be determined according to the similarity requirements.

[0103] 4. After building the distributed message publishing and subscription system Kafka, new paper data is written to Kafka using the data stream method; Spark Streaming collects the paper data set S of the Kafka message queue within the rolling window T time period, and the paper data set S contains paper s. For example, the paper data set S now has a paper s1, whose label elements are shown in Table 11, but its relevant weight matrix is ​​calculated without considering the relevant weight matrix of the paper collection S. The secondary classification number-paper weight matrix c_s of the paper collection S is constructed, and the formula is:

[0104]

[0105] The numerator f sc is the number of times paper s is marked with secondary classification code c, the denominator is the cumulative number of secondary classification number labels c without considering the paper collection S. If the original paper collection does not contain classification number c, then w sc =1.

[0106] Such as s1TQ , w s1P2 The calculation is as follows, where w s1TQ is the weight of the secondary classification number TQ in paper s1, w s1P2 The weight of the secondary classification number P2 in paper s1 is:

[0107]

[0108] There are two secondary classification number labels in the paper, TQ and P2, where P2 is a new label after word segmentation. The original paper collection does not include this classification number P2. The weight of P2 in s is 0.5, that is, the numerator f sP2 =0.5; the denominator is the cumulative number of secondary classification number P2 in the original collection of papers. The original collection of papers does not contain secondary classification number P2, that is, Therefore, we cannot use 0.5 / 0. However, for new labels, in order to highlight their importance, we set the weight in the extreme case to 1, that is, we set w s1P2 =1.

[0109] Construct the title-paper weight matrix t_s of the paper collection S, where f s1t4 The calculation is as follows:

[0110]

[0111] Construct the abstract-paper weight matrix a_s of the paper collection S, where f s1a4 The calculation is as follows:

[0112]

[0113] Construct the keyword-paper weight matrix k_s of the paper collection S, where f s1k1 The calculation is as follows:

[0114]

[0115] 5. Extract the relevant weight matrix of c_s, t_s, a_s, k_s, f(s i ,p j ; w) is the paper similarity, take the first N1 recommendations with similarity greater than T1, and then filter the recommendations with threshold greater than T2 or the first N2. In this embodiment, the threshold T1 is 0.1, N1 is 50, T2 is 0.3, and N2 is 15. The results are stored in the ADS layer, and its f(p 1 ,s 1 ; w) is calculated as follows:

[0116]

[0117] where w 1 is the paper title weight, w 2 is the paper keyword weight, w 3 is the weight of the paper abstract, the cumulative sum is 1, r is the adjustment factor, that is, the weighted weight of the secondary classification number. In this embodiment, r is taken as 0.05, and w 1 is 0.3, w 2 is 0.55, w 3 is 0.15, among which, keywords are the condensed information of the paper, so they have the largest weight. is the similarity of the paper elements, such as The calculation is as follows:

[0118]

[0119] The other results are calculated in the same way, and f(p 1 ,s 1 ; w)

[0120]

[0121] If the initial screening is satisfied, the similarity value is greater than the threshold of 0.1 and is in the top 50, it will be retained. When screening again, if the similarity value is greater than the threshold of 0.3 or is in the top 15, it will be retained and recommended.

[0122] 6. The real-time calculation recommendation results are updated to the ADS layer, merged with the offline calculation recommendation results of the ADS layer, and written into the business database, such as MySQL.

[0123] Among them, Table 1 is the ODS layer of the data warehouse Hive of the present invention;

[0124] Table 2 shows the DWD layer of the data warehouse Hive of the present invention;

[0125] Table 3 is the secondary classification code matrix of the paper;

[0126] Table 4 is the secondary classification number-paper weight matrix;

[0127] Table 5 is the paper abstract matrix;

[0128] Table 6 is the abstract-paper weight matrix;

[0129] Table 7 is the title-paper weight matrix;

[0130] Table 8 is the title-paper weight matrix;

[0131] Table 9 is the paper keyword matrix;

[0132] Table 10 is the keyword-paper weight matrix;

[0133] Table 11 shows the label elements of paper s1.

[0134] Table 1 ODS layer of Hive data warehouse of the present invention

[0135] Title: title1; Abstract: abstract1; Keywords: key1 key2 key3…; Classification: class1|class2|class3;… Title: title2; Abstract: abstract2; Keywords: key4 key2 key3…; Classification: class1;… Title: title3; Abstract: abstract3; Keywords: key6 key5 key3…; Classification: class1|class2;… Title: title4; Abstract: abstract4; Keywords: key7 key8 key3…; Classification: class3;… Title: title5; Abstract: abstract4; Keywords:; Classification: class3;…

[0136] Table 2 DWD layer of the data warehouse Hive of the present invention

[0137]

[0138]

[0139] Table 3 Paper secondary classification code matrix

[0140] ID TQ P6 TG TU TM p1 0.5 0.5 0 0 0 p2 0 0 1 0 0 p3 0 0 0 0.5 0.5 p4 1 0 0 0 0

[0141] Table 4 Secondary classification code-paper weight matrix

[0142] ID Secondary classification number Weight p1 TQ 0.333 p1 P6 1 p2 TG 1 p3 TU 1 p3 TM 1 p4 TQ 0.667

[0143] Table 5 Paper abstract matrix

[0144] ID a1 a2 a3 a4 a5 a6 p1 5 2 1 0 2 0 p2 6 0 1 0 2 0 p3 0 0 0 5 5 0 p4 4 0 0 2 0 1

[0145] Table 6 Summary - Paper Weight Matrix

[0146]

[0147]

[0148] Table 7 Title-paper weight matrix

[0149] ID t1 t2 t3 t4 t5 t6 p1 1 1 1 0 1 0 p2 1 0 1 0 1 0 p3 0 0 0 1 1 0 p4 1 0 0 1 0 1

[0150] Table 8 Title-paper weight matrix

[0151] ID Title Elements Weight p1 t1 0.056 p1 t2 0.229 p1 t3 0.128 p1 t5 0.056 p2 t1 0.074 p2 t3 0.057 p2 t5 0.074 p3 t4 0.255 p3 t5 0.112 p4 t1 0.074 p4 t4 0.170 p4 t6 0.305

[0152] Table 9 Paper keyword matrix

[0153]

[0154]

[0155] Table 10 Keyword-paper weight matrix

[0156] ID Title Elements Weight p1 k1 0.056 p1 k2 0.229 p1 k3 0.128 p1 k5 0.056 p2 k1 0.074 p2 k3 0.057 p2 k5 0.074 p3 k4 0.255 p3 k5 0.112 p4 k1 0.074 p4 k4 0.170 p4 k6 0.305

[0157] Table 11 Label elements of paper s1

[0158] ID Title tag element Summary Label Elements Keyword tag element Classification number label element s1 t1 t4 t5 t9 a1:5a4:2a5:1a6:2 k1 k3 k4 k6 TQP2

[0159] The above schematically describes the invention and its implementation methods, which are not restrictive. The invention can be implemented in other specific forms without departing from the spirit or basic features of the invention. What is shown in the accompanying drawings is only one of the implementation methods of the invention. Therefore, if a person of ordinary skill in the art is inspired by it and designs a structural method and an implementation method similar to the technical solution without creativity without departing from the purpose of the invention, they should all fall within the scope of protection of this patent.

Claims

1. A complementary method for real-time and offline paper recommendation based on Hadoop ecosystem, characterized in that: The following steps are involved: Step S1: Use MapReduce, an offline computing engine component of the Hadoop framework, to read the paper vectorized label data in the data warehouse Hive, run the offline recommendation algorithm for offline computing, and obtain the offline recommendation results; Step S2: Use the distributed message queue Kafka to receive paper data in real time, read the paper data in Kafka through the Spark framework's stream computing engine Spark Streaming, read the paper data in Hive through the database engine Spark SQL, and then run the real-time recommendation algorithm for real-time computing to obtain real-time recommendation results; Step S3: merge the real-time calculation recommendation results into the offline calculation recommendation results and push them; Wherein, step S1 includes the following steps: Step S11: Build the data warehouse Hive in the Hadoop ecosystem, and use the ODS layer of Hive to store the original data of the paper; format and ETL the semi-structured paper data in the ODS layer, convert it into structured paper data, and store it in the DWD layer; Step S12: Extract the secondary classification number labels of papers in the DWD layer, construct the secondary classification number-paper weight matrix c_p, and store it in the DWS layer, where the weight w of the paper belonging to each secondary classification number pc The formula is: where f pc is the number of times paper p is marked as secondary classification code c, is the cumulative number of times n papers p are marked with the secondary classification number c. The total number of secondary classification number labels for each paper is specified to be 1. For papers with multiple secondary classification number labels, the number of secondary classification number labels is the frequency in the secondary classification number labels of the paper. Step S13: Extract the abstracts of the papers in the DWD layer and segment them to obtain abstract labels, construct the abstract-paper weight matrix a_p, and store it in the DWS layer. For each element f of the a_p matrix pa , which represents the weight of the abstract tag a in the paper p, the formula is: in is the number of abstract tags a in paper p, N pa is the number of abstract tags of paper p, N is the number of papers, and N a is the number of papers containing abstract label a; Step S14: Extract the titles of papers in the DWD layer and segment them to obtain title tags, construct the title-paper weight matrix t_p, and store it in the DWS layer. For each element f of the t_p matrix pt , which represents the weight of title tag t in paper p. If the title of paper p contains title tag t after word segmentation, then: Otherwise set f pt =0, where N pt is the number of title tags in paper p, N is the number of papers, and N t is the number of papers containing title tag t; Step S15: Extract keywords from papers in the DWD layer, construct a keyword-paper weight matrix k_p, and store it in the DWS layer. For each element f of the k_p matrix pk , which represents the weight of keyword k in paper p. If paper p contains keyword tag k, then: Otherwise set f pt =0, where N pk is the number of keywords in paper p, N is the number of papers, and N k is the number of papers containing keyword tag k; Step S16: Extract the relevant weight matrix of the abstract, title, keyword, and classification number of the DWS layer paper, f(p i , p j ; w) is the paper similarity, take the first N1 recommendations with similarity greater than T1, then filter the recommendations with threshold greater than T2 or the first N2, and then store them in the ADS layer. The formula is as follows: Among them, w1 is the weight of the paper title, w2 is the weight of the paper keywords, w3 is the weight of the paper abstract, and r is the adjustment factor, that is, the weighted weight of the secondary classification number, 0.01≤r≤0.1, is the title vector in paper i, is the keyword vector in paper i, is the abstract vector in paper i, For the paper i and p j The difference between the secondary classification numbers, is the similarity of paper elements, and the formula is as follows: When l=1, O is the title set T; when l=2, O is the keyword set K; when l=3, O is the abstract set A; Wherein, step S2 includes the following steps: Step S21: After building the distributed message publishing and subscription system Kafka, new paper data is written into Kafka using a data stream method; Spark Streaming collects a paper data set S from the Kafka message queue within a rolling window T time period, where the paper data set S includes paper s; Step S22: The weight w of each secondary classification number of the papers in the paper dataset S sc , the formula is: The numerator f sc is the number of times paper s is marked with secondary classification code c, the denominator is the cumulative number of secondary classification number labels c without considering the paper collection S. If the original paper collection does not contain classification number c, then w sc =1; Step S23: Construct the abstract-paper weight matrix a_s. For each element f of the a_s matrix sa , which represents the weight of the abstract label a in the paper s, and the formula is: in is the number of abstract tags a in paper s, N sa is the number of abstract tags of paper s, N is the number of papers, and N a is the number of papers containing abstract label a; Step S24: Construct the title-paper weight matrix t_s. For each element f of the t_s matrix st , which represents the weight of title tag t in paper s. If paper s contains title tag t, then: Otherwise set f st =0, where N st is the number of title tags in paper s, N is the number of papers, and N t is the number of papers containing title tag t; Step S25: Construct the keyword-paper weight matrix k_s. For each element f of the k_s matrix sk , which represents the weight of the abstract keyword k in paper s. If paper s contains keyword tag k, then: Otherwise set f sk =0, where N sk is the number of keywords in paper s, N is the number of papers, and N k is the number of papers containing keyword tag k; Step S26: f(p i ,s j ; w) is the paper similarity, take the first N1 recommendations with similarity greater than T1, then filter the recommendations with threshold greater than T2 or the first N2, and store them in the ADS layer. The formula is as follows: Among them, w1 is the weight of the paper title, w2 is the weight of the paper keywords, w3 is the weight of the paper abstract, 0≤w2≤w1≤w3≤1, w1+w2+w3=1, r is the adjustment factor, that is, the weighted weight of the secondary classification number, 0.01≤r≤0.1, is the title vector in paper i, is the keyword vector in paper i, is the abstract vector in paper i, For the paper i and j The difference between the secondary classification numbers, is the similarity of paper elements, and the formula is as follows: When l=1, O is the title set T; when l=2, O is the keyword set K; when l=3, O is the abstract set A.

2. The complementary method of real-time and offline paper recommendation based on Hadoop ecology according to claim 1, characterized in that: The T1 is the initial screening, which is constructed as a screening library, and T2 represents precise screening, T1≤T2.

Images