A personalized recommendation method based on big data

Abstract

The application discloses a kind of personalized recommendation method based on big data, the present application will elastic search data storage search and based on user's collaborative filtering algorithm and based on item's collaborative filtering recommendation algorithm is combined.For new user, recommend the top 15 data of highest click rate in elastic search database.For old user, the recommendation result of the user-based collaborative filtering algorithm and the item-based recommendation algorithm is linearly weighted fusion, and the top 10 data is selected for recommendation.The application can update the database regularly through the high automation of web crawler.Secondly, by combining TF-IDF algorithm with collaborative filtering algorithm, the sparsity problem of data can be alleviated to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a personalized recommendation method based on big data. BACKGROUND

[0002] With the rapid development of network technology, the Internet has spread to all aspects of life, and a large amount of data is generated every day. In the vast amount of information, the information that users want to obtain only accounts for a very small part of the total amount of information. In the face of the current situation of information overload, users often feel helpless. Therefore, how to enable users to efficiently obtain the information they want through a certain way is particularly important.

[0003] The commonly used recommendation methods at present mainly include content-based recommendation, association rule-based recommendation and collaborative filtering recommendation. The content-based recommendation and the collaborative filtering recommendation both have the problems of sparsity and new user, and the association rule-based recommendation has the problems of difficult rule extraction, time consumption and low degree of personalization. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the present application combines the elastic search data storage search and the user-based collaborative filtering algorithm and the item-based collaborative filtering recommendation algorithm. For new users, the top 15 data with the highest click rate in the elastic search database are recommended. For old users, the recommendation results of the user-based collaborative filtering algorithm and the item-based recommendation algorithm are linearly weighted and fused, and the top 10 data are selected for recommendation. The algorithm effectively improves the performance of the recommendation and to some extent alleviates the problems of sparsity and cold start. The technical scheme is as follows:

[0005] A personalized recommendation method based on big data, which comprises the following steps:

[0006] Step 1: Use the network crawler technology to specify the fields, including title, publication time, implementation time, timeliness, text, etc., to obtain data from related websites;

[0007] Step 2: The data obtained by crawling is first processed to remove meaningless data and de-duplicate the data, then new fields such as title and data-derived regions are added, and finally the cleaned data is saved in json format;

[0008] The data saved in json format is imported into the elastic search database, the ik tokenizer corresponding to the elasticsearch version is selected, and the most fine-grained ik_max_word is used to create an index; the least segmentation ik_smart is used for searching;

[0009] Step 3: Construct a user-based collaborative filtering algorithm, according to the user's historical behavior information on a certain data, including search, comment and collection, construct a user-data matrix U with size m x n, as follows:

[0010]

[0011] Where m represents the number of users, n represents the total number of data, if the user has historical behavior on a certain data, then score u mn m represents the score of user m on data n, if there is no historical behavior, then assign value 0;

[0012] According to the label information of the data, construct a data-label matrix C with size n x s, as follows:

[0013]

[0014] Where n represents the total number of data, s represents the total number of labels, c ns represents whether the data n contains the label s, if yes, then assign value 1, otherwise, assign value 0;

[0015] According to matrix U and matrix C, construct a user-label preference matrix P with size m x s, as follows:

[0016]

[0017] Where m represents the total number of users, s represents the total number of labels, p ms represents the preference degree of user m to label s;

[0018] Using TF-IDF algorithm, improve the user-label preference matrix P, as follows:

[0019]

[0020] Where p ua represents the preference degree of user u to label a, represents the number of times of user u marking label a, represents the total number of times of user u marking labels, represents the total number of times of labels, represents the total number of labels, n ua represents the number of users marking label a, n m represents the total number of users;

[0021] Using the improved user-label matrix, calculate the similarity between users using cosine similarity, the specific calculation formula is as follows:

[0022]

[0023] wherein w u,v The greater the value, the more similar the user u is to the user v, and thus the greater the probability that the user u will be recommended the data preferred by the user v; n u represents the set of data preferred by the user u, n v represents the set of data preferred by the user v; |n u ∩n v | represents the set of data preferred by both the user u and the user v;

[0024] The similarity is arranged in descending order, the first k users most similar to the target user are found, represented by the set S(u, k), and the preference degree of the target user u for the data i is calculated, and the specific calculation formula is as follows:

[0025]

[0026] wherein n i represents the set of users who have historical behavior information on the data i, u vi represents the score of the user v on the data i with historical behavior information;

[0027] Step 4, the recommendation result of the item-based collaborative filtering algorithm is constructed, which is basically the same as the implementation process of the user-based collaborative filtering algorithm in step 3: first, according to the historical behavior information of different users on different data, a data-user matrix is constructed; second, the cosine similarity is used to calculate the similarity between the data; third, the recommendation result is obtained, and the data recommended for the target user is the data which has no historical behavior of the target user and has relatively high similarity with the data which has historical behavior of the target user, and the higher the recommendation score, the more interested the target user is in the recommended data;

[0028] Step 5: for new users, the top 15 data with the highest click rate in the elastic search database are recommended; for old users, the recommendation results of the user-based collaborative filtering algorithm and the item-based recommendation algorithm are linearly weighted and fused, and the top 10 data are selected for recommendation.

[0029] Preferably, in step 1, different crawling objects are selected according to the different department attributes of the users, and relevant legal regulations, news, microblog, and case data are obtained.

[0030] Preferably, in step 3, the value standard of the user-data matrix U is: 1 for searching a data, 2 for commenting on a data, and 3 for collecting a data.

[0031] Preferably, in step 3, the first k users most similar to the target user are counted, and k is selected as 20.

[0032] Preferably, the recommendation method for new users in step 5 can also let users select their own interested fields during the initialization of platform design, and then select relevant field data for recommendation.

[0033] Preferably, step 6 can also add subsequent iterative optimization for the model, mainly divided into two parts: the first part is data optimization, based on the crawler program, constantly updating the database; the second part is the update of the user-data matrix, according to whether the target user has behavior feedback on the recommended data after recommending the data to the user with behavior information, and recording it in the user behavior information log table, so as to constantly update the user-data matrix and the final recommendation list.

[0034] Compared with the prior art, the beneficial effects of the present application are: first, through the high automation of the network crawler, the database can be updated regularly; second, by combining the TF-IDF algorithm with the collaborative filtering algorithm, the data sparsity problem can be alleviated to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a schematic diagram of the model implementation process.

[0036] Figure 2 It is a schematic diagram of the model optimization process. DETAILED DESCRIPTION

[0037] In order to clarify the technical solutions and working principles of the present application, the following will further describe the embodiments of the present disclosure in detail. All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0038] The terms "step 1", "step 2", "step 3" and the like in the specification and claims of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those described herein.

[0039] The embodiments of the present disclosure provide a personalized recommendation method based on big data, which comprises the following steps:

[0040] Step 1: Based on the crawler technology, relevant data is obtained.

[0041] Using network crawler technology, specify fields such as title, publication time, implementation time, timeliness, body, etc., to obtain data from relevant websites.

[0042] Preferably, different crawling objects are selected according to different department attributes of users, and relevant laws and regulations, news, microblog, and case data are obtained.

[0043] Step 2: The data is cleaned and stored in the elastic search database.

[0044] The crawled data is first processed in the field, the meaningless data is removed, and the data is de-duplicated, then the title and data derived region are derived, and finally the cleaned data is saved as a json format.

[0045] The data saved as a json format is imported into the elastic search database, the ik tokenizer corresponding to the elasticsearch version is selected, and the most fine-grained ik_max_word is used to create an index; the least segmentation ik_smart is used for searching. At the same time, the dictionary can also be customized and expanded to improve the accuracy of the search.

[0046] Step 3: Use the collaborative filtering algorithm, and combine the user's potential interest and the TF-IDF algorithm to build a user-label matrix.

[0047] According to the historical behavior information of a user on a certain data, including search, comment and collection, a user-data matrix U of size m x u is constructed, as follows:

[0048]

[0049] Wherein, m represents the number of users, n represents the total number of data, if the user has historical behavior on a certain data, the score is given, u mn represents the score of user m on data n, if there is no historical behavior, the value is 0.

[0050] Preferably, if a certain data is searched in this paper, the value is 1; if it is commented, the value is 2; if it is collected, the value is 3.

[0051] According to the label information of the data, a data-label matrix C of n x s is constructed, as follows:

[0052]

[0053] Wherein, n represents the total number of data, s represents the total number of labels, c ns represents whether the data n contains the label s, if it contains, the value is 1, otherwise, the value is 0.

[0054] According to the matrix U and the matrix C, a user-label preference matrix P of size m x s is constructed, as follows:

[0055]

[0056] wherein m represents the total number of users, s represents the total number of tags, p ms represents the preference degree of user m to tag s.

[0057] The TF-IDF algorithm is used to improve the user-tag preference matrix P, and the specific process is as follows:

[0058]

[0059] wherein p ua represents the preference degree of user u to tag a, represents the number of times that user u tags tag a, represents the total number of times that user u tags tags, represents the total number of times that tag a is tagged, represents the total number of tags, n ua represents the number of users who have tagged tag a, n m represents the total number of users.

[0060] The improved user-tag matrix is used to calculate the similarity between users by using cosine similarity, and the specific calculation formula is as follows:

[0061]

[0062] wherein w u,v The greater the value, the more similar user u is to user v, so the probability of recommending data preferred by user v to user u will be greater; n u represents the set of data preferred by user u, n v represents the set of data preferred by user v. |n u ∩n v | represents the set of data preferred by user u and v.

[0063] The similarity is arranged in descending order, and the top k users most similar to the target user are found, represented by set S(u, k), and the preference degree of target user u to data i is calculated, and the specific calculation formula is as follows:

[0064]

[0065] wherein n i represents the set of users who have historical behavior on data i, u vi represents the score of user v on data i, including search, comment and collection, which are respectively assigned values of 1, 2 and 3.

[0066] Preferably, k is selected as 20.

[0067] Step 4, the recommendation result of the item-based collaborative filtering algorithm can be basically consistent with the implementation process of the user-based collaborative filtering algorithm. First, according to the historical behavior information of different users on different data, a data-user matrix is constructed; second, the similarity between data is calculated using cosine similarity; third, the recommendation result is obtained, and the data recommended for the target user is the data that the target user has no historical behavior and has a relatively high similarity with the target user that has historical behavior, and the higher the recommendation score, the more interested the target user is in the recommended data.

[0068] Finally, the recommendation results of the user-based collaborative filtering algorithm and the item-based recommendation algorithm are linearly weighted and fused, and the top 10 data are selected for recommendation.

[0069] Step 5: Different recommendation algorithms are used according to the nature of the user.

[0070] For new users, the top 15 data with the highest click rate in the elastic search database can also be recommended in the platform design initialization, and a label is set, for example: let the user select the field of interest, and then select the relevant field data for recommendation. For old users, the recommendation results of the user-based collaborative filtering algorithm and the item-based recommendation algorithm are linearly weighted and fused, and the top 10 data are selected for recommendation.

[0071] Step 6: Iterative optimization of the model.

[0072] The iterative optimization of the model mainly includes two parts. The first part is the optimization of data, based on the crawler program, and the database is constantly updated; the second part is the update of the user-data matrix, as shown in Figure 2 After recommending data to users with behavior information, whether the target user has behavior feedback on the recommended data is recorded in the user behavior information log table, so as to constantly update the user-data matrix and the final recommendation list.

[0073] The above has exemplarily described the present application, and obviously, the specific implementation of the present application is not limited by the above method, and various non-essential improvements are made by adopting the method concept and technical solution of the present application; or without improvement, equivalent replacement, the above concept and technical solution of the present application are directly applied to other occasions, which are within the protection scope of the present application.

Claims

1. A personalized recommendation method based on big data, characterized in that, The method includes the following steps: Step 1: Use web crawling technology to obtain data from relevant websites by specifying fields, including: title, publication time, implementation time, timeliness, and body text; Step 2: First, process the fields of the crawled data, remove meaningless data and deduplicate the data. Then, derive new fields based on the title and data region. Finally, save the cleaned data in JSON format. Import the data saved in JSON format into the Elasticsearch database, select the IK tokenizer corresponding to the Elasticsearch version, and create an index using the finest granularity ik_max_word; perform the search using the least granular ik_smart. Step 3: Construct a user-based collaborative filtering algorithm. Based on users' historical behavior information regarding certain data, including searches, comments, and favorites, construct a user-data matrix U of size m×n, as shown below: Where m represents the number of users, n represents the total number of data points, and if a user has historical behavior related to a particular data point, a rating is assigned. mn This represents the rating of user m for data n. If there is no historical behavior, the value is 0. Construct an n×s data-label matrix C based on the label information of the data, as shown below: Where n represents the total number of data points, s represents the total number of tags, and c ns This indicates whether the data n contains the label s. If it does, assign a value of 1; otherwise, assign a value of 0. Based on matrices U and C, construct a user-label preference matrix P of size m×s, as shown below: Where m represents the total number of users, s represents the total number of tags, and p ms This indicates the degree of preference of user m for label s; The user-label preference matrix P is improved using the TF-IDF algorithm, as follows: Where, p ua This indicates the degree of preference user u has for label a. This indicates the number of times user u marks label a. This indicates the total number of times user u marked the label. Indicates the total number of times the label is displayed. n represents the total number of labels. ua n represents the number of users who have labeled 'a'. m This represents the total number of users; Using the improved user-tag matrix, cosine similarity is employed to calculate the similarity between users. The specific calculation formula is shown below: Among them, w u，v The larger the value, the more similar user u is to user v, and therefore the greater the probability of recommending data preferred by user v to user u; n u n represents the set of data that user u prefers. v Represents the set of data that user v prefers; |n u ∩n v | represents the set of data that users u and v both prefer; The similarity scores are sorted in descending order to find the top k users most similar to the target user, denoted by the set S(u,k). The preference of target user u for data i is then calculated using the following formula: Where, n i Let u represent the set of users who have historical behavior with data i. vi This indicates that user v has rated the historical behavior information of data i; Step 4, constructing the recommendation results of the project-based collaborative filtering algorithm, is basically the same as the implementation process of the user-based collaborative filtering algorithm in Step 3: First, construct a data-user matrix based on the historical behavior information of different users on different data; second, use cosine similarity to calculate the similarity between data; third, obtain the recommendation results, where the data recommended to the target user is data that the target user has not had any historical behavior with, and at the same time, the similarity to the data that the target user has had historical behavior with is relatively high. The higher the recommendation score, the more interested the target user is in the recommended data. Step 5: For new users, recommend the top 15 most clicked items from the Elastic Search database; for existing users, use a linear weighted fusion of the results from user-based collaborative filtering and item-based recommendation algorithms to select the top 10 items for recommendation.

2. The personalized recommendation method based on big data according to claim 1, characterized in that, In step 1, different crawling objects are selected based on the user's different department attributes, and relevant laws and regulations, news updates, microblogs, and case data are obtained.

3. The personalized recommendation method based on big data according to claim 1, characterized in that, In step 3, the assignment criteria for the user-data matrix U are as follows: if a data point has been searched, assign a value of 1; if it has been commented on, assign a value of 2; if it has been favorited, assign a value of 3.

4. The personalized recommendation method based on big data according to claim 3, characterized in that, In step 3, we count the top k users who are most similar to the target user, and we choose k = 20.

5. The personalized recommendation method based on big data according to claim 1, characterized in that, In step 5, the recommendation method for new users can also be implemented by allowing users to select their areas of interest during the platform's initial design, and then selecting relevant data from those areas for recommendation.

6. The personalized recommendation method based on big data according to claim 1, characterized in that, Step 6 can be added to perform subsequent iterative optimization of the model, which is divided into two parts: the first part is data optimization, which continuously updates the database based on the crawler program; the second part is updating the user-data matrix, which continuously updates the user-data matrix and the final recommendation list by recording whether the target user has any behavioral feedback on the recommended data after recommending data to users with behavioral information in the user behavior information log table.

Images