A recommendation method and system, and a storage medium
By using distributed computing and establishing a similarity dictionary, the problems of accuracy and efficiency in similarity calculation under sparse rating matrices are solved, resulting in more efficient recommendation performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILEHANGZHOUINFORMATION TECH CO LTD
- Filing Date
- 2021-08-20
- Publication Date
- 2026-08-04
AI Technical Summary
Existing similarity algorithms have poor accuracy in sparse rating matrices and cannot be parallelized, resulting in low computational efficiency and making it difficult to build large-scale recommendation models.
By scanning user/item ratings one by one, recording median similarity values, performing distributed computation, and building a similarity dictionary, storage space is reduced and computational efficiency and accuracy are improved.
Parallel computation of similarity was achieved, improving computational efficiency and accuracy, and enhancing the feasibility of large-scale data recommendation models.
Smart Images
Figure CN115712761B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data service technology, and in particular to a recommendation method and system, as well as a storage medium. Background Technology
[0002] Similarity algorithms are the core of recommendation models. Currently, commonly used similarity algorithms mainly include Euclidean distance similarity, cosine similarity, and Pearson similarity. However, all of these similarity algorithms require all ratings of two objects to be calculated as vectors. When the rating matrix is relatively sparse, that is, when there are few cases where two items or two users are rated at the same time, a large number of single user or single item ratings will cause similarity bias, resulting in poor accuracy of similarity calculation. At the same time, commonly used similarity algorithms not only require a large amount of memory to store the rating vector matrix, but also cannot achieve parallel computing, which greatly reduces computational efficiency and the feasibility of building recommendation models using large-scale data. Summary of the Invention
[0003] This application provides a recommendation method and system, as well as a storage medium, which can improve the accuracy and efficiency of similarity calculation, thereby achieving better recommendation results and improving the feasibility of the recommendation model.
[0004] The technical solution of this application embodiment is implemented as follows:
[0005] In a first aspect, embodiments of this application provide a recommended method, the method comprising:
[0006] Determine the data set based on the original data;
[0007] The similarity parameters are obtained by performing distributed computation processing on the dataset.
[0008] A similarity dictionary is obtained based on the similarity parameters;
[0009] Recommendation processing is performed based on the similarity dictionary.
[0010] Secondly, embodiments of this application provide a recommendation system, which includes a determining unit, an acquiring unit, and a recommending unit.
[0011] The determining unit is used to determine the data set based on the original data;
[0012] The acquisition unit is used to perform distributed computing processing based on the dataset to obtain similarity parameters; and to obtain a similarity dictionary based on the similarity parameters.
[0013] The recommendation unit is used to perform recommendation processing based on the similarity dictionary.
[0014] Thirdly, embodiments of this application provide a recommendation system, which further includes a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the recommendation method described above is implemented.
[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program applied in a recommendation system, wherein when the program is executed by a processor, it implements the recommendation method as described above.
[0016] This application provides a recommendation method and system, as well as a storage medium. The recommendation system determines a data set based on original data; performs distributed computation processing on the data set to obtain similarity parameters; obtains a similarity dictionary based on the similarity parameters; and performs recommendation processing based on the similarity dictionary. In other words, in this application's embodiments, a data set can be determined based on original data, then distributed computation processing can be performed on the data set to obtain similarity, and a similarity dictionary can be obtained based on the similarity obtained after distributed computation processing. Finally, recommendation processing is performed based on the similarity dictionary. Therefore, the recommendation system achieves parallel computation of similarity by obtaining similarity through distributed computation processing of the data set, and then builds a similarity dictionary based on the similarity, thus improving computational efficiency. Furthermore, the recommendation system effectively improves the accuracy of similarity calculation by building a similarity dictionary based on the similarity, thereby ultimately enabling recommendation processing using the similarity dictionary, obtaining better recommendation results, and improving the feasibility of the recommendation model. Attached Figure Description
[0017] Figure 1 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 1 ;
[0018] Figure 2 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 2 ;
[0019] Figure 3 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 3 ;
[0020] Figure 4 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 4 ;
[0021] Figure 5 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 5 ;
[0022] Figure 6 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 6 ;
[0023] Figure 7 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 7 ;
[0024] Figure 8 This is a schematic diagram illustrating the implementation of the recommended method proposed in the embodiments of this application;
[0025] Figure 9 This is a schematic diagram of the composition structure of the recommendation system proposed in the embodiments of this application. Figure 1 ;
[0026] Figure 10 This is a schematic diagram of the composition structure of the recommendation system proposed in the embodiments of this application. Figure 2 . Detailed Implementation
[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.
[0028] Similarity algorithms are the core of recommendation models. Currently, the three most commonly used similarity algorithms are Euclidean distance similarity, cosine similarity, and Pearson similarity. Euclidean distance is the linear distance in Euclidean space, and it is usually used in two-dimensional space to represent the similarity of features between two objects. Euclidean distance can be expressed by the following formula:
[0029]
[0030] Similarity values range from 0 to 1, with higher similarity indicating closer distances. Therefore, Euclidean distance similarity is defined by the following formula:
[0031]
[0032] Cosine similarity is calculated by taking the cosine of the angle between two vectors, and it characterizes the similarity between the two vectors. If the angle is 90 degrees, the similarity is 0; if the two vectors have the same direction, the similarity is 1. The cosine similarity between two vectors is defined by the following formula:
[0033]
[0034] Where ||X|| and ||Y|| represent the 2-norm of vectors X and Y, respectively.
[0035] The Pearson correlation coefficient is also an important metric for measuring the correlation between two vectors, and it is defined by the following formula:
[0036]
[0037] Since the Pearson correlation coefficient ranges from -1 to 1, it is often processed as follows to normalize the similarity to between 0 and 1:
[0038] Simi x,y =0.5+0.5ρ x,y (5)
[0039] However, existing technologies cannot overcome issues such as data storage and operational efficiency, nor can they overcome the similarity calculation bias caused by sparse matrices. Firstly, all three existing recommendation methods require all ratings for two objects to be calculated as vectors. This approach not only requires a large amount of memory to store the rating vector matrix but also prevents parallel computation, significantly reducing computational efficiency and the feasibility of building recommendation models with large-scale data. Secondly, the aforementioned recommendation methods use all rating data to characterize the similarity between two objects, treating missing values (i.e., no rating) as 0. When the rating matrix is sparse, meaning there are few instances where two items or two users are rated simultaneously, the large number of single-user / single-item reviews will cause similarity bias.
[0040] To address the problems of existing recommendation methods, this application implements a parallel calculation of similarity by scanning all user / item ratings one by one and recording the median similarity between users / items. Furthermore, it only considers cases where users jointly rate two items, or where an item is simultaneously rated by two users—that is, it considers the probability of two objects being rated simultaneously—which better aligns with the original intent of calculating rating similarity, thus resulting in higher accuracy for the recommendation model. In addition, this solution stores the similarity matrix in the form of a similarity dictionary instead of the rating vector matrix, significantly reducing storage space, improving query efficiency, and enhancing the feasibility of large-scale data recommendation models.
[0041] Specifically, this application provides a recommendation method and system, as well as a storage medium. The recommendation system determines a data set based on the original data; performs distributed computing processing on the data set to obtain similarity parameters; obtains a similarity dictionary based on the similarity parameters; and performs recommendation processing based on the similarity dictionary. This can improve the accuracy and computational efficiency of similarity calculation.
[0042] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0043] Example 1
[0044] This application provides a recommendation method, which includes the following steps for a recommendation system to calculate similarity:
[0045] Step 101: Determine the data set based on the original data.
[0046] In the embodiments of this application, when the recommendation system performs similarity calculation, it can first determine the data set based on the original data.
[0047] It should be noted that, in the embodiments of this application, the original data refers to the rating data of users on items; there can be multiple users and items; that is, the original data includes ratings of different items by different users.
[0048] Furthermore, in the embodiments of this application, the original data can be processed according to a preset data type; for example, the original data is data in the format of Resilient Distributed Dataset (RDD); firstly, the original data in RDD format can be processed into a data type with ((User, item)score) as the primary key, so that the corresponding score value can be quickly obtained through the primary key.
[0049] Furthermore, in the embodiments of this application, after processing the original data according to a preset data type, the original data can also be averaged. For example, if there are multiple ratings of the same item by a user in the original data, the average of the ratings for the item is taken. For example, if user A has rated item P three times, with ratings of 7, 7, and 8 respectively, then user A's rating for item P is 22 / 3.
[0050] It should be noted that, in the embodiments of this application, the data set is determined based on the original data. The determination of the data set can include two methods: one is to integrate and process the original data based on the item dimension in the original data to obtain a first data set; the other is to integrate and process the original data based on the user dimension in the original data to obtain a second data set. Furthermore, the data set is determined by one of the above two methods, that is, the data set can be either the first data set or the second data set.
[0051] Furthermore, in the embodiments of this application, when determining the data set based on the original data, it is necessary to first determine the basis of the original data, and then obtain the corresponding data set based on the basis; wherein, the basis can be the user dimension or the item dimension in the original data.
[0052] Furthermore, in the embodiments of this application, the smaller of the number of users and the number of items in the original data is used as the basis; for example, in the original data, the number of items is 3,000 and the number of users is 40,000. Since the number of items is less than the number of users, the item dimension is determined as the basis, thereby performing integration processing to obtain the second data set.
[0053] Furthermore, in the embodiments of this application, the integration process refers to grouping the data based on the new primary key and integrating the original data in the form of a list; for example, if the item is determined as the base, the new primary key is (user x, (the user's rating list for all items)); if the user is determined as the base, the new primary key is (item x, (the rating list for the item by all users)).
[0054] It is understood that, in the embodiments of this application, the first data set and the second data set are different forms of rating lists obtained after integration processing; that is, the first data set can be a rating list of all items by a specific user; the second data set can be a rating list of a specific item by all users; for example, in the original data, users include A, B, and C; items include 1, 2, and 3; then the first data set can include the rating data of A, B, and C for 1, the rating data of A, B, and C for 2, and the rating data of A, B, and C for 3; while the second data set can include the rating data of A for 1, 2, and 3, the rating data of B for 1, 2, and 3, and the rating data of C for 1, 2, and 3.
[0055] Step 102: Perform distributed computation processing based on the dataset to obtain similarity parameters.
[0056] In the embodiments of this application, after determining the data set based on the original data, the recommendation system can perform distributed computing processing based on the data set to obtain similarity parameters.
[0057] It should be noted that, in the embodiments of this application, distributed computing processing refers to performing multi-line parallel computing. That is, after obtaining the data set, multi-line parallel computing is performed using the data set to obtain similarity parameters, thereby effectively improving computing efficiency through distributed computing processing.
[0058] For example, in the embodiments of this application, based on the above example, the data set determined according to the original data is a second data set. The second data set includes the rating data of A for 1, 2, and 3, the rating data of B for 1, 2, and 3, and the rating data of C for 1, 2, and 3. The rating data of A for 1, 2, and 3 can be placed on host a for similarity parameter calculation, the rating data of B for 1, 2, and 3 can be placed on host b for similarity parameter calculation, and the rating data of C for 1, 2, and 3 can be placed on host c for similarity parameter calculation; thereby improving the computational efficiency through multi-line parallel computation.
[0059] Furthermore, in the embodiments of this application, the distributed computing processing is based on the map function, that is, the similarity parameter is obtained by using the map function and the distributed computing framework.
[0060] For example, in an embodiment of this application, the dataset is a second dataset, and the similarity can be expressed by the following formula:
[0061]
[0062] in, Let r be the similarity parameter between the ratings of user u, item i, and item j. ui For user u's rating of item i, r uj For user u, rate item j.
[0063] It is understood that, in the embodiments of this application, based on the above example, the more consistent the user u's ratings of two items i and j, the higher the similarity between item i and item j.
[0064] Step 103: Obtain the similarity dictionary based on the similarity parameters.
[0065] In the embodiments of this application, after the recommendation system performs distributed computing processing based on the dataset to obtain similarity parameters, a similarity dictionary can be obtained based on the similarity parameters.
[0066] It should be noted that in the embodiments of this application, the similarity dictionary is determined based on similarity, and the similarity is obtained through distributed computing. Therefore, the similarity dictionary can be obtained through cross-partition processing.
[0067] For example, in the embodiments of this application, based on the above example, the data set is a first data set; the different similarities obtained on hosts a, b, and c are summarized, and then a similarity dictionary is obtained based on these similarities; that is, a similarity dictionary is obtained through cross-partition processing.
[0068] Specifically, in the embodiments of this application, a similarity dictionary is obtained based on the similarity parameters. The similarity parameters can be summed to obtain the total similarity. Then, the number of evaluations corresponding to the similarity parameters can be summed to obtain the total number of evaluations. Finally, the total similarity and the total number of evaluations can be divided to obtain the similarity dictionary.
[0069] Furthermore, in the embodiments of this application, based on the above examples, the similarity dictionary can be represented by the following formula:
[0070]
[0071] Where, ρ ij p represents the similarity obtained based on the ratings of item i and item j by all users simultaneously; ij S represents the probability that all users simultaneously rate item i and item j; u The set of items for which user u is rated, where U is the set of users. The sum of similarity. The total number of users who rated item i and item j.
[0072] Furthermore, in the embodiments of this application, a similarity dictionary can be obtained using a reduce operation.
[0073] Step 104: Perform recommendation processing based on the similarity dictionary.
[0074] In the embodiments of this application, after the recommendation system obtains a similarity dictionary based on the similarity parameters, recommendation processing can be performed based on the similarity dictionary.
[0075] It is understood that in the embodiments of this application, the similarity dictionary can reflect the similarity between items; therefore, item recommendations can be made based on the similarity dictionary, i.e., recommendation processing can be implemented.
[0076] Specifically, in the embodiments of this application, when performing recommendation processing using a similarity dictionary, the evaluated set and the unevaluated set can be obtained first; then, the score of the unevaluated set can be obtained based on the similarity dictionary and the evaluated set; finally, recommendation processing can be performed based on the score of the unevaluated set.
[0077] For example, in an embodiment of this application, when recommending items to user A, the rated set and the unrated set of user A are first obtained. Since the items in the unrated set have no rating, it is impossible to determine the degree of user A's liking for the items in the unrated set. Therefore, the rating of the unrated set can be obtained first using a similarity dictionary and the rated set, and then the recommendation process can be performed to user A by combining the rating of the unrated set.
[0078] Furthermore, in the embodiments of this application, when obtaining the score of the unrated set based on the similarity dictionary and the rated set, the score of the unrated set can be obtained by the following formula:
[0079]
[0080] in, The rating of unrated items in the unrated set, r k Simi rates the items already rated in the rated collection. ki Let be the similarity between item i and item k.
[0081] It is understood that, in the embodiments of this application, the principle of obtaining the rating of the unrated set based on the above formula (8) is that when the recommender has not rated item i, the rating of item i by other users who have rated item i can be used to obtain the similarity between item i and item j that the recommender has rated. Thus, based on the similarity between item i and item j, and the rating of item j by the recommender, the rating of the user relative to item i can be obtained.
[0082] Furthermore, in the embodiments of this application, the recommendation process involves sorting the items by rating from high to low and then recommending the top N items; where N can be any number, and this application does not impose any restrictions.
[0083] Figure 2 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 2 ,like Figure 2 As shown, the recommendation system performs distributed computation processing based on the dataset to obtain similarity parameters. Step 102 may include the following steps:
[0084] Step 102a: Calculate the ratio between a user's ratings of different items in the first dataset using a preset algorithm to obtain the first similarity.
[0085] In the embodiments of this application, the recommendation system performs distributed computing processing based on the data set to obtain similarity parameters. Specifically, the recommendation system can first use a preset algorithm to calculate the ratio between a user's ratings of different items in the first data set to obtain a first similarity.
[0086] It is understood that in the embodiments of this application, the preset algorithm refers to the map function.
[0087] Furthermore, in the embodiments of this application, when the data set is a first data set, the first similarity refers to the ratio between a user's ratings of different items in the first data set.
[0088] For example, in the embodiments of this application, the users include A, B, and C, and the items include 1, 2, and 3. The first similarity is calculated using the above formula (6). The first similarity includes the similarity between any two of A's ratings of items 1, 2, and 3, the similarity between any two of B's ratings of items 1, 2, and 3, and the similarity between any two of C's ratings of items 1, 2, and 3.
[0089] Step 102b: Perform minimum value processing on the first similarity to obtain the similarity parameter.
[0090] In the embodiments of this application, after the recommendation system uses a preset algorithm to calculate the ratio between a user's ratings of different items in the first data set to obtain a first similarity, the first similarity can be minimized to obtain a similarity parameter.
[0091] It is understood that, in the embodiments of this application, minimum value processing refers to taking the minimum value of the first similarity; the calculation formula for the first similarity is as shown in the aforementioned formula (6), r ui and r uj To rate any two items, and to ensure that the similarity values are between 0 and 1, the minimum value of the first similarity score needs to be taken.
[0092] Figure 3 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 3 ,like Figure 3 As shown, the recommendation system performs distributed computation processing based on the dataset to obtain similarity parameters. Therefore, step 102 may also include the following steps:
[0093] Step 102c: Using a preset algorithm, calculate the ratio between different ratings of an item by different users for that item in the second data set to obtain the second similarity.
[0094] In the embodiments of this application, the recommendation system performs distributed computing processing based on the data set to obtain similarity parameters. Specifically, the recommendation system can also first use a preset algorithm to calculate the ratio between different ratings of an item by different users for an item in the second data set to obtain a second similarity.
[0095] It should be noted that, in the embodiments of this application, the second similarity refers to the similarity obtained by calculating the ratio between different ratings of an item by different users for an item in the second data set; that is, the second similarity can characterize the similarity between different users for the same item.
[0096] It is understood that, in the embodiments of this application, since the second similarity can characterize the similarity between different users for the same item, when obtaining the similarity dictionary based on the similarity parameter, if the similarity dictionary is obtained based on the second similarity, then the similarity dictionary can describe the similarity between different users. Thus, when using the similarity dictionary to perform recommendation processing on the recommended object, other users' favorite items that are highly similar to the recommended object can be recommended to the recommended object to achieve recommendation processing.
[0097] Step 102d: Perform minimum value processing on the second similarity to obtain the similarity parameter.
[0098] In the embodiments of this application, the recommendation system calculates the ratio between different ratings of an item by different users for an item in the second data set using a preset algorithm to obtain a second similarity. Then, the second similarity can be minimized to obtain a similarity parameter.
[0099] It is understood that, in the embodiments of this application, in order to ensure that the range of the second similarity value is between 0 and 1, it is necessary to take the minimum value of the second similarity.
[0100] Figure 4 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 4 ,like Figure 4 As shown, obtaining a similarity dictionary based on similarity parameters, i.e., the method proposed in step 103, may include the following steps:
[0101] Step 103a: Sum the similarity parameters to obtain the total similarity.
[0102] In the embodiments of this application, the recommendation system obtains a similarity dictionary based on similarity parameters. Specifically, the recommendation system can perform a summation operation on the similarity parameters to obtain the total similarity.
[0103] It is understood that, in the embodiments of this application, for example, as shown in the aforementioned formula (7), when the similarity is the first similarity, the total similarity is obtained by summing the similarity parameters.
[0104] Step 103b: Sum the number of evaluations corresponding to the similarity parameters to obtain the total number of evaluations.
[0105] Furthermore, in the embodiments of this application, the recommendation system obtains a similarity dictionary based on the similarity parameters. Specifically, the total number of evaluations corresponding to the similarity parameters can be obtained by summing the results.
[0106] It is understood that, in the embodiments of this application, for example, as shown in the aforementioned formula (7), when the similarity is the first similarity, the summation operation is performed on the number of evaluations corresponding to the similarity parameter to obtain the total number of evaluations.
[0107] Step 103c: Perform a division operation on the total similarity and the total number of evaluations to obtain a similarity dictionary.
[0108] In the embodiments of this application, after the recommendation system sums the similarity parameters to obtain the total similarity and sums the number of evaluations corresponding to the similarity parameters to obtain the total number of evaluations, it can perform a division operation on the total similarity and the total number of evaluations to obtain a similarity dictionary.
[0109] It is understood that, in the embodiments of this application, exemplarily, as shown in the aforementioned formula (7), when the similarity is the first similarity; for the total similarity and total number of evaluations Perform division to obtain the similarity dictionary Simi i,j .
[0110] Figure 5 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 5 ,like Figure 5 As shown, after obtaining the similarity dictionary based on the similarity parameters, i.e., after step 103, the method for the recommendation system to calculate similarity may also include the following steps:
[0111] Step 105: Obtain incremental data.
[0112] In the embodiments of this application, after obtaining the similarity dictionary based on the similarity parameters, the recommendation system can also obtain incremental data.
[0113] It should be noted that, in the embodiments of this application, incremental data can also be called online data, that is, newly generated online data in the recommendation system.
[0114] It is understood that, in the embodiments of this application, the incremental data may include newly added users, items, and user ratings of items.
[0115] For example, in an embodiment of this application, the incremental data may be a new user D and user D's rating of the item.
[0116] Step 106: Obtain the incremental similarity dictionary based on the incremental data.
[0117] In the embodiments of this application, after the recommendation system obtains incremental data, an incremental similarity dictionary can be obtained based on the incremental data.
[0118] It is understood that, in the embodiments of this application, after obtaining the similarity dictionary, the recommendation system can first store the similarity dictionary, and then obtain the incremental similarity dictionary based on the incremental data after obtaining the incremental data.
[0119] Specifically, in the embodiments of this application, the incremental similarity dictionary can be calculated based on incremental data, and the calculation method is the same as the aforementioned similarity calculation method, the only difference being that the calculated data is incremental data; for example, the incremental similarity and the number of increments can be calculated first based on the incremental data; the incremental similarity can be expressed as: Where U1 is the newly added user dataset, i.e., incremental data; the number of increments is... in, This is the set of user ratings corresponding to the incremental data.
[0120] Step 107: Update the similarity dictionary based on the incremental similarity dictionary.
[0121] In the embodiments of this application, after the recommendation system obtains incremental similarity based on incremental data, the similarity dictionary can be updated based on the incremental similarity.
[0122] It is understood that, in the embodiments of this application, the similarity dictionary can be updated based on the incremental similarity dictionary; specifically, the incremental similarity dictionary can be merged with the original similarity dictionary to obtain an updated similarity dictionary.
[0123] For example, in an embodiment of this application, the updated similarity dictionary can be represented by the following formula:
[0124]
[0125] Figure 6 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 6 ,like Figure 6 As shown, the method for recommendation processing based on the similarity dictionary, i.e., the method proposed in step 104, may include the following steps:
[0126] Step 104a: Obtain the evaluated set and the unevaluated set.
[0127] In the embodiments of this application, the recommendation system performs recommendation processing based on a similarity dictionary. Specifically, the recommendation system may first obtain the evaluated set and the unevaluated set.
[0128] For example, in an embodiment of this application, the rated set may be a set of items that the recommended object has rated, while the unrated set may be a set of items that the recommended object has not rated; the unrated set may include at least one unrated item.
[0129] Step 104b: Obtain the score of the unrated set based on the similarity dictionary and the rated set.
[0130] In the embodiments of this application, after the recommendation system obtains the rated set and the unrated set, it can obtain the rating of the unrated set based on the similarity dictionary and the rated set.
[0131] It is understood that in the embodiments of this application, the score of the unrated set is obtained based on the similarity dictionary and the rated set; for example, the score of the unrated set can be calculated by the aforementioned formula (8).
[0132] Step 104c: Perform recommendation processing based on the ratings of the unrated set.
[0133] In the embodiments of this application, after the recommendation system obtains the rating of the unrated set based on the similarity dictionary and the rated set, it can perform recommendation processing based on the rating of the unrated set.
[0134] It should be noted that, in the embodiments of this application, after obtaining the ratings of the unrated set, the recommendation system can sort all the item ratings from high to low, and recommend the items corresponding to the top N ratings as recommended items.
[0135] Figure 7 A schematic diagram of the implementation process of the recommended method proposed in the embodiments of this application. Figure 7 ,like Figure 7 As shown, the recommendation system obtains the rating of the unrated set based on the similarity dictionary and the already rated set. That is, step 104b may include the following steps:
[0136] Step 201: Query the similarity dictionary to obtain the recommended similarity between the evaluated set and the unevaluated set, and the rating of the evaluated set.
[0137] In the embodiments of this application, the recommendation system obtains the score of the unrated set based on the similarity dictionary and the rated set. Specifically, the recommendation system can first perform query processing on the similarity dictionary to obtain the recommendation similarity between the rated set and the unrated set, and the score of the rated set.
[0138] It is understood that, in the embodiments of this application, the recommended similarity between the evaluated set and the unevaluated set, and the rating of the evaluated set can be obtained by querying the similarity dictionary.
[0139] For example, in an embodiment of this application, as shown in the aforementioned formula (8), the recommended similarity can be Simi. ki The rating of the evaluated set can be r.k .
[0140] Step 202: Obtain the score of the unrated set based on the recommendation similarity and the score of the rated set.
[0141] In the embodiments of this application, the recommendation system queries the similarity dictionary to obtain the recommendation similarity between the evaluated set and the unevaluated set, and the score of the evaluated set. Then, it can obtain the score of the unevaluated set based on the recommendation similarity and the score of the evaluated set.
[0142] It is understood that, in the embodiments of this application, the score of the unrated set can be obtained based on the recommendation similarity and the score of the rated set.
[0143] For example, in an embodiment of this application, the score of the unrated set can be calculated according to the aforementioned formula (8).
[0144] This application provides a recommendation method in which the recommendation system determines a data set based on original data; performs distributed computation processing on the data set to obtain similarity parameters; obtains a similarity dictionary based on the similarity parameters; and performs recommendation processing based on the similarity dictionary. In other words, in this application's embodiment, a data set can be determined based on original data, then distributed computation processing can be performed on the data set to obtain similarity, and a similarity dictionary can be obtained based on the similarity obtained after distributed computation processing. Finally, recommendation processing is performed based on the similarity dictionary. Therefore, the recommendation system achieves parallel computation of similarity by obtaining similarity through distributed computation processing of the data set, and then builds a similarity dictionary based on the similarity, thus improving computational efficiency. Furthermore, the recommendation system effectively improves the accuracy of similarity calculation by building a similarity dictionary based on the similarity, thereby ultimately enabling recommendation processing using the similarity dictionary, obtaining better recommendation results, and improving the feasibility of the recommendation model.
[0145] Example 2
[0146] For example, in another embodiment of this application, based on a collaborative filtering algorithm, Figure 8 This is a schematic diagram illustrating the implementation of the recommended method proposed in the embodiments of this application, such as... Figure 8 As shown, by creating and updating a similarity dictionary, and then using the recommendation process implemented by the similarity dictionary, the recommendation results can be fed back to online or offline data.
[0147] For example, in the embodiments of this application, after obtaining the original data, i.e. the existing data, the data set can be determined first based on the original data; specifically, the original data in RDD format is processed into a data type with ((User, item) score) as the primary key, and the original data is averaged.
[0148] Furthermore, in the embodiments of this application, after the above processing, it is necessary to determine the basis of the original data, and then determine the data set based on the basis; the basis can be one of the two dimensions of items and users, and the basis is the smaller of the number of users and the number of items in the original data.
[0149] For example, in the embodiments of this application, the data set is determined based on the item dimension and the original data, i.e., the first data set, for the subsequent creation of the similarity dictionary.
[0150] Specifically, in the embodiments of this application, the first data set needs to be processed by distributed computing to obtain the first similarity. The calculation method of the first similarity is as shown in the aforementioned formula (6).
[0151] Furthermore, in the embodiments of this application, the distributed computing processing is based on the map function, that is, the first similarity is obtained by using the map function and the distributed computing framework.
[0152] Therefore, in the embodiments of this application, the first similarity can be obtained through a large amount of parallel computing, thereby improving the similarity calculation speed and thus increasing the creation speed of the similarity dictionary.
[0153] Furthermore, in the embodiments of this application, a similarity dictionary can be obtained based on the first similarity. The similarity dictionary is calculated as shown in the aforementioned formula (7), that is, the first similarity is summed to obtain the total similarity of the first similarity; then the number of evaluations corresponding to the first similarity is summed to obtain the total number of evaluations of the first similarity; finally, the number of evaluations of the first similarity and the total number of evaluations of the first similarity are divided to obtain the similarity dictionary.
[0154] Furthermore, in the embodiments of this application, the similarity dictionary can also be updated based on the incremental data. Similarly, after obtaining the incremental data, basic preprocessing can be performed on the incremental data, including adjusting the data type and mean value processing, etc.; then, the incremental similarity dictionary corresponding to the incremental data is determined based on the incremental data, and the similarity dictionary is updated based on the incremental similarity dictionary.
[0155] It is understood that, in the embodiments of this application, since the original similarity dictionary has already been stored, the similarity dictionary can be updated based on the original similarity dictionary, thus eliminating the need for secondary calculation of the original similarity dictionary and providing feasibility for combining offline and online recommendations.
[0156] Furthermore, in the embodiments of this application, after obtaining the similarity dictionary, recommendation processing can be implemented based on the similarity dictionary;
[0157] Specifically, in the embodiments of this application, the evaluated set and the unevaluated set can be obtained first; then the score of the unevaluated set can be obtained according to the similarity dictionary and the evaluated set; finally, the recommendation process can be performed according to the score of the unevaluated set. The score of the unevaluated set can be calculated by the aforementioned formula (8).
[0158] Specifically, in the embodiments of this application, after obtaining the ratings of the unrated set, the top N items with the highest ratings can be used to generate a recommendation list, thereby achieving recommendation processing.
[0159] This application provides a recommendation method in which the recommendation system determines a data set based on original data; performs distributed computation processing on the data set to obtain similarity parameters; obtains a similarity dictionary based on the similarity parameters; and performs recommendation processing based on the similarity dictionary. In other words, in this application's embodiment, a data set can be determined based on original data, then distributed computation processing can be performed on the data set to obtain similarity, and a similarity dictionary can be obtained based on the similarity obtained after distributed computation processing. Finally, recommendation processing is performed based on the similarity dictionary. Therefore, the recommendation system achieves parallel computation of similarity by obtaining similarity through distributed computation processing of the data set, and then builds a similarity dictionary based on the similarity, thus improving computational efficiency. Furthermore, the recommendation system effectively improves the accuracy of similarity calculation by building a similarity dictionary based on the similarity, thereby ultimately enabling recommendation processing using the similarity dictionary, obtaining better recommendation results, and improving the feasibility of the recommendation model.
[0160] Example 3
[0161] Based on the above embodiments, in another embodiment of this application... Figure 9 This is a schematic diagram of the composition structure of the recommendation system proposed in the embodiments of this application. Figure 1 ,like Figure 9 As shown, the recommendation system 10 proposed in this application embodiment may include a determining unit 11, an obtaining unit 12, a recommending unit 13, and an updating unit 14.
[0162] The determining unit 11 is used to determine the data set based on the original data.
[0163] The acquisition unit 12 is used to perform distributed computing processing based on the data set to obtain similarity parameters; and to obtain a similarity dictionary based on the similarity parameters.
[0164] The recommendation unit 13 is used to perform recommendation processing based on the similarity dictionary.
[0165] Furthermore, the determining unit 11 is specifically used to integrate and process the original data based on the user dimension in the original data to obtain a first data set; or, based on the item dimension in the original data, to integrate and process the original data to obtain a second data set.
[0166] Furthermore, the acquisition unit 12 is specifically used to calculate the ratio between a user's ratings of different items in the first data set using a preset algorithm to obtain a first similarity; and to perform minimum value processing on the first similarity to obtain the second similarity.
[0167] Furthermore, the acquisition unit 12 is specifically used to calculate the ratio between different ratings of an item by different users for the same item in the second data set using a preset algorithm to obtain a second similarity; and to perform minimum value processing on the second similarity to obtain the similarity.
[0168] Furthermore, the acquisition unit 12 is specifically used to perform a summation operation on the similarity parameters to obtain the total similarity; to perform a summation operation on the number of evaluations corresponding to the similarity parameters to obtain the total number of evaluations; and to perform a division operation on the total similarity and the total number of evaluations to obtain the similarity dictionary.
[0169] Furthermore, the acquisition unit 12 is also used to acquire incremental data after obtaining the similarity dictionary based on the similarity parameters; and to obtain an incremental similarity dictionary based on the incremental data.
[0170] Furthermore, the update unit 14 is used to update the similarity dictionary according to the incremental similarity dictionary.
[0171] Furthermore, the recommendation unit 13 is specifically used to obtain the evaluated set and the unevaluated set; to obtain the score of the unevaluated set according to the similarity dictionary and the evaluated set; and to perform the recommendation process according to the score of the unevaluated set.
[0172] Furthermore, the acquisition unit 12 is specifically used to perform query processing on the similarity dictionary to obtain the recommended similarity between the evaluated set and the unevaluated set, and the rating of the evaluated set; and to obtain the rating of the unevaluated set based on the recommended similarity and the rating of the evaluated set.
[0173] Figure 10 This is a schematic diagram of the composition structure of the recommendation system proposed in the embodiments of this application. Figure 2 ,like Figure 10As shown, the recommendation system 10 proposed in this application embodiment may further include a processor 15, a memory 16 storing instructions executable by the processor 15, and further, the recommendation system 10 may also include a communication interface 17 and a bus 18 for connecting the processor 15, the memory 16 and the communication interface 17.
[0174] In the embodiments of this application, the processor 15 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and this application embodiment does not specifically limit it. The processor 15 may also include a memory 16, which can be connected to the processor 15. The memory 16 is used to store executable program code, which includes computer operation instructions. The memory 16 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.
[0175] In embodiments of this application, bus 18 is used to connect communication interface 17, processor 15, and memory 16, as well as the mutual communication between these devices.
[0176] In embodiments of this application, memory 16 is used to store instructions and data.
[0177] Furthermore, in an embodiment of this application, the processor 15 is used to determine a data set based on the original data;
[0178] The similarity parameters are obtained by performing distributed computation processing on the dataset.
[0179] A similarity dictionary is obtained based on the similarity parameters;
[0180] Recommendation processing is performed based on the similarity dictionary.
[0181] In practical applications, the aforementioned memory 16 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 15.
[0182] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.
[0183] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0184] This application provides a recommendation system. The system determines a data set based on original data; performs distributed computation processing on the data set to obtain similarity parameters; obtains a similarity dictionary based on the similarity parameters; and performs recommendation processing based on the similarity dictionary. In other words, in this application, a data set can be determined based on original data, then distributed computation processing can be performed on the data set to obtain similarity, and a similarity dictionary can be obtained based on the similarity obtained after distributed computation processing. Finally, recommendation processing is performed based on the similarity dictionary. Therefore, the recommendation system achieves parallel computation of similarity by obtaining similarity through distributed computation processing of the data set, and then builds a similarity dictionary based on the similarity, thus improving computational efficiency. Furthermore, the recommendation system effectively improves the accuracy of similarity calculation by building a similarity dictionary based on the similarity, thereby ultimately enabling recommendation processing using the similarity dictionary, obtaining better recommendation results, and improving the feasibility of the recommendation model.
[0185] Specifically, the program instructions corresponding to a recommended method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a recommended method in the storage media are read or executed by an electronic device, the following steps are included:
[0186] Determine the data set based on the original data;
[0187] The similarity parameters are obtained by performing distributed computation processing on the dataset.
[0188] A similarity dictionary is obtained based on the similarity parameters;
[0189] Recommendation processing is performed based on the similarity dictionary.
[0190] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0191] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0193] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0194] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.
Claims
1. A recommendation method, characterized in that, The method includes: Determine the data set based on the original data; Distributed computing is performed on the dataset to obtain a similarity parameter, which represents the consistency of a user's ratings for different items. A similarity dictionary is obtained based on the similarity parameters; Recommendation processing is performed based on the similarity dictionary; The step of obtaining the similarity dictionary based on the similarity parameter includes: The total similarity is obtained by summing the similarity parameters. The total number of evaluations is obtained by summing the number of evaluations corresponding to the similarity parameters. The similarity dictionary is obtained by dividing the total similarity by the total number of evaluations. The similarity dictionary is calculated using the following formula: ,in, To be based on all users simultaneously viewing the item and items The similarity obtained from the rating; For all users to view items simultaneously and items The probability of giving a score; The recommendation process based on the similarity dictionary includes: Get the evaluated set and the unevaluated set; The score of the unrated set is obtained based on the similarity dictionary and the rated set; The recommendation process is performed based on the ratings of the unrated set.
2. The method according to claim 1, characterized in that, The process of determining the data set based on the original data includes: Based on the user dimension in the original data, the original data is integrated and processed to obtain a first data set; or, Based on the item dimension in the original data, the original data is integrated and processed to obtain a second data set.
3. The method according to claim 2, characterized in that, The step of performing distributed computation processing based on the dataset to obtain similarity parameters includes: The first similarity is obtained by calculating the ratio between a user's ratings of different items in the first dataset using a preset algorithm. The first similarity is processed to obtain the minimum value to obtain the similarity parameter.
4. The method according to claim 2, characterized in that, The step of performing distributed computation processing based on the dataset to obtain similarity parameters includes: A second similarity is obtained by calculating the ratio between different ratings of an item by different users for that item under the second data set using a preset algorithm. The second similarity is processed by minimizing the value to obtain the similarity parameter.
5. The method according to claim 1, characterized in that, After obtaining the similarity dictionary based on the similarity parameters, the method further includes: Acquire incremental data; An incremental similarity dictionary is obtained based on the incremental data; The similarity dictionary is updated based on the incremental similarity dictionary.
6. The method according to claim 5, characterized in that, The step of obtaining a score for the unrated set based on the similarity dictionary and the rated set includes: The similarity dictionary is queried to obtain the recommended similarity between the evaluated set and the unevaluated set, and the rating of the evaluated set; The score for the unrated set is obtained based on the recommended similarity and the score of the rated set.
7. A recommendation system, characterized in that, The recommendation system includes a determination unit, an acquisition unit, and a recommendation unit. The determining unit is used to determine the data set based on the original data; The acquisition unit is configured to perform distributed computing processing based on the dataset to obtain a similarity parameter, wherein the similarity parameter represents the consistency of a user's ratings for different items; and to obtain a similarity dictionary based on the similarity parameter. The recommendation unit is used to perform recommendation processing based on the similarity dictionary; The acquisition unit is further configured to perform a summation operation on the similarity parameters to obtain a total similarity; perform a summation operation on the number of evaluations corresponding to the similarity parameters to obtain a total number of evaluations; and perform a division operation on the total similarity and the total number of evaluations to obtain the similarity dictionary, which is calculated by the following formula: ,in, To be based on all users simultaneously viewing the item and items The similarity obtained from the rating; For all users to view items simultaneously and items The probability of giving a score; The recommendation unit is also used to obtain the evaluated set and the unevaluated set; and to obtain a score for the unevaluated set based on the similarity dictionary and the evaluated set. The recommendation process is performed based on the ratings of the unrated set.
8. A recommendation system, characterized in that, The recommendation system further includes a processor and a memory storing processor-executable instructions, which, when executed by the processor, implement the method as described in any one of claims 1-6.
9. A computer-readable storage medium having a program stored thereon for use in a recommendation system, wherein the program, when executed by a processor, implements the method as described in any one of claims 1-6.