Data processing method, device and computer readable storage medium
By assigning the scoring matrix to multiple nodes in parallel to decompose it, the decomposition matrix is obtained to predict the user's rating of the object, solving the problem of recommendation accuracy and inefficiency, and achieving accurate recommendation and efficient processing of unoperated objects.
Patent Information
- Application Number
- CN201911220013.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-12-03
AI Technical Summary
When recommending objects, existing algorithms cannot accurately recommend objects that users have not performed related operations, and the data processing efficiency of massive users and objects is low.
By allocating the data in the scoring matrix to multiple nodes and decomposing it in parallel, two decomposition matrices are obtained, which are used to predict the user's ratings of the object and recommend the object to the user based on the scoring prediction matrix.
It improves the accuracy and processing efficiency of recommendations. Even if the user has not operated on certain objects, he can score and predict through massive data, recommend objects more accurately, and improves the computing efficiency through multi-node parallel processing.
Smart Images

Figure CN112905873B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device and computer-readable storage medium. Background Art
[0002] Nowadays, the application of the Internet has penetrated into people's lives. Users use online platforms to shop, socialize and view various information.
[0003] Online platforms can collect big data on user operations and recommend objects to users based on their behavior, such as products, videos, news, etc. Summary of the invention
[0004] The inventors found that some current algorithms promote objects of the same type to users based on their ratings of objects. However, some users have not performed any related operations on some objects. Therefore, it is impossible to accurately recommend objects without any related operations. Furthermore, the data of users and objects on the network platform is massive. Currently, the rating matrix is generally processed as a whole, which leads to low processing efficiency.
[0005] A technical problem to be solved by the present disclosure is: how to predict the user's rating of an object, improve the recommendation accuracy and how to improve the processing efficiency.
[0006] According to some embodiments of the present disclosure, a data processing method is provided, comprising: allocating data in a rating matrix to each node; wherein the density of the rating matrix is not 1, and each element in the rating matrix represents a rating of different objects by different users; a 0 element represents no rating between the corresponding user and the object; according to the data of the allocated rating matrix, determining corresponding decomposition data in parallel at each node; according to the decomposition data corresponding to each node, determining two decomposition matrices into which the rating matrix is decomposed; according to the two decomposition matrices, determining a rating prediction matrix; wherein the rating prediction matrix includes: predicted ratings of each user for each object; according to the rating prediction matrix, determining objects recommended to the user.
[0007] In some embodiments, determining corresponding decomposition data in parallel at each node based on data of an assigned scoring matrix includes: dividing the data of the assigned scoring matrix into multiple threads at each node; determining decomposition data corresponding to the thread in each thread in parallel based on the data of the scoring matrix divided into the thread; and determining decomposition data corresponding to the node based on the decomposition data corresponding to each thread.
[0008] In some embodiments, the two decomposition matrices include: a first matrix and a second matrix; according to the data of the assigned scoring matrix, the corresponding decomposition data is determined in parallel at each node, including: in each node, according to the data of the assigned scoring matrix, the first sub-matrix updated in the previous cycle, and the second matrix updated in the previous cycle, determine the first sub-matrix updated in the current cycle as the decomposition data corresponding to the node; wherein the first sub-matrix updated in the previous cycle is the first matrix updated in the previous cycle divided and distributed to each node in the same manner as the scoring matrix.
[0009] In some embodiments, determining the two decomposition matrices into which the scoring matrix is decomposed according to the decomposition data corresponding to each node includes: forming the first sub-matrix updated in the current cycle obtained by each node into the first matrix updated in the current cycle; determining the second matrix updated in the current cycle according to the first matrix updated in the current cycle, the second matrix updated in the previous cycle, and the scoring matrix; multiplying the first matrix updated in the current cycle and the second matrix updated in the current cycle to obtain a product matrix; and calculating the error between the product matrix and the scoring matrix; when the error meets a preset condition, using the first matrix updated in the current cycle and the second matrix updated in the current cycle as the two decomposition matrices into which the scoring matrix is decomposed.
[0010] In some embodiments, it also includes: initializing two decomposition matrices, wherein the number of rows of the first matrix is equal to the number of rows of the scoring matrix, and the number of columns of the second matrix is equal to the number of columns of the scoring matrix; dividing the data of the initialized first matrix into multiple initialized first sub-matrices in the same distribution method as the scoring matrix, and distributing them to each node; sending the initialized second decomposition matrix to each node; wherein the first sub-matrix updated in the previous cycle is initially the initialized first sub-matrix, and the second matrix updated in the previous cycle is initially the initialized second decomposition matrix.
[0011] In some embodiments, allocating data in the scoring matrix to each node includes: determining the number of rows allocated to each node based on the ratio of the total number of rows in the scoring matrix to the number of nodes; and allocating each row of data in the scoring matrix to each node according to the number of rows allocated to each node.
[0012] In some embodiments, determining the rating prediction matrix according to the two decomposition matrices includes: multiplying the two decomposition matrices to obtain a product matrix, and using the product matrix as the rating prediction matrix.
[0013] In some embodiments, determining the objects recommended to the user based on the rating prediction matrix includes: comparing the ratings of each user on each object in the rating prediction matrix with a threshold, and recommending the object corresponding to the rating to the user corresponding to the rating when the rating exceeds the threshold.
[0014] According to some other embodiments of the present disclosure, a data processing device is provided, including: a data allocation module, a decomposition module, a prediction module, a recommendation module and multiple nodes; the data allocation module is used to allocate data in a rating matrix to each node; wherein the density of the rating matrix is not 1, and each element in the rating matrix represents the rating of different users on different objects; a 0 element represents that there is no rating between the corresponding user and the object; each node is used to determine the corresponding decomposition data in parallel according to the data of the allocated rating matrix; the decomposition module is used to determine the two decomposition matrices into which the rating matrix is decomposed according to the decomposition data corresponding to each node; the prediction module is used to determine the rating prediction matrix according to the two decomposition matrices; wherein the rating prediction matrix includes: the predicted ratings of each user on each object; the recommendation module is used to determine the objects recommended to the user according to the rating prediction matrix.
[0015] In some embodiments, each node is used to divide the data of the assigned scoring matrix into multiple threads; in parallel, in each thread, the decomposition data corresponding to the thread is determined according to the data of the scoring matrix divided into the thread; and according to the decomposition data corresponding to each thread, the decomposition data corresponding to the node is determined.
[0016] In some embodiments, the two decomposition matrices include: a first matrix and a second matrix; each node is used to determine the first submatrix updated in the current cycle as the decomposition data corresponding to the node based on the data of the assigned scoring matrix, the first submatrix updated in the previous cycle, and the second matrix updated in the previous cycle; wherein the first submatrix updated in the previous cycle is the first matrix updated in the previous cycle divided and distributed to each node in the same manner as the scoring matrix.
[0017] In some embodiments, the decomposition module is used to form a first matrix updated in the current cycle from the first sub-matrices updated in the current cycle obtained by each node; determine the second matrix updated in the current cycle according to the first matrix updated in the current cycle, the second matrix updated in the previous cycle, and the scoring matrix; multiply the first matrix updated in the current cycle and the second matrix updated in the current cycle to obtain a product matrix; and calculate the error between the product matrix and the scoring matrix; when the error meets a preset condition, the first matrix updated in the current cycle and the second matrix updated in the current cycle are used as the two decomposition matrices into which the scoring matrix is decomposed.
[0018] In some embodiments, it also includes: an initialization module, used to initialize two decomposition matrices, wherein the number of rows of the first matrix is equal to the number of rows of the scoring matrix, and the number of columns of the second matrix is equal to the number of columns of the scoring matrix; the data of the initialized first matrix is divided into multiple initialized first sub-matrices according to the same distribution method as the scoring matrix, and distributed to each node; the initialized second decomposition matrix is sent to each node; wherein the first sub-matrix updated in the previous cycle is initially the initialized first sub-matrix, and the second matrix updated in the previous cycle is initially the initialized second decomposition matrix.
[0019] In some embodiments, the data allocation module is used to determine the number of rows allocated to each node based on the ratio of the total number of rows in the scoring matrix to the number of nodes; and allocate each row of data in the scoring matrix to each node according to the number of rows allocated to each node.
[0020] In some embodiments, the prediction module is used to multiply two decomposition matrices to obtain a product matrix, and use the product matrix as a score prediction matrix.
[0021] In some embodiments, the recommendation module is used to compare the scores of each user on each object in the score prediction matrix with a threshold, and when the score exceeds the threshold, recommend the object corresponding to the score to the user corresponding to the score.
[0022] According to some further embodiments of the present disclosure, a data processing device is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the data processing method as described in any of the aforementioned embodiments.
[0023] According to some further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the data processing method of any of the aforementioned embodiments.
[0024] In the present disclosure, the rating matrix is decomposed, and the two decomposed matrices respectively represent the user's preference for several factors and the hidden factors of the object's preference. Based on the two decomposition matrices, the ratings between the user and the object corresponding to the 0 element in the rating matrix can be predicted. Therefore, even if the user has not performed relevant operations on certain objects, the user's ratings on the objects that have not been operated can be predicted through the data of massive users, and then the objects can be recommended to the user more accurately. In the data processing process, the data of the rating matrix is allocated to multiple nodes, and the multiple nodes process the data of the allocated rating matrix in parallel to obtain decomposed data. According to the corresponding decomposed data of each node, the two decomposition matrices into which the rating matrix is decomposed are determined. Therefore, the multi-node parallel data processing method disclosed in the present disclosure improves processing efficiency.
[0025] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0027] Figure 1 A flowchart diagram showing a data processing method according to some embodiments of the present disclosure is provided.
[0028] Figure 2 A schematic flowchart showing a data processing method according to other embodiments of the present disclosure.
[0029] Figure 3 A schematic diagram showing the structure of a data processing device according to some embodiments of the present disclosure.
[0030] Figure 4 A schematic diagram showing the structure of a data processing device according to other embodiments of the present disclosure.
[0031] Figure 5 A schematic diagram showing the structure of a data processing device according to some further embodiments of the present disclosure. DETAILED DESCRIPTION
[0032] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0033] This disclosure proposes a data processing method that can improve the accuracy and efficiency of recommending objects to users. Figure 1 Some embodiments of the data processing method of the present disclosure are described.
[0034] Figure 1 Flowcharts of some embodiments of the data processing method disclosed herein. Figure 1 As shown, the method of this embodiment includes: steps S102 to S110.
[0035] In step S102, the data in the scoring matrix is distributed to each node.
[0036] The ratings of each object by each user in the network platform can be obtained, such as the ratings of movies by users, the ratings of commodities by users, etc. The objects can be of the same type. The ratings of each user on each object are combined into a rating matrix, for example, each row in the rating matrix corresponds to each user, and each column corresponds to each object. If the user has no rating on the object, it can be replaced by a preset element, for example, the preset element is 0. These preset elements are the elements to be predicted later, and the predicted values are determined for these elements to be predicted, that is, the ratings between the corresponding users and the objects. The density of the rating matrix is not 1, and each element in the rating matrix represents the ratings of different objects by different users; the 0 element indicates that there is no rating between the corresponding user and the object.
[0037] User portrait information can be obtained for all users of the network platform, similar users can be obtained based on the user portrait information, and the ratings of each object by these similar users can be used to generate a rating matrix. For example, user portrait information includes: age, gender, hobbies, etc. Using similar users to determine the rating matrix can improve the prediction accuracy.
[0038] In some embodiments, the data in the scoring matrix can be evenly distributed to each node. The number of rows allocated to each node is determined according to the ratio of the total number of rows in the scoring matrix to the number of nodes; and each row of data in the scoring matrix is allocated to each node according to the number of rows allocated to each node. That is, the scoring matrix is evenly distributed to each node by row. MPI (Message Passing Interface) technology can be used to divide the scoring matrix into multiple parts by row, and then each process uses the allocated data for calculation.
[0039] In step S104, corresponding decomposition data is determined in parallel at each node according to the data of the assigned scoring matrix.
[0040] In some embodiments, at each node, data of the assigned scoring matrix is divided into multiple threads; decomposition data corresponding to the thread is determined in each thread in parallel according to the data of the scoring matrix divided into the thread; and decomposition data corresponding to the node is determined according to the decomposition data corresponding to each thread.
[0041] CUDA (Compute Unified Device Architecture) technology can be used at each node to accelerate matrix operations. The Kernel function in CUDA can evenly divide data into multiple threads, and each thread performs corresponding operations in parallel. The hybrid parallel processing method of parallel operation of multiple nodes and parallel operation of multiple threads in each node can greatly improve the operation efficiency.
[0042] In some embodiments, the method further includes: initializing two decomposition matrices, wherein the number of rows of the first matrix is equal to the number of rows of the scoring matrix, and the number of columns of the second matrix is equal to the number of columns of the scoring matrix; dividing the data of the initialized first matrix into a plurality of initialized first sub-matrices in the same allocation manner as the scoring matrix, and allocating them to each node; and sending the initialized second decomposition matrix to each node.
[0043] In each node, a gradient descent method can be used to iteratively update the initialized first sub-matrix according to the data of the assigned scoring matrix, and each update obtains the first sub-matrix after the current cycle as the decomposition data corresponding to the node.
[0044] In step S106, two decomposition matrices into which the scoring matrix is decomposed are determined according to the decomposition data corresponding to each node.
[0045] The first submatrix of each node after the current cycle update is obtained, and the second matrix after the current cycle update is determined according to the first submatrix of each node after the current cycle update. Therefore, the update of the second matrix is also performed iteratively, and the gradient descent method can also be used. For example, the first matrix after the current cycle update and the second matrix after the current cycle update are multiplied to obtain a product matrix; and the error between the product matrix and the scoring matrix is calculated; when the error meets the preset conditions, the first matrix after the current cycle update and the second matrix after the current cycle update are used as two decomposition matrices into which the scoring matrix is decomposed.
[0046] In some embodiments, in each node, according to the data of the assigned scoring matrix, the first submatrix updated in the previous cycle, and the second matrix updated in the previous cycle, the first submatrix updated in the current cycle is determined as the decomposition data corresponding to the node; wherein the first submatrix updated in the previous cycle is the first matrix updated in the previous cycle divided and allocated to each node in the same manner as the scoring matrix. The first submatrix updated in the previous cycle is initially the initialized first submatrix, and the second matrix updated in the previous cycle is initially the initialized second decomposition matrix.
[0047] Furthermore, the first submatrices updated in the current cycle obtained by each node are combined into a first matrix updated in the current cycle; the second matrix updated in the current cycle is determined according to the first matrix updated in the current cycle, the second matrix updated in the previous cycle, and the scoring matrix; the first matrix updated in the current cycle and the second matrix updated in the current cycle are multiplied to obtain a product matrix; and the error between the product matrix and the scoring matrix is calculated; when the error meets a preset condition, the first matrix updated in the current cycle and the second matrix updated in the current cycle are used as two decomposition matrices into which the scoring matrix is decomposed.
[0048] In step S108, a rating prediction matrix is determined based on the two decomposition matrices.
[0049] The rating prediction matrix includes: the predicted ratings of each user for each object. For example, the two decomposition matrices are multiplied to obtain a product matrix, and the product matrix is used as the rating prediction matrix. The rating prediction matrix has the same number of rows and columns as the rating matrix, and the elements in the rating prediction matrix corresponding to the preset element positions in the rating matrix are the predicted ratings between the corresponding users and objects.
[0050] In step S110, objects to be recommended to the user are determined based on the rating prediction matrix.
[0051] In some embodiments, the ratings of each user on each object in the rating prediction matrix are compared with a threshold value, and when the rating exceeds the threshold value, the object corresponding to the rating is recommended to the user corresponding to the rating.
[0052] In the above embodiment, the rating matrix is decomposed, and the two decomposed matrices respectively represent the user's preference for several factors and the hidden factors of the object being preferred. Based on the two decomposition matrices, the rating between the user and the object corresponding to the 0 element in the rating matrix can be predicted. Therefore, even if the user has not performed relevant operations on certain objects, the user's rating for the object that has not been operated can be predicted through the data of a large number of users, and then the object is recommended to the user more accurately. In the data processing process, the data of the rating matrix is allocated to multiple nodes, and multiple nodes process the data of the allocated rating matrix in parallel to obtain decomposed data. According to the corresponding decomposed data of each node, the two decomposition matrices into which the rating matrix is decomposed are determined. Therefore, the multi-node parallel data processing method in the above embodiment improves the processing efficiency. In addition, a hybrid parallel processing method of parallel operation of multiple nodes and parallel operation of multiple threads in each node is adopted to further improve the operation efficiency.
[0053] Combine the following Figure 2 Describe how to decompose the rating matrix into two decomposition matrices.
[0054] Figure 2 Flowcharts of other embodiments of the data processing method disclosed herein. Figure 2 As shown, the method of this embodiment includes: steps S202 to S218.
[0055] In step S202, the root node obtains a rating matrix and determines the number of users and the number of objects.
[0056] The scoring matrix is, for example, R M×N , M represents the number of users, and N represents the number of objects.
[0057] In step S204, the root node initializes the first matrix and the second matrix according to the number of users and the number of objects, and initializes the relevant parameters of the stochastic gradient descent algorithm.
[0058] The first matrix P M×K The number of rows is equal to the number of users M, the number of columns is the number of features K, and the second matrix Q K×N The number of rows is the number of features K, and the number of columns is the number of objects N. The relevant parameters of the stochastic gradient descent algorithm include, for example: α, β, and the iteration number threshold iters. α and β are weight values in the stochastic gradient descent algorithm.
[0059] In step S206, the root node divides the data of the scoring matrix into multiple parts according to the number of nodes, and divides the data of the initialized first matrix into multiple initialized first sub-matrices in the same allocation manner as the scoring matrix.
[0060] In step S208, the root node sends the data corresponding to each scoring matrix, initializes the first sub-matrix, initializes the second matrix and initializes related parameters to each node.
[0061] In step S210, each node is divided into multiple threads, which determine the updated first sub-matrix of the current cycle in parallel according to the data of the assigned scoring matrix, the updated first sub-matrix of the previous cycle, the updated second matrix of the previous cycle and the updated related parameters of the previous cycle.
[0062] In the first iteration process, the first sub-matrix updated in the previous cycle, the second matrix updated in the previous cycle and the related parameters updated in the previous cycle are respectively the initialized first sub-matrix, the initialized second matrix and the initialized related parameters.
[0063] For example, each node may determine the first sub-matrix after updating in the current cycle according to the following formula.
[0064]
[0065]
[0066] In formulas (1) and (2), r i,j represents the element in the i-th row and j-th column of the submatrix of the score matrix assigned to the node, p i,k represents the element in the i-th row and k-th column of the first submatrix after the previous cycle update, q k,j Represents the element in the k-th row and j-th column of the second matrix after update in the previous cycle, where i, j and k are positive integers. represents the element in the i-th row and k-th column of the first submatrix after the current cycle update, and α and β represent the weights after the previous cycle update.
[0067] In step S212, each node sends the first sub-matrix updated in the corresponding current cycle to the root node.
[0068] In step S214, the root node combines the first sub-matrices updated in the current cycle corresponding to each node into the first matrix updated in the current cycle, and determines the second matrix updated in the current cycle according to the scoring matrix, the first matrix updated in the current cycle, the second matrix updated in the previous cycle and the relevant parameters updated in the previous cycle.
[0069] For example, the root node may determine the second matrix updated in the current cycle according to the following formula.
[0070]
[0071]
[0072] represents the element in the kth row and jth column of the second matrix after update in the current cycle. Other parameters are the same as the above formula.
[0073] In step S216, the root node multiplies the first matrix updated in the current cycle with the second matrix updated in the current cycle to obtain a product matrix; and calculates the error between the product matrix and the scoring matrix to determine whether the error meets the preset condition or reaches the iteration number threshold. If the error meets the preset condition or reaches the iteration number threshold, step S218 is executed, otherwise step S217 is executed.
[0074] In step S217, the root node sends the second matrix updated in the current cycle to each node, and each node uses the first matrix updated in the current cycle and the second matrix updated in the current cycle as the first matrix updated in the previous cycle and the second matrix updated in the previous cycle, respectively, and returns to step S210 to restart execution.
[0075] In step S218, the first matrix updated in the current cycle and the second matrix updated in the current cycle are used as two decomposition matrices decomposed from the scoring matrix.
[0076] The present disclosure also provides a data processing device, Figure 3 Give a description.
[0077] Figure 3 Figure 1 is a structural diagram of some embodiments of the data processing device disclosed in the present invention. Figure 3 As shown, the device 30 of this embodiment includes: a data allocation module 310, a decomposition module 320, a prediction module 330, a recommendation module 340 and multiple nodes 350; the data allocation module 310, the decomposition module 320, the prediction module 330, the recommendation module 340 can be set at a root node, and the root node is different from the node 350.
[0078] The data distribution module 310 is used to distribute the data in the rating matrix to each node; wherein the density of the rating matrix is not 1, and each element in the rating matrix represents the rating of different users on different objects; and a 0 element indicates that there is no rating between the corresponding user and the object.
[0079] In some embodiments, the data allocation module 310 is used to determine the number of rows allocated to each node based on the ratio of the total number of rows in the scoring matrix to the number of nodes; and allocate each row of data in the scoring matrix to each node according to the number of rows allocated to each node.
[0080] Each node 350 is used to determine corresponding decomposition data in parallel according to the data of the assigned scoring matrix.
[0081] In some embodiments, each node 350 is used to divide the data of the assigned scoring matrix into multiple threads; determine the decomposition data corresponding to the thread in each thread in parallel according to the data of the scoring matrix divided into the thread; and determine the decomposition data corresponding to the node according to the decomposition data corresponding to each thread.
[0082] In some embodiments, the two decomposition matrices include: a first matrix and a second matrix; each node 350 is used to determine the first submatrix updated in the current cycle as the decomposition data corresponding to the node based on the data of the assigned scoring matrix, the first submatrix updated in the previous cycle, and the second matrix updated in the previous cycle; wherein the first submatrix updated in the previous cycle is the first matrix updated in the previous cycle divided and allocated to each node in the same manner as the scoring matrix.
[0083] The decomposition module 320 is used to determine the two decomposition matrices into which the rating matrix is decomposed according to the decomposition data corresponding to each node.
[0084] In some embodiments, the decomposition module 320 is used to form the first sub-matrix updated in the current cycle obtained by each node into the first matrix updated in the current cycle; determine the second matrix updated in the current cycle according to the first matrix updated in the current cycle, the second matrix updated in the previous cycle, and the scoring matrix; multiply the first matrix updated in the current cycle and the second matrix updated in the current cycle to obtain a product matrix; and calculate the error between the product matrix and the scoring matrix; when the error meets the preset conditions, the first matrix updated in the current cycle and the second matrix updated in the current cycle are used as the two decomposition matrices into which the scoring matrix is decomposed.
[0085] The prediction module 330 is used to determine a rating prediction matrix based on the two decomposition matrices; wherein the rating prediction matrix includes: predicted ratings of each user for each object.
[0086] In some embodiments, the prediction module 330 is used to multiply the two decomposition matrices to obtain a product matrix, and use the product matrix as a score prediction matrix.
[0087] The recommendation module 340 is used to determine the objects to be recommended to the user according to the rating prediction matrix.
[0088] In some embodiments, the recommendation module 340 is used to compare the scores of each user on each object in the score prediction matrix with a threshold, and when the score exceeds the threshold, recommend the object corresponding to the score to the user corresponding to the score.
[0089] In some embodiments, the device 30 also includes: an initialization module 360, which is used to initialize two decomposition matrices, wherein the number of rows of the first matrix is equal to the number of rows of the scoring matrix, and the number of columns of the second matrix is equal to the number of columns of the scoring matrix; the data of the initialized first matrix is divided into multiple initialized first sub-matrices in the same distribution method as the scoring matrix, and distributed to each node; the initialized second decomposition matrix is sent to each node; wherein the first sub-matrix updated in the previous cycle is initially the initialized first sub-matrix, and the second matrix updated in the previous cycle is initially the initialized second decomposition matrix.
[0090] The data processing apparatus in the embodiments of the present disclosure may be implemented by various computing devices or computer systems. Figure 4 as well as Figure 5 Give a description.
[0091] Figure 4 Figure 1 is a structural diagram of some embodiments of the data processing device disclosed in the present invention. Figure 4 As shown, the device 40 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410 , and the processor 420 is configured to execute the data processing method in any of some embodiments of the present disclosure based on the instructions stored in the memory 410 .
[0092] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.
[0093] Figure 5 FIG. 1 is a structural diagram of some other embodiments of the data processing device disclosed in the present invention. Figure 5As shown, the device 50 of this embodiment includes: a memory 510 and a processor 520, which are similar to the memory 410 and the processor 420 respectively. It can also include an input and output interface 530, a network interface 540, a storage interface 550, etc. These interfaces 530, 540, 550 and the memory 510 and the processor 520 can be connected, for example, through a bus 560. Among them, the input and output interface 530 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 540 provides a connection interface for various networked devices, for example, it can be connected to a database server or a cloud storage server. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0094] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0095] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0096] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0098] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A data processing method, comprising: The data in the scoring matrix is distributed to each node; wherein the density of the scoring matrix is not 1, each element in the scoring matrix represents the score of different users on different objects, and the 0 element represents that there is no score between the corresponding user and the object, wherein user portrait information is obtained for all users of the network platform, similar users are obtained according to the user portrait information, and the scores of similar users on each object are used to generate the scoring matrix; Determining corresponding decomposition data in parallel at each node according to the assigned data of the scoring matrix, including: dividing each node into a plurality of threads, and determining the first sub-matrix updated in the current cycle in parallel according to the assigned data of the scoring matrix, the first sub-matrix updated in the previous cycle, the second matrix updated in the previous cycle, and the related parameters updated in the previous cycle; Determine the two decomposition matrices into which the scoring matrix is decomposed according to the decomposition data corresponding to each node, including: each node sends the corresponding first sub-matrix updated in the current cycle to the root node, the root node composes the first sub-matrix updated in the current cycle corresponding to each node into the first matrix updated in the current cycle, and determines the second matrix updated in the current cycle according to the scoring matrix, the first matrix updated in the current cycle, the second matrix updated in the previous cycle, and the relevant parameters updated in the previous cycle, and the root node multiplies the first matrix updated in the current cycle and the second matrix updated in the current cycle to obtain a product matrix; and calculate the error between the product matrix and the scoring matrix to determine whether the error meets a preset condition or reaches a threshold of the number of iterations, and if the error meets the preset condition or reaches the threshold of the number of iterations, the first matrix updated in the current cycle and the second matrix updated in the current cycle are used as the two decomposition matrices decomposed into the scoring matrix; Determine a rating prediction matrix based on the two decomposition matrices; wherein the rating prediction matrix includes: predicted ratings of each user for each object; According to the rating prediction matrix, determine the object to be recommended to the user.
2. The data processing method according to claim 1, further comprising: Initializing the two decomposition matrices, wherein the number of rows of the first matrix is equal to the number of rows of the scoring matrix, and the number of columns of the second matrix is equal to the number of columns of the scoring matrix; Dividing the data of the initialized first matrix into a plurality of initialized first sub-matrices in the same allocation manner as the scoring matrix, and allocating the data to each node; Sending the initialized second matrix to each node; The first sub-matrix updated in the previous cycle is initially the initialized first sub-matrix, and the second matrix updated in the previous cycle is initially the initialized second matrix.
3. The data processing method according to claim 1, wherein: The step of allocating the data in the scoring matrix to each node includes: Determining the number of rows allocated to each node according to the ratio of the total number of rows of the scoring matrix to the number of nodes; According to the number of rows assigned to each node, each row of data in the scoring matrix is respectively assigned to each node.
4. The data processing method according to claim 1, wherein: Determining the score prediction matrix according to the two decomposition matrices includes: The two decomposition matrices are multiplied to obtain a product matrix, and the product matrix is used as a score prediction matrix.
5. The data processing method according to claim 1, wherein: Determining the object to recommend to the user according to the rating prediction matrix includes: The scores of each user on each object in the score prediction matrix are compared with a threshold value, and when the score exceeds the threshold value, the object corresponding to the score is recommended to the user corresponding to the score.
6. A data processing device, comprising: Data allocation module, decomposition module, prediction module, recommendation module and multiple nodes; The data distribution module is used to distribute the data in the scoring matrix to each node; wherein the density of the scoring matrix is not 1, each element in the scoring matrix represents the score of different users on different objects, and the 0 element represents that there is no score between the corresponding user and the object, wherein user portrait information is obtained for all users of the network platform, similar users are obtained based on the user portrait information, and the scores of similar users on each object are used to generate the scoring matrix; Each node is used to determine the corresponding decomposition data in parallel according to the assigned data of the scoring matrix, including: each node is divided into multiple threads, and in parallel, according to the assigned data of the scoring matrix, the first sub-matrix updated in the previous cycle, the second matrix updated in the previous cycle, and the related parameters updated in the previous cycle, the first sub-matrix updated in the current cycle is determined; The decomposition module is used to determine the two decomposition matrices into which the scoring matrix is decomposed according to the decomposition data corresponding to each node, including: each node sends the corresponding first sub-matrix updated in the current cycle to the root node, the root node composes the first sub-matrix updated in the current cycle corresponding to each node into the first matrix updated in the current cycle, and determines the second matrix updated in the current cycle according to the scoring matrix, the first matrix updated in the current cycle, the second matrix updated in the previous cycle and the relevant parameters updated in the previous cycle, and the root node multiplies the first matrix updated in the current cycle and the second matrix updated in the current cycle to obtain a product matrix; and calculates the error between the product matrix and the scoring matrix to determine whether the error meets the preset condition or reaches the iteration number threshold, and if the error meets the preset condition or reaches the iteration number threshold, the first matrix updated in the current cycle and the second matrix updated in the current cycle are used as the two decomposition matrices decomposed into the scoring matrix; The prediction module is used to determine a rating prediction matrix based on the two decomposition matrices; wherein the rating prediction matrix includes: the predicted ratings of each user for each object; The recommendation module is used to determine the objects recommended to the user according to the rating prediction matrix.
7. A data processing device, comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the data processing method according to any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.
