Method and device for generating statistical data matrix of user behavior

Abstract

The present invention discloses a method and device for generating a statistical data matrix of user behavior, which relates to the field of computer technology. A specific implementation of the method includes: obtaining multiple single behavior data of multiple users for multiple items within a preset time interval; determining the non-zero statistical results of the target dimension data of each user for each item and the key-value pair data formed by the non-zero statistical results and the item identifier based on the single behavior data; storing the key-value pair data of the same user separately in a set, and storing the item identifier in the first subset, and storing the non-zero statistical results in the second subset; converting the item identifier in each set into the column number or row number of the matrix, so that each set becomes a row vector or column vector for forming a statistical data matrix. This implementation can quickly obtain the statistical results of user behavior data without using a structured query language, thereby generating a statistical data matrix that does not waste storage space.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for generating a statistical data matrix of user behavior. Background Art

[0002] In the field of computer technology, it's common to analyze user behavior, such as browsing, searching, and purchasing. To more intuitively demonstrate the differentiation between different users' behaviors regarding different items, a user behavior statistics matrix can be created. Each row of the matrix corresponds to a user, each column to an item, and each element represents the statistical result of a user's behavior data for a specific item within a preset time interval.

[0003] In existing distributed systems using Hive (a data warehouse tool used in distributed systems) as the query engine, these statistical results are typically obtained using Hive SQL (SQL stands for Structured Query Language). The number of Hive SQL statements is comparable to the number of item types. In real-world scenarios, the number of item types is often quite large, which makes the writing, storage, and execution of Hive SQL statements resource-intensive, severely impacting the efficiency of generating the statistical data matrix. Furthermore, the number of users is often similarly large, and the statistical data matrix is generally sparse. Therefore, storing the statistical data matrix using the existing method (i.e., displaying every element in the matrix) results in slow reading and significant space waste. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a method and device for generating a statistical data matrix of user behavior, which can quickly obtain statistical results of user behavior data without using structured query language, and then generate a statistical data matrix stored in a manner that does not waste space through data aggregation and format conversion.

[0005] To achieve the above objective, according to one aspect of the present invention, a method for generating a statistical data matrix of user behavior is provided.

[0006] The method for generating a statistical data matrix of user behavior in an embodiment of the present invention includes: obtaining multiple single behavior data of multiple users for multiple items within a preset time interval; wherein the single behavior data includes data of preset target dimensions; determining the statistical results of the target dimension data of each user for each item based on the single behavior data, and selecting non-zero statistical results therefrom; wherein each non-zero statistical result and its corresponding item identifier form key-value pair data; storing the key-value pair data of the same user separately in a set; in the set, the item identifier in the key-value pair data is stored in a first subset, and the non-zero statistical results in the key-value pair data are stored in a second subset; wherein, in the first subset and the second subset, the data in the same storage position correspond to each other; and, converting the item identifier in each set into a column number of a matrix, so that each set becomes a row vector for constituting the statistical data matrix; or, converting the item identifier in each set into a row number of a matrix, so that each set becomes a column vector for constituting the statistical data matrix.

[0007] Optionally, any element in the statistical data matrix represents: the statistical results in the target dimension of the behavior performed by one of the multiple users with respect to one of the multiple items within the time interval; any element in the statistical data matrix that is in a different row and column from the element represents: the statistical results in the target dimension of the behavior performed by another user among the multiple users with respect to another item among the multiple items within the time interval.

[0008] Optionally, the method further includes: after storing the key-value pair data of the same user separately in a set, sorting the key-value pair data in each set according to a preset order; wherein, the sorting arranges the matrix column numbers or row numbers corresponding to the item identifiers in the key-value pair data in ascending order; storing the item identifiers in the key-value pair data in the first subset and storing the non-zero statistical results in the key-value pair data in the second subset specifically includes: storing the item identifiers in the key-value pair data in the first subset and storing the non-zero statistical results in the key-value pair data in the second subset according to the arrangement order of the key-value pair data formed after the sorting.

[0009] Optionally, the method further comprises: before making each set into the row vector or the column vector, determining the number of types of the plurality of items and storing the number in each set.

[0010] Optionally, the method is used in a Spark distributed system with Hive as the query engine, and the single behavior data is stored in a Hive data table; the key reduction operator in Spark is used to obtain non-zero statistical results, and the key-value pair data of the same user is stored separately in a set using the key combination operator in Spark; the behavior includes at least one of the following: browsing, searching, and purchasing; the target dimension includes at least one of the following: number of browsing, browsing duration, number of searches, and number of purchases; and the first subset and the second subset are arrays, and the statistical data matrix is a sparse matrix.

[0011] To achieve the above-mentioned objective, according to another aspect of the present invention, a device for generating a statistical data matrix of user behavior is provided.

[0012] The device for generating a statistical data matrix of user behavior in an embodiment of the present invention may include: a data acquisition unit, used to acquire multiple single behavior data of multiple users for multiple items within a preset time interval; wherein the single behavior data includes data of preset target dimensions; a statistical unit, used to determine the statistical results of the target dimension data of each user for each item based on the single behavior data, and select non-zero statistical results therefrom; wherein each non-zero statistical result and its corresponding item identifier form key-value pair data; an aggregation unit, used to store the key-value pair data of the same user separately in a set; in the set, the item identifiers in the key-value pair data are stored in a first subset, and the non-zero statistical results in the key-value pair data are stored in a second subset; wherein in the first subset and the second subset, the data in the same storage position correspond to each other; a matrix generation unit, used to convert the item identifiers in each set into the column number of the matrix, so that each set becomes a row vector for constituting the statistical data matrix; or, convert the item identifiers in each set into the row number of the matrix, so that each set becomes a column vector for constituting the statistical data matrix.

[0013] Optionally, any element in the statistical data matrix represents: the statistical results in the target dimension of the behavior performed by one of the multiple users with respect to one of the multiple items within the time interval; any element in the statistical data matrix that is in a different row and column from the element represents: the statistical results in the target dimension of the behavior performed by another user among the multiple users with respect to another item among the multiple items within the time interval.

[0014] Optionally, the device may further include a sorting unit and a computing unit; the sorting unit is used to: after storing the key-value pair data of the same user separately in a set, sort the key-value pair data in each set according to a preset order; wherein the sorting makes the matrix column numbers or row numbers corresponding to the item identifiers in the key-value pair data arranged in ascending order; the computing unit is used to: before making each set into the row vector or column vector, determine the number of types of the multiple items and store the number in each set.

[0015] Optionally, the aggregation unit may be further configured to: store the item identifiers in the key-value pair data in a first subset and store the non-zero statistical results in the key-value pair data in a second subset according to the arrangement order of the key-value pair data formed after the sorting.

[0016] Optionally, the device is used in a Spark distributed system with Hive as the query engine, and the single behavior data is stored in a Hive data table; the key reduction operator in Spark is used to obtain non-zero statistical results, and the key-value pair data of the same user is stored separately in a set using the key combination operator in Spark; the behavior includes at least one of the following: browsing, searching, and purchasing; the target dimension includes at least one of the following: number of browsing times, browsing duration, number of searches, and number of purchases; the first subset and the second subset are arrays, and the statistical data matrix is a sparse matrix.

[0017] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0018] An electronic device of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a statistical data matrix of user behavior provided by the present invention.

[0019] To achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0020] A computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the method for generating a statistical data matrix of user behavior provided by the present invention.

[0021] According to the technical solution of the present invention, one embodiment of the above invention has the following advantages or beneficial effects:

[0022] First, after acquiring a user's single-time behavior data, the algorithm quickly calculates the non-zero statistical results for the target dimension data. These non-zero statistical results and their corresponding item identifiers form key-value pairs. Next, each user's key-value pair data is aggregated into a set. Within each set, the item identifiers and non-zero statistical results are sequentially stored in two subsets. Finally, the item identifiers are converted into matrix column or row numbers according to a preset strategy, generating a statistical data matrix stored in a novel format that is faster to read and saves storage space.

[0023] Second, in the Spark (a big data processing framework) distributed system, non-zero statistical results and data aggregation can be completed using Spark's existing operators, thereby improving data processing efficiency and overcoming the defect of the existing technology that requires a large number of Hive SQL statements to be written.

[0024] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0026] Figure 1 Schematic diagram of the main steps of the method for generating a statistical data matrix of user behavior in an embodiment of the present invention;

[0027] Figure 2 Schematic diagram of components of a device for generating a statistical data matrix of user behavior according to an embodiment of the present invention;

[0028] Figure 3 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0029] Figure 4 The figure is a schematic diagram of the structure of an electronic device used to implement the method for generating a statistical data matrix of user behavior in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0032] Figure 1 4 is a schematic diagram of the main steps of a method for generating a statistical data matrix of user behavior according to an embodiment of the present invention.

[0033] like Figure 1 As shown, the method for generating a statistical data matrix of user behavior according to an embodiment of the present invention can be specifically performed according to the following steps:

[0034] Step S101: Acquire multiple pieces of single behavior data of multiple users for multiple items within a preset time interval; wherein the single behavior data includes data of preset target dimensions.

[0035] In an embodiment of the present invention, a statistical data matrix of user behavior is used to store the statistical results of the behavioral data of multiple users for multiple items in a target dimension, so as to demonstrate the behavioral differentiation of multiple users. The above-mentioned behaviors may be browsing behaviors, searching behaviors, purchasing behaviors, etc. for items, and the above-mentioned behavioral data may be data on the number of views, browsing duration data, number of searches, and number of purchases for items. The target dimension and the statistical method for obtaining statistical results can be determined according to business needs. For example, when it is necessary to count the total number of views and the total browsing duration of each user for each item, the number of views and the browsing duration are the target dimensions, and the statistical method is to sum the single views and the single browsing duration.

[0036] In the statistical data matrix: row data corresponds to different users, column data corresponds to different types of items; or, row data corresponds to different types of items, column data corresponds to different users. It can be understood that these two situations only differ in the form of data, and there is no essential difference. The technical solution of the present invention will be described below using the first situation as an example. In the statistical data matrix: each row of data corresponds to a user, different row data corresponds to different users, each column of data corresponds to an item, and different column data corresponds to different types of items; any element in the statistical data matrix represents: during the above time interval, the statistical results of the behavior of one of the above multiple users on one of the above multiple items in the target dimension; any element in the statistical data matrix that is in a different row and column from the element represents: during the above time interval, the statistical results of the behavior of another user of the above multiple users on another of the above multiple items in the target dimension. In specific applications, different types of items can be distinguished by using stock keeping units (SKUs) as identifiers.

[0037] For example, when counting the number of times users a, b, and c viewed items A, B, C, D, E, and F within a certain time interval, according to the preset correspondence between user IDs and matrix row numbers, a, b, and c correspond to the 1st, 2nd, and 3rd rows of the matrix, respectively. A, B, C, D, E, and F are all item SKUs. According to the preset strategy for determining the correspondence between item IDs and matrix column numbers, these SKUs correspond to the 1st, 2nd, 3rd, 4th, 5th, and 6th columns of the matrix, respectively. If user a viewed item A twice and item C once within the time interval, user b viewed item B three times, item D four times, and item F once within the time interval, and user c viewed item C three times and item E twice within the time interval, then the statistical data matrix stored in a traditional manner would be as follows:

[0038]

[0039] In practical applications, the number of item types and users is often quite large, while most users only browse a small number of items. Therefore, the statistical data matrix is generally a sparse matrix. If the statistical data matrix is stored using the above-mentioned traditional method, the large number of zero elements in it will waste storage space and cause the matrix data to be read slowly. Therefore, in an embodiment of the present invention, a new matrix storage method is provided. This matrix storage method can be in the following two forms.

[0040] In the first form, each row vector of the matrix is represented as two data sets, where the first data set is used to store the column numbers of non-zero elements, and the second data set is used to store the non-zero elements that correspond one-to-one to the column numbers in the first data set. The data in the same storage location of the first and second data sets correspond to each other. In specific scenarios, the first and second data sets can be containers that can store data in order, such as arrays. If the matrix in the above example is represented in the first form, the first row vector is [AR(1,3), ar(2,1)], the second row vector is [AR(2,4,6), ar(3,4,1)], and the third row vector is [AR(3,5), ar(3,2)], where AR is the name of the first data set and ar is the name of the second data set. Taking the first row vector as an example, the "1" in the first data set AR(1,3) represents the first column, and "3" represents the third column. The "2" in the second data set ar(2,1) corresponds to the "1" in the first data set, indicating that the element in the first column of this row vector is "2". The "1" in the second subset ar(2,1) corresponds to the "3" in the first subset, indicating that the element in the third column of this row vector is "1".

[0041] In the second form, each column vector of the matrix is represented as a third data set and a fourth data set. The third data set is used to store the row numbers of non-zero elements, and the fourth data set is used to store non-zero elements that correspond one-to-one to the row numbers in the third data set. The data in the same storage location of the third data set and the fourth data set correspond to each other. In a specific scenario, the third data set and the fourth data set can be containers that can store data in order, such as arrays. If the matrix in the above example is represented in the second form, the first column vector is [ARR(1), arr(2)], the second column vector is [ARR(2), arr(3)], the third column vector is [ARR(1,3), arr(1,3)], the fourth column vector is [ARR(2), arr(4)], the fifth column vector is [ARR(3), arr(2)], and the sixth column vector is [ARR(2), arr(1)]. Among them, ARR is the name of the third data set and arr is the name of the fourth data set. Taking the third column vector as an example, the "1" and "3" in the third data set ARR(1,3) represent the first and third rows respectively. The "1" in the fourth data set arr(1,3) is the element in the first row of the column vector, and the "3" is the element in the third row of the column vector.

[0042] In the above two forms, to facilitate the actual use of data, the number of columns of the matrix can be added to each row vector (the number of columns indicates how many columns the matrix has), or the number of rows of the matrix can be added to each column vector (the number of rows indicates how many rows the matrix has). The above matrix storage method only stores the non-zero elements in the sparse matrix, thereby avoiding the occupation of storage space by a large number of zero elements and improving the reading speed of the matrix. In an embodiment of the present invention, the statistical data matrix finally obtained can be stored in this way.

[0043] In step S101, the above-mentioned single behavior data refers to the relevant data obtained after the user performs a single behavior. Taking browsing behavior as an example, single behavior data may include: user identification, browsed page URL (Uniform ResourceLocator), identification of items in the page, browsing time, etc. It can be understood that the appearance of the item identification indicates that the user has browsed the item once, so the item identification dimension in the single behavior data is equivalent to the number of views dimension. In addition, since it is necessary to use the single behavior data to obtain subsequent target dimension data statistics, the single behavior data contains target dimension data. In the Spark distributed system with Hive as the query engine, single behavior data is stored in the Hive data table. In specific applications, after obtaining the single behavior data, it is first necessary to perform data cleaning, delete abnormal values, and fill in null values.

[0044] Step S102: Determine the statistical results of the target dimension data of each user for each item based on the single behavior data, and select non-zero statistical results therefrom; wherein each non-zero statistical result and its corresponding item identifier form a key-value pair data.

[0045] In this step, statistics of target dimension data are first performed for each user and each item. Specifically, for any user among the above-mentioned multiple users, each of the above-mentioned multiple items is traversed, and the statistical results of the target dimension data in the single behavior data are calculated. For example, in the scenario of counting the total number of browsing times, the number of browsing times of each user for each item can be accumulated to obtain statistical results. Thereafter, the zero-valued statistical results can be removed from the statistical results to obtain non-zero statistical results. In the Spark distributed system, obtaining non-zero statistical results can be achieved by using the key reduction operator (i.e., reduceByKey operator) in Spark. After obtaining the non-zero statistical results, each non-zero statistical result and its corresponding item identifier form key-value pair data. Generally, in the key-value pair data, the item identifier is the key and the non-zero statistical result is the value.

[0046] For example, in the example recorded in step S101, the non-zero statistical results of each user for each item are as follows:

[0047] User ID Item Identification Views a A 2 a C 1 b B 3 b F 1 b D 4 c C 3 c E 2

[0048] Step S103: The key-value pair data of the same user are stored separately in a set; in the set, the item identifiers in the key-value pair data are stored in a first subset, and the non-zero statistical results in the key-value pair data are stored in a second subset.

[0049] In this step, all key-value pair data of the same user are first stored separately in a set, that is, different sets are used to store their respective key-value pair data for different users. In the Spark distributed system, this operation can be implemented by the key combination operator (i.e., combineByKey operator) in Spark. Thereafter, in any set, the item identifiers in the key-value pair data are stored in the first subset, and the non-zero statistical results in the key-value pair data are stored in the second subset. In the first subset and the second subset, the data in the same storage position correspond to each other, which is conducive to storing the statistical data matrix through the aforementioned new matrix storage method. Generally, the first subset and the second subset can be containers that can store data in order, such as arrays.

[0050] For example, in the above example, the set of key-value pair data stored by user a is [(A, 2), (C, 1)], the set of key-value pair data stored by user b is [(B, 3), (F, 1), (D, 4)], and the set of key-value pair data stored by user c is [(C, 3), (E, 2)]. After storing the data in the first subset and the second subset, the set of user a becomes [ARRAY(A, C), array(2, 1)], the set of user b becomes [ARRAY(B, F, D), array(3, 1, 4)], and the set of user c becomes [ARRAY(C, E), array(3, 2)]. Here, ARRAY is the name of the first subset, and array is the name of the second subset. Preferably, the number of item types (6 in this example) can also be stored in each set to form a more complete set content.

[0051] As a preferred solution, after storing the key-value pair data of the same user in the above set, the key-value pair data in each set can be sorted according to a preset order. The above sorting can make the matrix column number (or row number) corresponding to the item identifier in the key-value pair data arranged in ascending order. That is to say, through the above sorting, the column number (or row number) in the subsequent new matrix storage method can be arranged in ascending order, so that the matrix data has higher usability. In practical applications, the correspondence between the item identifier and the matrix column number (or row number) is generally determined by a preset strategy. The above strategy often used in specific scenarios is: sort the matrix column number (or row number) and the item identifier (such as SKU) in ascending order respectively, and determine that the matrix column number (or row number) and the item identifier in the same arrangement position have a corresponding relationship. For example, if A < B < C < D < E < F in the above example, then A can be corresponding to the first column (or the first row), B can be corresponding to the second column (or the second row), C can be corresponding to the third column (or the third row), D can be corresponding to the fourth column (or the fourth row), and E can be corresponding to the fifth column (or the fifth row). In this way, after obtaining the set of user b [(B, 3), (F, 1), (D, 4)], the key-value pair data in the set can be arranged as [(B, 3), (D, 4), (F, 1)].

[0052] In the embodiment of the present invention, after performing the above sorting, according to the arrangement order of the key-value pair data formed after sorting, the item identifiers in the key-value pair data can be stored in the first subset, and the non-zero statistical results in the key-value pair data can be stored in the second subset, so as to ensure that the data in the same storage position in the first subset and the second subset correspond to each other. For example, after obtaining the set of user b [(B, 3), (D, 4), (F, 1)], the item identifier and the non-zero statistical result can be stored separately to form the set [ARRAY(B, D, F), array(3, 4, 1)].

[0053] Step S104: Convert the item identifiers in each set into the column numbers of the matrix so that each set becomes a row vector for constituting the statistical data matrix; or, convert the item identifiers in each set into the row numbers of the matrix so that each set becomes a column vector for constituting the statistical data matrix.

[0054] In this step, the item identifiers in each set can be converted into column numbers or row numbers according to a preset strategy, so that each set becomes a row vector or column vector represented by the aforementioned new matrix storage method, thereby generating a statistical data matrix of user behavior. In the above example, after the above conversion, the set of user a is [ARRAY(1,3), array(2,1)], the set of user b is [ARRAY(2,4,6), array(3,4,1)], and the set of user c is [ARRAY(3,5), array(3,2)], thereby obtaining the statistical data matrix of users a, b, and c for items A, B, C, D, E, and F. It should be noted that the step of converting item identifiers into column numbers or row numbers can also be performed at other times, such as after obtaining the user's single behavior data, and the present invention does not limit the execution time of this step.

[0055] It's understandable that in certain scenarios, the row and column numbers of a matrix have unique rules, such as starting at zero and increasing in number. In this case, the correspondence between item IDs and column numbers (or row numbers) needs to be corrected. Furthermore, in the Spark distributed system, item IDs can be mapped to natural numbers that increase starting at 1 (i.e., item IDs are arranged in ascending order and mapped to 1, 2, 3, and so on). This way, after storing each user's key-value data in a collection, the resulting data is in the commonly used LIBSVM format (a simple and effective software package and data format), making it easier to process later.

[0056] In addition, in a statistical data matrix, the statistical results as its elements can be data of one target dimension corresponding to one behavior, data of multiple target dimensions corresponding to one behavior, or data of multiple target dimensions corresponding to multiple behaviors. When the statistical results are multidimensional data, they can be stored in the form of an array. In this case, each storage position of the array corresponds to one target dimension. For example, when counting the data of two target dimensions, namely the number of views and the browsing duration, in a statistical data matrix, each element (i.e., the statistical result) is a binary array. In the binary array, the first bit of data can be set to the statistical result of the number of views, and the second bit of data can be set to the statistical result of the browsing duration, and this corresponding relationship can be maintained throughout the execution process of the method of the present invention. Through the above settings, the adaptability of the statistical data matrix to multiple behaviors and multiple target dimensions can be improved.

[0057] In the technical solution of the embodiment of the present invention, after obtaining the user's single behavior data, the non-zero statistical results of the target dimension data are quickly calculated, and the non-zero statistical results and the corresponding item identifiers form key-value pair data. Afterwards, the key-value pair data of each user are aggregated into a set, and within each set, the item identifiers and the non-zero statistical results are stored in two subsets in order. Finally, the item identifiers are converted into the column numbers or row numbers of the matrix according to the preset strategy, thereby generating a statistical data matrix stored in a new way. This method is faster to read and does not waste storage space. In the Spark distributed system, the acquisition of non-zero statistical results and data aggregation can be completed using Spark's existing operators, thereby improving data processing efficiency and overcoming the defect of the existing technology that requires a large number of Hive SQL statements to be written.

[0058] It should be noted that for the aforementioned method embodiments, for ease of description, they are expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required to implement the present invention.

[0059] In order to better implement the above solutions of the embodiments of the present invention, relevant devices for implementing the above solutions are also provided below.

[0060] See also Figure 2 As shown, the apparatus 200 for generating a statistical data matrix of user behavior provided by an embodiment of the present invention may include: a data acquisition unit 201 , a statistical unit 202 , an aggregation unit 203 and a matrix generation unit 204 .

[0061] The data acquisition unit 201 can be used to obtain multiple single behavior data for multiple items from multiple users within a preset time interval; wherein the single behavior data includes data of preset target dimensions. The statistical unit 202 can be used to determine the statistical results of the target dimension data of each user for each item based on the single behavior data, and select non-zero statistical results therefrom; wherein each non-zero statistical result and its corresponding item identifier form key-value pair data. The aggregation unit 203 can be used to store the key-value pair data of the same user separately in a set; in the set, the item identifiers in the key-value pair data are stored in a first subset, and the non-zero statistical results in the key-value pair data are stored in a second subset; wherein in the first subset and the second subset, the data in the same storage position correspond to each other; the matrix generation unit 204 can be used to convert the item identifiers in each set into the column numbers of the matrix, so that each set becomes a row vector for constituting the statistical data matrix; or, convert the item identifiers in each set into the row numbers of the matrix, so that each set becomes a column vector for constituting the statistical data matrix.

[0062] In an embodiment of the present invention, any element in the statistical data matrix represents: the statistical results of the behavior of one of the multiple users with respect to one of the multiple items in the target dimension within the time interval; any element in the statistical data matrix that is in a different row and column from the element represents: the statistical results of the behavior of another user of the multiple users with respect to another item among the multiple items in the target dimension within the time interval.

[0063] Preferably, the apparatus 200 may further include a sorting unit and a computing unit. The sorting unit may be configured to, after storing the key-value pair data of the same user in a separate set, sort the key-value pair data in each set according to a preset order; wherein the sorting arranges the matrix column numbers or row numbers corresponding to the item identifiers in the key-value pair data in ascending order. The computing unit may be configured to, before converting each set into the row vector or column vector, determine the number of types of the multiple items and store the number in each set.

[0064] As a preferred solution, the aggregation unit 203 can be further used to: store the item identifiers in the key-value pair data in the first subset and store the non-zero statistical results in the key-value pair data in the second subset according to the arrangement order of the key-value pair data formed after the sorting.

[0065] In one embodiment, the device is used in a Spark distributed system using Hive as a query engine, and the single behavior data is stored in a Hive data table. To obtain non-zero statistical results, the key reduction operator in Spark can be used, and to store the key-value pair data of the same user separately in a set, the key combination operator in Spark can be used. In addition, in an embodiment of the present invention, the behavior includes at least one of the following: browsing, searching, and purchasing; the target dimension includes at least one of the following: number of views, browsing duration, number of searches, and number of purchases; the first subset and the second subset are arrays, and the statistical data matrix is a sparse matrix.

[0066] In the technical solution of the embodiment of the present invention, after obtaining the user's single behavior data, the non-zero statistical results of the target dimension data are quickly calculated, and the non-zero statistical results and the corresponding item identifiers form key-value pair data. Afterwards, the key-value pair data of each user are aggregated into a set, and within each set, the item identifiers and the non-zero statistical results are stored in two subsets in order. Finally, the item identifiers are converted into the column numbers or row numbers of the matrix according to the preset strategy, thereby generating a statistical data matrix stored in a new way. This method is faster to read and does not waste storage space. In the Spark distributed system, the acquisition of non-zero statistical results and data aggregation can be completed using Spark's existing operators, thereby improving data processing efficiency and overcoming the defect of the existing technology that requires a large number of Hive SQL statements to be written.

[0067] Figure 3 An exemplary system architecture 300 is shown to which the method for generating a statistical data matrix of user behavior or the apparatus for generating a statistical data matrix of user behavior according to an embodiment of the present invention may be applied.

[0068] like Figure 3 As shown, system architecture 300 may include terminal devices 301, 302, and 303, a network 304, and a server 305 (this architecture is merely an example, and the components included in the specific architecture may be adjusted based on the specific application). Network 304 is used to provide a medium for communication links between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0069] Users can use terminal devices 301, 302, 303 to interact with server 305 via network 304 to receive or send messages, etc. Various client applications can be installed on terminal devices 301, 302, 303, such as numerical calculation applications, matrix generation applications, etc. (only as examples).

[0070] The terminal devices 301 , 302 , and 303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0071] Server 305 may be a server that provides various services, such as a data analysis server (for example only) that supports a matrix generation application operated by a user using terminal devices 301, 302, and 303. The data analysis server may process a received matrix generation request and feed back the processing result (for example, a generated matrix - for example only) to terminal devices 301, 302, and 303.

[0072] It should be noted that the method for generating the statistical data matrix of user behavior provided in the embodiment of the present invention is generally executed by the server 305 . Accordingly, the device for generating the statistical data matrix of user behavior is generally set in the server 305 .

[0073] It should be understood that Figure 3 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0074] The present invention also provides an electronic device. The electronic device in an embodiment of the present invention includes: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a statistical data matrix of user behavior provided by the present invention.

[0075] Reference below Figure 4 , which shows a schematic structural diagram of a computer system 400 of an electronic device suitable for implementing an embodiment of the present invention. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0076] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the computer system 400 are also stored in the RAM 403. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0077] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 408 including a hard disk; and a communication section 409 including a network interface card such as a LAN card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed in the storage section 408 as needed.

[0078] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 409 and / or installed from removable media 411. When the computer program is executed by the central processing unit 401, the above-described functions defined in the system of the present invention are performed.

[0079] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0081] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor including a data acquisition unit, a statistics unit, an aggregation unit, and a matrix generation unit. The names of these units do not, in some cases, limit the units themselves. For example, the data acquisition unit may also be described as a "unit that provides single behavior data to the statistics unit."

[0082] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the device, the steps executed by the device include: obtaining multiple single behavior data of multiple users for multiple items within a preset time interval; wherein, the single behavior data includes data of preset target dimensions; determining the statistical results of the target dimension data of each user for each item based on the single behavior data, and selecting non-zero statistical results therefrom; wherein, each non-zero statistical result and its corresponding item identifier form key-value pair data; storing the key-value pair data of the same user separately in a set; in the set, the item identifier in the key-value pair data is stored in a first subset, and the non-zero statistical results in the key-value pair data are stored in a second subset; wherein, in the first subset and the second subset, the data in the same storage position correspond to each other; and, converting the item identifier in each set into a column number of a matrix, so that each set becomes a row vector for constituting a statistical data matrix; or, converting the item identifier in each set into a row number of a matrix, so that each set becomes a column vector for constituting a statistical data matrix.

[0083] In the technical solution of the embodiment of the present invention, after obtaining the user's single behavior data, the non-zero statistical results of the target dimension data are quickly calculated, and the non-zero statistical results and the corresponding item identifiers form key-value pair data. Afterwards, the key-value pair data of each user are aggregated into a set, and within each set, the item identifiers and the non-zero statistical results are stored in two subsets in order. Finally, the item identifiers are converted into the column numbers or row numbers of the matrix according to the preset strategy, thereby generating a statistical data matrix stored in a new way. This method is faster to read and does not waste storage space. In the Spark distributed system, the acquisition of non-zero statistical results and data aggregation can be completed using Spark's existing operators, thereby improving data processing efficiency and overcoming the defect of the existing technology that requires a large number of Hive SQL statements to be written.

[0084] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for generating a statistical data matrix of user behavior, characterized in that: include: Acquire multiple pieces of single-time behavior data of multiple users for multiple items within a preset time interval; wherein the single-time behavior data includes data of preset target dimensions; Determine statistical results of target dimension data for each user for each item based on the single behavior data, and select non-zero statistical results therefrom; wherein each non-zero statistical result and its corresponding item identifier form key-value pair data, and the target dimension data is data on the target dimension of the behaviors of the multiple users for the multiple items; The key-value pair data of the same user are stored separately in a set; in the set, the item identifiers in the key-value pair data are stored in a first subset, and the non-zero statistical results in the key-value pair data are stored in a second subset; wherein, in the first subset and the second subset, the data in the same storage position correspond to each other; and The item identifiers in each set are converted into the column numbers of the matrix so that each set becomes a row vector for constituting the statistical data matrix; or, the item identifiers in each set are converted into the row numbers of the matrix so that each set becomes a column vector for constituting the statistical data matrix.

2. The method according to claim 1, characterized in that Any element in the statistical data matrix represents: a statistical result of a behavior performed by one of the multiple users on one of the multiple items in the target dimension within the time interval; Any element in the statistical data matrix that is in a different row and column from the element represents: the statistical results of the behavior of another user among the multiple users on another item among the multiple items in the target dimension within the time interval.

3. The method according to claim 1, characterized in that The method further includes: after storing the key-value pair data of the same user separately in a set, sorting the key-value pair data in each set according to a preset order; wherein the sorting is such that the matrix column numbers or row numbers corresponding to the item identifiers in the key-value pair data are arranged in ascending order; The storing of the item identifiers in the key-value pair data in the first subset and the storing of the non-zero statistical results in the key-value pair data in the second subset specifically includes: storing the item identifiers in the key-value pair data in the first subset and the storing of the non-zero statistical results in the key-value pair data in the second subset according to the arrangement order of the key-value pair data formed after the sorting.

4. The method according to claim 1, wherein The method further comprises: Before making each set into the row vector or the column vector, the number of types of the plurality of items is determined and the number is stored in each set.

5. The method according to any one of claims 1 to 4, characterized in that: The method is used in a Spark distributed system using Hive as a query engine, and the single behavior data is stored in a Hive data table; To obtain non-zero statistical results, Spark's key reduction operator is used. To store the key-value pairs of the same user in a separate collection, Spark's key combination operator is used. The behavior includes at least one of the following: browsing, searching, and purchasing; the target dimension includes at least one of the following: number of views, browsing duration, number of searches, and number of purchases; Furthermore, the first subset and the second subset are arrays, and the statistical data matrix is a sparse matrix.

6. A device for generating a statistical data matrix of user behavior, characterized in that: include: A data acquisition unit, configured to acquire a plurality of pieces of single-time behavior data of a plurality of users for a plurality of items within a preset time interval; wherein the single-time behavior data includes data of a preset target dimension; a statistical unit, configured to determine statistical results of target dimension data for each user for each item based on the single behavior data, and select non-zero statistical results therefrom; wherein each non-zero statistical result and its corresponding item identifier form key-value pair data, and the target dimension data is data on the target dimension of the behaviors of the multiple users for the multiple items; an aggregation unit, configured to store the key-value pair data of the same user separately in a set; in the set, storing the item identifiers in the key-value pair data in a first subset, and storing the non-zero statistical results in the key-value pair data in a second subset; wherein, in the first subset and the second subset, data in the same storage location correspond to each other; and A matrix generating unit is used to convert the item identifiers in each set into the column number of the matrix so that each set becomes a row vector for constituting the statistical data matrix; or to convert the item identifiers in each set into the row number of the matrix so that each set becomes a column vector for constituting the statistical data matrix.

7. The device according to claim 6, characterized in that Any element in the statistical data matrix represents: a statistical result of a behavior performed by one of the multiple users on one of the multiple items in the target dimension within the time interval; Any element in the statistical data matrix that is in a different row and column from the element represents: the statistical results of the behavior of another user among the multiple users on another item among the multiple items in the target dimension within the time interval.

8. The device according to claim 6 or 7, characterized in that The apparatus further includes a sorting unit and a computing unit; the sorting unit is configured to: after storing the key-value pair data of the same user separately in a set, sort the key-value pair data in each set according to a preset order; wherein the sorting arranges the matrix column numbers or row numbers corresponding to the item identifiers in the key-value pair data in ascending order; the computing unit is configured to: before converting each set into the row vector or column vector, determine the number of types of the plurality of items and store the number in each set; The aggregation unit is further configured to: store the item identifiers in the key-value pair data in the first subset and store the non-zero statistical results in the key-value pair data in the second subset according to the order of the key-value pair data formed after the sorting; The device is used in a Spark distributed system using Hive as a query engine, wherein the single behavior data is stored in a Hive data table; a key reduction operator in Spark is used to obtain non-zero statistical results, and a key combination operator in Spark is used to store key-value pair data of the same user separately in a set; and The behavior includes at least one of the following: browsing, searching, and purchasing; the target dimension includes at least one of the following: number of browsing times, browsing time, number of searches, and number of purchases; the first subset and the second subset are arrays, and the statistical data matrix is a sparse matrix.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images