A data clustering method, device and apparatus

By combining the clustering results of basic data and incremental data, using the preselected center algorithm and k-means algorithm to process flow data, the problems of long data clustering time and large memory consumption in the existing technology are solved, and real-time clustering and Resource savings.

CN114638316BActive Publication Date: 2024-11-26国投融合科技股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210328668.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-11-26
Estimated Expiration
2042-03-30

AI Technical Summary

Technical Problem

The existing data clustering methods process data for a long time and consume a lot of memory, so they cannot effectively process valuable information in streaming data.

Method used

By determining the clustering results of the basic data and the clustering results of the incremental data in each stage, the target clustering results of the full data are obtained after merging, reducing the calculation time and avoiding multiple reads and readings of the full data. Preselected center algorithm and k-means clustering algorithm perform data processing.

Benefits of technology

Real-time clustering results of full data are achieved, reducing the processing time and memory consumption of data clustering methods, and reducing the cost of hardware equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114638316B_ABST
    Figure CN114638316B_ABST
Patent Text Reader

Abstract

The present invention provides a data clustering method, device and equipment, the data clustering method comprising: determining a first clustering result of the basic data based on the basic data on a network platform; obtaining incremental data added to the basic data at each stage; determining a second clustering result of the incremental data at each stage based on the incremental data at each stage; obtaining a target clustering result of the full data based on the first clustering result and the second clustering result; wherein the full data includes the basic data, the incremental data added to the basic data at the current stage, and the incremental data added to the basic data before the current stage. The scheme of the present invention can achieve real-time acquisition of clustering results of the full data, reduce the long data processing time of the data clustering method, and avoid multiple reading in and out of the full data, effectively reduce memory consumption, and reduce the cost of hardware equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data clustering method, device and equipment. Background Art

[0002] With the rapid development of technologies such as the Internet of Things and 5G networks, a large amount of continuous dynamic streaming data has been generated. Timely and rapid analysis of valuable information in streaming data will bring huge profits. Due to the new characteristics of streaming data, traditional methods for streaming data processing have disadvantages in terms of time and resources. First, using traditional static data clustering methods will consume a lot of processing time. Since the value of data is inversely proportional to time, long processing time will reduce the value of data. Secondly, traditional methods use the entire amount of data as the calculation object, and each execution requires a large amount of memory space, resulting in a waste of computing resources and increasing the cost of data analysis.

[0003] Therefore, a data clustering method is needed to solve the problem that existing data clustering methods take a long time to process data and consume a lot of memory. Summary of the invention

[0004] The embodiments of the present invention provide a data clustering method, device and equipment to solve the problem in the prior art that the data clustering method takes a long time to process data and consumes a lot of memory.

[0005] In order to solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] An embodiment of the present invention provides a data clustering method, comprising:

[0007] Determine a first clustering result of the basic data according to the basic data on the network platform;

[0008] Obtaining incremental data added to the basic data at each stage;

[0009] Determine a second clustering result of the incremental data of each stage according to the incremental data of each stage;

[0010] Obtaining a target clustering result of the entire data according to the first clustering result and the second clustering result;

[0011] The full amount of data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage.

[0012] Optionally, the method further comprises:

[0013] According to the target clustering result, the target feature parameters of the entire data are obtained.

[0014] Optionally, determining a first clustering result of the basic data according to the basic data on the network platform includes:

[0015] Determining a first initial center of the basic data by a pre-selected center algorithm;

[0016] The basic data is clustered according to the first initial center to obtain the first clustering result.

[0017] Optionally, obtaining incremental data added to the basic data at each stage includes:

[0018] When the amount of target data added to the basic data in the target stage reaches a preset threshold, determining that the target data is incremental data added to the basic data in the target stage;

[0019] or,

[0020] When the target data added to the basic data in the target stage has been added for a predetermined period of time, determining that the target data is incremental data added to the basic data in the target stage;

[0021] The target stage is one of each of the stages.

[0022] Optionally, determining a second clustering result of the incremental data of each stage according to the incremental data of each stage includes:

[0023] Determine the second initial center of the incremental data in each stage by a pre-selected center algorithm;

[0024] The incremental data of each stage is clustered according to the second initial center to obtain the second clustering result.

[0025] Optionally, obtaining a target clustering result of the entire data according to the first clustering result and the second clustering result includes:

[0026] Determining a third initial center of the first clustering result and the second clustering result by a pre-selected center algorithm;

[0027] Clustering the entire data according to the third initial center to obtain a third clustering result;

[0028] The target clustering result is obtained according to the third clustering result and a preset timeliness requirement.

[0029] Optionally, obtaining the target clustering result according to the third clustering result and a preset timeliness requirement includes:

[0030] In a case where the preset timeliness requirement is the first timeliness requirement, taking the third clustering result as the target clustering result;

[0031] In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result;

[0032] Among them, the first timeliness requirement is higher than the second timeliness requirement.

[0033] Optionally, the method further comprises:

[0034] The basic data and the incremental data of each stage are stored.

[0035] The embodiment of the present invention further provides a data clustering device, comprising:

[0036] A first determination module, used to determine a first clustering result of the basic data according to the basic data on the network platform;

[0037] An acquisition module, used for acquiring incremental data added to the basic data at each stage;

[0038] A second determination module, configured to determine a second clustering result of the incremental data of each stage according to the incremental data of each stage;

[0039] A third determination module, configured to obtain a target clustering result of the entire data according to the first clustering result and the second clustering result;

[0040] The full amount of data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage.

[0041] Optionally, the device further comprises:

[0042] The fourth determination module is used to obtain the target feature parameters of the full amount of data according to the target clustering result.

[0043] Optionally, the first determining module includes:

[0044] A first determining unit, configured to determine a first initial center of the basic data by using a pre-selected center algorithm;

[0045] The first clustering unit is used to cluster the basic data according to the first initial center to obtain the first clustering result.

[0046] Optionally, the acquisition module includes:

[0047] An acquiring unit, configured to determine that the target data is incremental data added to the basic data in the target stage when the amount of target data added to the basic data in the target stage reaches a preset threshold;

[0048] or,

[0049] When the target data added to the basic data in the target stage has been added for a predetermined period of time, determining that the target data is incremental data added to the basic data in the target stage;

[0050] The target stage is one of each of the stages.

[0051] Optionally, the second determining module includes:

[0052] A second determining unit, configured to determine a second initial center of the incremental data of each stage by a pre-selected center algorithm;

[0053] The second clustering unit is used to cluster the incremental data of each stage according to the second initial center to obtain the second clustering result.

[0054] Optionally, the third determining module includes:

[0055] a third determining unit, configured to determine a third initial center of the first clustering result and the second clustering result by using a pre-selected center algorithm;

[0056] A third clustering unit, configured to cluster the entire data according to the third initial center to obtain a third clustering result;

[0057] The fourth determining unit is used to obtain the target clustering result according to the third clustering result and a preset timeliness requirement.

[0058] Optionally, the fourth determining unit is specifically configured to use the third clustering result as the target clustering result when the preset timeliness requirement is the first timeliness requirement;

[0059] In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result;

[0060] Among them, the first timeliness requirement is higher than the second timeliness requirement.

[0061] Optionally, the device further comprises:

[0062] A storage module is used to store the basic data and the incremental data of each stage.

[0063] An embodiment of the present invention further provides a data clustering device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program implements the steps of the data clustering method described above when executed by the processor.

[0064] An embodiment of the present invention further provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the steps in the data clustering method described above are implemented.

[0065] The beneficial effects of the present invention are:

[0066] The scheme of the present invention, by determining the first clustering result of the basic data, and determining the second clustering result of the incremental data added to the basic data in each stage, and determining the target clustering result of the full data based on the first clustering result and the second clustering result, can achieve real-time acquisition of the clustering result of the full data, reduce the long data processing time of the data clustering method, and avoid multiple reading and reading of the full data, effectively reduce memory consumption, and reduce the cost of hardware equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of a data clustering method provided by an embodiment of the present invention is shown;

[0068] Figure 2 A schematic diagram showing the effective range of the initial center provided by an embodiment of the present invention;

[0069] Figure 3 A schematic diagram showing that the initial center in the initial center set provided by an embodiment of the present invention remains unchanged;

[0070] Figure 4 A schematic diagram showing a newly added initial center in the initial center set provided by an embodiment of the present invention;

[0071] Figure 5 A second flowchart showing the data clustering method provided by an embodiment of the present invention;

[0072] Figure 6 A flowchart showing a product recommendation method provided by an embodiment of the present invention;

[0073] Figure 7 A schematic diagram showing the structure of a data clustering device provided by an embodiment of the present invention;

[0074] Figure 8 A schematic diagram showing the structure of a data clustering device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] First, some concepts are explained as follows:

[0077] Streaming Data

[0078] Stream data is usually defined as a sequence of tuples consisting of continuously arriving components. It is a continuous, infinite data set with no clear end boundary that changes over time. It has the characteristics of an infinite total amount of data and a data value that gradually decreases with time.

[0079] Clustering Algorithms

[0080] Clustering is to divide a data set into different classes or clusters according to a specific standard (such as distance), so that the characteristics of the data in the same cluster have the greatest similarity possible, and the characteristics of the data in different clusters are as different as possible.

[0081] Classic data clustering algorithm

[0082] Classic data clustering algorithms include: density-based clustering algorithms, such as the DBSCAN classic algorithm and the DenStream data stream clustering algorithm; partition-based clustering algorithms, such as the K-means classic algorithm and the STREAM data stream clustering algorithm; hierarchy-based clustering algorithms, such as the CURE classic algorithm and the CluStream data stream clustering algorithm; grid-based clustering algorithms, such as the STING classic algorithm and the D-Stream data stream clustering algorithm.

[0083] The present invention aims to solve the problem in the prior art that data clustering methods take a long time to process data and consume a lot of memory, and provides a data clustering method, device and equipment.

[0084] like Figure 1 As shown, an embodiment of the present invention provides a data clustering method, including:

[0085] Step 101: Determine a first clustering result of the basic data based on the basic data on the network platform.

[0086] It should be noted that the data clustering method provided in the embodiment of the present invention is implemented using the stream processing framework Apache Flink.

[0087] In this step, the basic data on the network platform is first read in. After the basic data is read in, clustering is performed on the basic data using a clustering algorithm to obtain a first clustering result of the basic data.

[0088] Step 102: Obtain the incremental data added to the basic data in each stage.

[0089] As streaming data is continuously input, in this step, the data flowing into the basic data is divided into different stages, and the incremental data added to the basic data in each stage is read in turn.

[0090] Step 103: Determine a second clustering result of the incremental data in each stage according to the incremental data in each stage.

[0091] In this step, after the incremental data of each stage is read in, a clustering algorithm is used for the incremental data of each stage to obtain a second clustering result of the incremental data of each stage.

[0092] Step 104: Obtaining a target clustering result of the entire data according to the first clustering result and the second clustering result;

[0093] The full amount of data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage.

[0094] In this step, the second clustering result of the incremental data of each stage is obtained, and clustering is performed through a clustering algorithm based on the first clustering result and the second clustering result to obtain the target clustering result of the full data.

[0095] The embodiments of the present invention reduce the computational time of clustering by means of incremental clustering, and first cluster the basic data, then cluster the incremental data, and then merge and cluster the first clustering result of the basic data and the second clustering result of the incremental data, thereby avoiding multiple readings in and out of the full amount of data, effectively reducing memory consumption, and lowering hardware equipment costs.

[0096] Optionally, the method further comprises:

[0097] According to the target clustering result, the target feature parameters of the entire data are obtained.

[0098] After obtaining the target clustering results of the full amount of data, the target clustering results are analyzed to obtain the target feature parameters of the full amount of data, so as to facilitate the subsequent application of the target feature parameters.

[0099] Optionally, determining a first clustering result of the basic data according to the basic data on the network platform includes:

[0100] Determining a first initial center of the basic data by a pre-selected center algorithm;

[0101] The basic data is clustered according to the first initial center to obtain the first clustering result.

[0102] The following specifically describes the process of obtaining the first clustering result of the basic data:

[0103] Read in the basic data, record the basic data set as D, and use the pre-selected center algorithm to determine the first initial center set C0 of the basic data. Use C0 as the initial cluster center set. After re-reading the basic data, perform clustering through the k-means clustering algorithm to obtain the first clustering result C corresponding to the basic data. As shown in the following formula:

[0104] C = f x (C0,D)

[0105] Among them, x is the number of iterations in the clustering process, C0 is the set of the first initial centers, and D is the set of basic data.

[0106] In the k-means clustering algorithm, the iteration variables are k initial center points:

[0107]

[0108] The new initial center value is calculated continuously. The input iteration variable is the iteration value, and the output iteration variable is the advanced value. The function of the iteration function is to calculate the distance from each data point to the center. The embodiment of the present invention uses the Euclidean distance to calculate the similarity between data points, and uses the similarity as the basis for dividing data points into specific classes, that is, it is considered that the distance between two data is inversely proportional to the similarity. Assume that all data exist in a set D, and any element d in D is a tuple<id,x,y> (Take two-dimensional features as an example), id is the unique identifier of the data, and x and y are the coordinates of the data after transformation. Therefore, the iterative function of the k-means clustering algorithm is expressed as follows:

[0109]

[0110] The steps to use the pre-selected center algorithm are as follows:

[0111] Read the first data point X1 of the basic data, add it to the initial center set and record it as C1. The effective range of the initial center C1 is the circular area with C1 as the center and the distance threshold T as the radius. For details, please refer to Figure 2 Similarly, the effective range of the initial center C2 in the initial center set is a circular area with C2 as the center and the distance threshold T as the radius, and the common samples within the effective range of the initial center C1 and the initial center C2 are the samples within the overlapping area of ​​the two circular areas.

[0112] Calculate the next data point X of the base data i The similarity with all the initial centers in the initial center set is expressed by data point X iThe distance S from the initial center k If there is a data point X i The distance between the data point X and a certain initial center is less than the distance threshold T, which means that the data point X i The features of the samples within the effective range of the initial center are similar and can be clustered into one category. In this case, the initial center set remains unchanged. For example, see Figure 3 , the initial center set includes the initial center C1 and the initial center C2, the data point X i The distance S1 from the initial center C1 is greater than T, and the data point X i The distance S2 from the initial center C2 is less than T, which means that the data point X i The characteristics of the samples within the effective range of the initial center C2 are similar and can be clustered into one category. In this case, the initial center set remains unchanged. i The distance between the data point X and any initial center is not less than the distance threshold T, which means that the data point X i does not belong to the valid range of any existing initial center, that is, the data point X i The characteristics of the data points X are quite different from those of the data sets corresponding to the initial centers. i Join the initial center set and serve as a new initial center. For example, see Figure 4 , the initial center set includes the initial center C1 and the initial center C2, the data point X i The distance S1 from the initial center C1 is greater than T, and the data point X i The distance S2 from the initial center C2 is greater than T, and the data point X i The characteristics of the data set corresponding to the initial center C1 and the characteristics of the data set corresponding to the initial center C2 are quite different. i Add the initial center set and use it as a new initial center C3. Loop and perform the above steps for each data point of the basic data in the data stream.

[0113] After the above steps, all the initial centers required by the k-means clustering algorithm can be obtained for each data point in the basic data. At the same time, the number of initial centers in the initial center set can be further calculated.

[0114] Optionally, obtaining incremental data added to the basic data at each stage includes:

[0115] When the amount of target data added to the basic data in the target stage reaches a preset threshold, determining that the target data is incremental data added to the basic data in the target stage;

[0116] or,

[0117] When the target data added to the basic data in the target stage has been added for a predetermined period of time, determining that the target data is incremental data added to the basic data in the target stage;

[0118] The target stage is one of each of the stages.

[0119] That is, when the incremental data added to the basic data reaches a preset threshold (fixed data amount), or when the duration of adding the incremental data to the basic data reaches a preset duration (data within a fixed time), it is determined as the incremental data of a stage, and the second clustering result is determined for the incremental data of this stage.

[0120] Optionally, determining a second clustering result of the incremental data of each stage according to the incremental data of each stage includes:

[0121] Determine the second initial center of the incremental data in each stage by a pre-selected center algorithm;

[0122] The incremental data of each stage is clustered according to the second initial center to obtain the second clustering result.

[0123] The following specifically describes the process of obtaining the second clustering result of incremental data:

[0124] As streaming data is continuously input, the pre-selected center algorithm is used to determine the set of the second initial centers during the data reading process. Get the newly added incremental data that did not participate in the previous stage of clustering, and record the set of incremental data for this stage as d. At this time, the union of the basic data set D and the incremental data set d is recorded as the full data. When the newly added data for this stage reaches the preset threshold or the joining time reaches the preset time, the pre-selected center algorithm is used to determine the set of the second initial centers ΔC0 of the set d of incremental data for this stage. ΔC0 is used as the initial clustering center set. After re-reading the incremental data for this stage, clustering is performed using the k-means clustering algorithm to obtain the second clustering result ΔC corresponding to the incremental data. As shown in the following formula:

[0125] ΔC=f y (ΔC0,d)

[0126] Where y is the number of iterations in the clustering process, ΔC0 is the set of the second initial centers, and d is the set of incremental data.

[0127] The steps of using the pre-selected center algorithm are basically the same as above and will not be repeated here.

[0128] Optionally, obtaining a target clustering result of the entire data according to the first clustering result and the second clustering result includes:

[0129] Determining a third initial center of the first clustering result and the second clustering result by a pre-selected center algorithm;

[0130] Clustering the entire data according to the third initial center to obtain a third clustering result;

[0131] The target clustering result is obtained according to the third clustering result and a preset timeliness requirement.

[0132] The data in the first clustering result C and the second clustering result ΔC are used as the data points of the full data. The pre-selected center algorithm is used to obtain the third initial center set c. C is used as the initial clustering center set. The k-means clustering algorithm is used to perform clustering to obtain the third clustering result. It should be noted that the third clustering result is also the initial clustering center of the full data. As shown below:

[0133]

[0134] Wherein, z is the number of iterations in the clustering process, c is the set of the third initial centers, C is the first clustering result C, and ΔC is the second clustering result.

[0135] The steps of using the pre-selected center algorithm are basically the same as above and will not be repeated here.

[0136] The first clustering result C and the second clustering result ΔC are the advance clustering results of the basic data D and the incremental data d. This step is to merge the first clustering result C and the second clustering result ΔC to find a better initial point for the full data and perform clustering. After that, the target clustering result of the full data can be obtained according to the third clustering result and the preset timeliness requirements.

[0137] Optionally, obtaining the target clustering result according to the third clustering result and a preset timeliness requirement includes:

[0138] In a case where the preset timeliness requirement is the first timeliness requirement, taking the third clustering result as the target clustering result;

[0139] In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result;

[0140] Among them, the first timeliness requirement is higher than the second timeliness requirement.

[0141] Specifically, when obtaining the third clustering result After that, on the one hand, in the scenario with high real-time requirement (first timeliness requirement), the third clustering result can be directly As the target clustering result of the full data, on the other hand, in the scenario where the real-time requirement is not high (the second timeliness requirement), the third clustering result can be As the initial clustering center, the k-means clustering algorithm is used to cluster the basic data of the incremental data added to the next stage, that is, the new full data, to obtain the target clustering result, as shown in the following formula:

[0142]

[0143] Among them, C * is the target clustering result, is the third clustering result, B is the clustering result of the basic data after adding the incremental data of the next stage, and ΔB is the clustering result of the incremental data of the next stage.

[0144] The target clustering result obtained is more accurate, and by using the first clustering result C and the second clustering result ΔC as the advance clustering results of the basic data D and the incremental data d, the first clustering result C and the second clustering result ΔC are merged to find a better initial point for the full data and cluster them, which can greatly reduce the number of iterations for obtaining the target clustering result of the full data.

[0145] Optionally, the method further comprises:

[0146] The basic data and the incremental data of each stage are stored.

[0147] After obtaining the basic data and the incremental data of each stage, the basic data and the incremental data are stored in the database to facilitate subsequent offline analysis of the basic data and the incremental data.

[0148] Combine the following Figure 5 , specifically illustrate the data clustering method of an embodiment of the present invention.

[0149] Basic step. Perform basic step operations on the existing basic data, specifically: read in the basic data, record the set of basic data as D, and use the pre-selected center algorithm to determine the first initial center set C0 of the basic data, use C0 as the initial cluster center set, and after re-reading the basic data, perform clustering through the k-means clustering algorithm to obtain the first clustering result C corresponding to the basic data, that is, the basic result set.

[0150] Incremental step. With the continuous input of streaming data, in the process of reading the incremental data of a stage, the pre-selected center algorithm is used to determine the set of the second initial centers. The set of newly input incremental data that did not participate in the clustering of the previous stage is recorded as d. At this time, the union of the set D of basic data and the set d of incremental data is recorded as the full data. When the newly added data of this stage reaches the preset threshold or the joining time reaches the preset time, the incremental step is executed, and the set ΔC0 of the second initial centers of the set d of incremental data of this stage determined by the pre-selected center algorithm is used as the initial clustering center set. After re-reading the incremental data of this stage, clustering is performed by the k-means clustering algorithm to obtain the second clustering result ΔC corresponding to the incremental data, that is, the incremental result set. Merging step. Using the data in the first clustering result C and the second clustering result ΔC as the data points of the full data, the pre-selected center algorithm is used to obtain the set c of the third initial centers. Using c as the initial clustering center set, clustering is performed by the k-means clustering algorithm to obtain the third clustering result, which is also the initial clustering center of the full data. The first clustering result C and the second clustering result ΔC are the advance clustering results of the basic data D and the incremental data d. The merging step is to merge the first clustering result C and the second clustering result ΔC to find the optimal initial point of the full data and perform clustering. After that, on the one hand, in the scenario with high real-time requirement (first timeliness requirement), the third clustering result can be directly As the target clustering result of the full data, on the other hand, in the scenario where the real-time requirement is not high (the second timeliness requirement), the third clustering result can be As the initial clustering center, the next stage of iteration is carried out, that is, the k-means clustering algorithm is used to perform clustering on the basic data of the incremental data added to the next stage, that is, the new full data, to obtain the target clustering result, that is, the advanced result set.

[0151] The data clustering method provided by the embodiment of the present invention can be applied to unbounded streaming data. In view of the characteristics of continuously adding new data in the streaming data scenario, combined with the iterative characteristics in the k-means clustering algorithm process, the characteristics of the basic data and the characteristics of the new data are taken into consideration, and the two data sets are processed separately. This can realize real-time processing of streaming data, and can reduce the waste of computing resources and performance loss caused by iterative re-clustering of the full amount of streaming data, and also avoid multiple readings of the full amount of data in and out, effectively reducing memory consumption and reducing the cost of hardware equipment.

[0152] The development of the Internet has enabled people to access a large number of online resources in a short period of time. The function of the current shopping website recommendation system is to recommend the things that users are most likely to want based on their behavioral data, so that users can more easily find the products they like and are willing to buy. The personalized recommendation system aims to effectively deal with the troubles caused by excessive information to users. By collecting and analyzing user behavior information on the Internet, a unique recommendation plan is formulated for users. The clustering algorithm clusters the closest individuals into a cluster based on similarity and then finds the product categories and specific products that users are concerned about. Due to its natural characteristics, it is often used in various recommendation scenarios.

[0153] Combine the following Figure 6 , describing a commodity recommendation method using the data clustering method of an embodiment of the present invention.

[0154] Collect user browsing, collection and purchase behavior data as stream data at the data platform layer (WEB layer); transmit the stream data to the Apache Flink stream processing framework in real time and reliably through the Apache Kafka message queue for task processing. In the Apache Flink stream processing framework, data cleaning and processing are performed, and fields of multiple dimensions are merged, and the user feature segment, behavior feature segment and product feature segment generated by the same behavior are taken as a record. Within the time window, several records generated by the user's behavior operation are taken as an interest, that is, a new data set, and the interest feature data (new data) is clustered using the data clustering method of the present invention to obtain a clustering result. According to the clustering results, the user behavior characteristics are analyzed, and the user group with similar behavior is determined, and the user group data is stored in the redis database for real-time viewing on the web end; the user data (several records generated by the user's behavior operation) is stored in the hbase database for subsequent offline analysis. According to the user group data results, a recommended list of associated products is obtained, and the list content is pushed to the user to complete the recommendation process.

[0155] The commodity recommendation method provided by the embodiment of the present invention uses an incremental clustering method to reduce the time consumption of clustering calculations, so that users can obtain real-time feedback from the platform during the shopping process, thereby improving user satisfaction during shopping. For user behavior data, each time a single record in the data stream is processed, it is very easy to cause a large deviation in the clustering results, and the clustering results are too greatly affected by the current operation records, resulting in overcorrection. The embodiment of the present invention uses multi-dimensional data within a recent window of the user for incremental clustering, which increases the weight of the user's recent behavior data in the full amount of data, reduces the impact of old data and erroneous operation data on the clustering results, and improves the accuracy of the recommended content. Since the user's behavioral characteristics are continuous and uncertain, the method of the embodiment of the present invention can perform cluster analysis on each group of user's behavioral data based on the time window, thereby minimizing the impact of the user's misoperation and irrelevant operation on the analysis result; the data is divided into incremental data and basic data, and calculated separately according to the data volume and timeliness, thereby reducing the waste of computing resources and performance loss caused by re-clustering iterations; the user's current behavior within a period of time is analyzed in real time, and the analysis results are applied to the recommendation plan for the user, thereby maximizing the value of the user's browsing data and improving the accuracy of the recommendation plan and user satisfaction.

[0156] like Figure 7 As shown, the embodiment of the present invention further provides a data clustering device, including:

[0157] A first determination module 701 is used to determine a first clustering result of the basic data according to the basic data on the network platform;

[0158] An acquisition module 702 is used to acquire incremental data added to the basic data at each stage;

[0159] A second determination module 703, configured to determine a second clustering result of the incremental data of each stage according to the incremental data of each stage;

[0160] A third determination module 704 is used to obtain a target clustering result of the entire data according to the first clustering result and the second clustering result;

[0161] The full amount of data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage.

[0162] The embodiment of the present invention determines the first clustering result of the basic data, determines the second clustering result of the incremental data added to the basic data in each stage, and determines the target clustering result of the full data based on the first clustering result and the second clustering result. It is possible to obtain the clustering result of the full data in real time, reduce the long data processing time of the data clustering method, avoid multiple reading and reading of the full data, effectively reduce memory consumption, and reduce the cost of hardware equipment.

[0163] Optionally, the device further comprises:

[0164] The fourth determination module is used to obtain the target feature parameters of the full amount of data according to the target clustering result.

[0165] Optionally, the first determining module 701 includes:

[0166] A first determining unit, configured to determine a first initial center of the basic data by using a pre-selected center algorithm;

[0167] The first clustering unit is used to cluster the basic data according to the first initial center to obtain the first clustering result.

[0168] Optionally, the acquisition module 702 includes:

[0169] An acquiring unit, configured to determine that the target data is incremental data added to the basic data in the target stage when the amount of target data added to the basic data in the target stage reaches a preset threshold;

[0170] or,

[0171] When the target data added to the basic data in the target stage has been added for a predetermined period of time, determining that the target data is incremental data added to the basic data in the target stage;

[0172] The target stage is one of each of the stages.

[0173] Optionally, the second determining module 703 includes:

[0174] A second determining unit, configured to determine a second initial center of the incremental data of each stage by a pre-selected center algorithm;

[0175] The second clustering unit is used to cluster the incremental data of each stage according to the second initial center to obtain the second clustering result.

[0176] Optionally, the third determining module 704 includes:

[0177] a third determining unit, configured to determine a third initial center of the first clustering result and the second clustering result by using a pre-selected center algorithm;

[0178] A third clustering unit, configured to cluster the entire data according to the third initial center to obtain a third clustering result;

[0179] The fourth determining unit is used to obtain the target clustering result according to the third clustering result and a preset timeliness requirement.

[0180] Optionally, the fourth determining unit is specifically configured to use the third clustering result as the target clustering result when the preset timeliness requirement is the first timeliness requirement;

[0181] In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result;

[0182] Among them, the first timeliness requirement is higher than the second timeliness requirement.

[0183] Optionally, the device further comprises:

[0184] A storage module is used to store the basic data and the incremental data of each stage.

[0185] It should be noted that the data clustering device provided in the embodiment of the present invention is a device capable of executing the above-mentioned data clustering method, and all embodiments of the above-mentioned data clustering method are applicable to the device and can achieve the same or similar technical effects.

[0186] like Figure 8 As shown, an embodiment of the present invention further provides a data clustering device, comprising: a processor 801, a memory 802, and a program stored in the memory 802 and executable on the processor 801, wherein the program implements the above-mentioned vehicle collision warning method when executed by the processor 801.

[0187] Optionally, it further includes: a transceiver 803, wherein the transceiver 803 is used to receive and send data under the control of the processor 801.

[0188] Specifically, the processor 801 is used to: determine a first clustering result of the basic data based on the basic data on the network platform; obtain incremental data added to the basic data in each stage; determine a second clustering result of the incremental data in each stage based on the incremental data in each stage; obtain a target clustering result of the full data based on the first clustering result and the second clustering result; wherein the full data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage.

[0189] Optionally, the processor 801 is further configured to:

[0190] According to the target clustering result, the target feature parameters of the entire data are obtained.

[0191] Optionally, the processor 801 is specifically configured to:

[0192] Determining a first initial center of the basic data by a pre-selected center algorithm;

[0193] The basic data is clustered according to the first initial center to obtain the first clustering result.

[0194] Optionally, the processor 801 is specifically configured to:

[0195] When the amount of target data added to the basic data in the target stage reaches a preset threshold, determining that the target data is incremental data added to the basic data in the target stage;

[0196] or,

[0197] When the target data added to the basic data in the target stage has been added for a predetermined period of time, determining that the target data is incremental data added to the basic data in the target stage;

[0198] The target stage is one of each of the stages.

[0199] Optionally, the processor 801 is specifically configured to:

[0200] Determine the second initial center of the incremental data in each stage by a pre-selected center algorithm;

[0201] The incremental data of each stage is clustered according to the second initial center to obtain the second clustering result.

[0202] Optionally, the processor 801 is specifically configured to:

[0203] Determining a third initial center of the first clustering result and the second clustering result by a pre-selected center algorithm;

[0204] Clustering the entire data according to the third initial center to obtain a third clustering result;

[0205] The target clustering result is obtained according to the third clustering result and a preset timeliness requirement.

[0206] Optionally, the processor 801 is specifically configured to:

[0207] In a case where the preset timeliness requirement is the first timeliness requirement, taking the third clustering result as the target clustering result;

[0208] In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result;

[0209] Among them, the first timeliness requirement is higher than the second timeliness requirement.

[0210] Optionally, the processor 801 is further configured to:

[0211] The basic data and the incremental data of each stage are stored.

[0212] Among them, Figure 8 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 801 and memory represented by memory 802. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides a user interface 804. The transceiver 803 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 801 is responsible for managing the bus architecture and general processing, and the memory 802 may store data used by the processor 801 when performing operations.

[0213] In addition, a specific embodiment of the present invention further provides a readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, the steps in the data clustering method described in any one of the above are implemented.

[0214] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles described in the present invention. These improvements and modifications are also within the scope of protection of the present invention.

Claims

1. A data clustering method, characterized in that: include: Determine a first clustering result of the basic data according to the basic data on the network platform; Obtaining incremental data added to the basic data at each stage; Determine a second clustering result of the incremental data of each stage according to the incremental data of each stage; Obtaining a target clustering result of the entire data according to the first clustering result and the second clustering result; The full amount of data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage; Wherein, obtaining a target clustering result of the entire data according to the first clustering result and the second clustering result includes: Determining a third initial center of the first clustering result and the second clustering result by a pre-selected center algorithm; Clustering the entire data according to the third initial center to obtain a third clustering result; Obtaining the target clustering result according to the third clustering result and the preset timeliness requirement; Wherein, obtaining the target clustering result according to the third clustering result and the preset timeliness requirement includes: In a case where the preset timeliness requirement is the first timeliness requirement, taking the third clustering result as the target clustering result; In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result; Among them, the first timeliness requirement is higher than the second timeliness requirement.

2. The data clustering method according to claim 1, characterized in that: The method further comprises: According to the target clustering result, the target feature parameters of the entire data are obtained.

3. The data clustering method according to claim 1, characterized in that: Determining a first clustering result of the basic data according to the basic data on the network platform includes: Determining a first initial center of the basic data by a pre-selected center algorithm; The basic data is clustered according to the first initial center to obtain the first clustering result.

4. The data clustering method according to claim 1, characterized in that: Obtain the incremental data added to the basic data at each stage, including: When the amount of target data added to the basic data in the target stage reaches a preset threshold, determining that the target data is incremental data added to the basic data in the target stage; or, When the target data added to the basic data in the target stage has been added for a predetermined period of time, determining that the target data is incremental data added to the basic data in the target stage; The target stage is one of each of the stages.

5. The data clustering method according to claim 1, characterized in that: Determining a second clustering result of the incremental data of each stage according to the incremental data of each stage includes: Determine the second initial center of the incremental data in each stage by a pre-selected center algorithm; The incremental data of each stage is clustered according to the second initial center to obtain the second clustering result.

6. The data clustering method according to claim 1, characterized in that: The method further comprises: The basic data and the incremental data of each stage are stored.

7. A data clustering device, characterized in that: include: A first determination module, used to determine a first clustering result of the basic data according to the basic data on the network platform; An acquisition module, used for acquiring incremental data added to the basic data at each stage; A second determination module, configured to determine a second clustering result of the incremental data of each stage according to the incremental data of each stage; A third determination module, configured to obtain a target clustering result of the entire data according to the first clustering result and the second clustering result; The full amount of data includes the basic data, the incremental data added to the basic data in the current stage, and the incremental data added to the basic data before the current stage; Wherein, the third determination module includes: a third determining unit, configured to determine a third initial center of the first clustering result and the second clustering result by using a pre-selected center algorithm; A third clustering unit, configured to cluster the entire data according to the third initial center to obtain a third clustering result; A fourth determining unit, configured to obtain the target clustering result according to the third clustering result and a preset timeliness requirement; The fourth determining unit is specifically configured to use the third clustering result as the target clustering result when the preset timeliness requirement is the first timeliness requirement; In a case where the preset timeliness requirement is the second timeliness requirement, adding incremental data of the next stage to the full data, clustering the full data according to the third clustering result, and obtaining the target clustering result; Among them, the first timeliness requirement is higher than the second timeliness requirement.

8. A data clustering device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the data clustering method according to any one of claims 1 to 6 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by a processor, the steps in the data clustering method according to any one of claims 1 to 6 are implemented.