Data clustering method and device

By determining the minimum clustering radius set and inflection point algorithm of the data set, selecting the clustering radius and clustering the data set, the problem of low accuracy when the number of clusters needs to be given in advance and density changes in the existing technology is solved, and the problem of low accuracy is achieved. Efficient multi-density data clustering.

CN110298371BActive Publication Date: 2024-11-29BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201810239450.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-03-22
Publication Date
2024-11-29
Estimated Expiration
2038-03-22

Smart Images

  • Figure CN110298371B_ABST
    Figure CN110298371B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for data clustering, and relates to the field of computer technology. A specific implementation of the method includes: determining a minimum clustering radius set of a data set to be clustered; determining at least one clustering radius from the minimum clustering radius set; and clustering the data set to be clustered according to at least one clustering radius. This implementation can obtain multiple clustering radii, and can divide the data set into clusters of different densities based on the multiple clustering radii, thereby improving the accuracy of clustering and reducing the complexity of calculation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data clustering. Background Art

[0002] In recent years, with the continuous innovation and progress of information technology, when conducting interactive data mining, users often use different analysis techniques, such as clustering, matching, filtering and visualization, to perform various tasks, so that users can make wise decisions. Among them, clustering technology can make the data of the same cluster (or class) clustered together as much as possible, and the data of different clusters (or classes) separated as much as possible, and clustering is widely used in market analysis, information security, finance and entertainment. Therefore, how to accurately cluster data is becoming increasingly important in interactive data mining in the era of big data.

[0003] The existing technology generally uses the K-MEANS algorithm or the DBSCAN algorithm based on a single density to cluster data. Among them, the K-MEANS algorithm is an algorithm that inputs the number of clusters k and a database containing n data, and outputs k clusters that meet the minimum variance standard. The DBSCAN algorithm based on a single density (Density-Based Spatial Clustering ofApplications with Noise, a more representative density-based clustering algorithm) defines a cluster as the largest set of density-connected points, which can divide areas with sufficiently high density into clusters, and can find clusters of any shape in a noisy spatial data set.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: First, the number of clusters of the K-MEANS algorithm needs to be given in advance, and the initial cluster center needs to be determined manually, and the algorithm cannot automatically detect outliers, where the outlier category refers to a category with very few data points after clustering is completed, and the data points in the outlier category are called outliers; Second, the DBSCAN algorithm based on a single density clusters data based on a single radius. If the data set has multiple densities, the algorithm cannot accurately distinguish the changing densities, and the algorithm needs to calculate the Euclidean distance, which is highly complex. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for data clustering, which can improve the accuracy of clustering and reduce the complexity of calculation.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data clustering is provided.

[0007] A data push method according to an embodiment of the present invention includes: determining a minimum clustering radius set of data sets to be clustered; determining at least one clustering radius from the minimum clustering radius set; and clustering the data sets to be clustered according to the at least one clustering radius.

[0008] Optionally, obtaining the minimum clustering radius set of the data set to be clustered includes: for each data in the data set to be clustered, selecting multiple reference data from the data set to be clustered based on a data selection rule, then calculating multiple Manhattan distances between each data and the multiple reference data, and calculating the mean of the multiple Manhattan distances; generating the minimum clustering radius set according to the mean of the Manhattan distance corresponding to each data.

[0009] Optionally, the data selection rule includes: at least one of a rule for selecting reference data based on distance and a rule for selecting reference data based on quantity, wherein the rule for selecting reference data based on distance includes: for data a in data set A, selecting data from data set A whose Manhattan distance with the data a is less than a preset distance value; and the rule for selecting reference data based on quantity includes: for data a in data set A, arranging the Manhattan distances between other data in data set A and the data a from small to large, and then selecting a preset number of data starting from other data corresponding to the minimum distance value.

[0010] Optionally, determining at least one clustering radius from the minimum clustering radius set includes: arranging the elements in the minimum clustering radius set in ascending or descending order; calculating the inflection point of the sorted minimum clustering radius set using an inflection point algorithm; and determining at least one clustering radius based on the number element corresponding to the inflection point.

[0011] Optionally, clustering the data set to be clustered according to the at least one clustering radius includes: clustering the target data set based on each clustering radius in the order of the at least one clustering radius from small to large to obtain clusters and noise data sets corresponding to the clustering radius, wherein the target data set is the noise data set obtained after the previous clustering, and the target data set of the initial clustering is the data set to be clustered.

[0012] To achieve the above objective, according to another aspect of an embodiment of the present invention, a device for data clustering is provided.

[0013] A data clustering device according to an embodiment of the present invention includes: a first determination module, used to determine a minimum clustering radius set of a data set to be clustered; a second determination module, used to determine at least one clustering radius from the minimum clustering radius set; and a clustering module, used to cluster the data set to be clustered according to the at least one clustering radius.

[0014] Optionally, the first determination module is also used to: for each data in the data set to be clustered, select multiple reference data from the data set to be clustered based on data selection rules, then calculate multiple Manhattan distances between each data and the multiple reference data, and calculate the mean of the multiple Manhattan distances; generate the minimum clustering radius set according to the mean of the Manhattan distance corresponding to each data.

[0015] Optionally, the data selection rule includes: at least one of a rule for selecting reference data based on distance and a rule for selecting reference data based on quantity, wherein the rule for selecting reference data based on distance includes: for data a in data set A, selecting data from data set A whose Manhattan distance with the data a is less than a preset distance value; and the rule for selecting reference data based on quantity includes: for data a in data set A, arranging the Manhattan distances between other data in data set A and the data a from small to large, and then selecting a preset number of data starting from other data corresponding to the minimum distance value.

[0016] Optionally, the second determination module is also used to: arrange the elements in the minimum clustering radius set in ascending or descending order; calculate the inflection point of the sorted minimum clustering radius set using an inflection point algorithm; and determine at least one clustering radius based on the number element corresponding to the inflection point.

[0017] Optionally, the clustering module is also used to: cluster the target data set based on each clustering radius in the order of the at least one clustering radius from small to large to obtain a cluster and a noise data set corresponding to the clustering radius, wherein the target data set is the noise data set obtained after the previous clustering, and the target data set of the initial clustering is the data set to be clustered.

[0018] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0019] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data clustering method according to the embodiment of the present invention.

[0020] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0021] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the method for data clustering according to an embodiment of the present invention is implemented.

[0022] An embodiment of the above invention has the following advantages or beneficial effects: multiple clustering radii can be obtained, so that the data set can be divided into clusters of different densities based on multiple clustering radii, thereby improving the accuracy of clustering and reducing the complexity of calculation; in the embodiment of the present invention, the corresponding reference data is selected for each data, and then the minimum clustering radius set of the data set is determined by calculating the Manhattan distance between the data and the reference data, thereby greatly reducing the complexity of distance calculation while ensuring the accuracy of distance calculation; in the embodiment of the present invention, reference data can be selected from multiple angles of distance and quantity, so that adaptive adjustments can be made according to actual conditions, further improving the practicality and accuracy of the solution; in the embodiment of the present invention, the inflection point algorithm is used to select the clustering radius from the minimum clustering radius set, so that different clustering radii can be determined, and then the data set can be divided into clusters of different densities; in the embodiment of the present invention, the data set is clustered in the order of clustering radius from small to large, so that clusters with very high density, medium density and small density can be selected from the data set in turn, further improving the accuracy of clustering.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0025] Figure 1 is a schematic diagram of data sets with different densities;

[0026] Figure 2 is a schematic diagram of main steps of a method for data clustering according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of determining a clustering radius according to a method for data clustering according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of the main process of the method for data clustering according to an embodiment of the present invention;

[0029] Figure 5 is a schematic diagram of main modules of a data clustering device according to an embodiment of the present invention;

[0030] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0031] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0033] Clustering refers to dividing a data set into different clusters (or classes) according to a certain standard (such as a distance criterion or a similarity criterion, i.e., the distance or similarity between data points), so that the similarity of user features within the same cluster (or class) is as large as possible or the distance is as small as possible, and the difference of user features in different clusters (or classes) is as large as possible. The basic steps of the K-MEANS algorithm for implementing data clustering in the prior art include: (1) arbitrarily selecting k objects from n data objects as the initial cluster centers; (2) calculating the distance between each object and these center objects based on the mean (center object) of each cluster object, and re-dividing the corresponding objects based on the minimum distance; (3) recalculating the mean (center object) of each cluster with changes; (4) calculating the standard measurement function, and when certain conditions are met, such as when the function converges, the algorithm terminates; if the conditions are not met, return to step (2).

[0034] Although the K-MEANS algorithm can realize data clustering, it needs to determine k in advance and cannot identify noise points (i.e., outliers). Therefore, the DBSCAN algorithm based on single density came into being. The DBSCAN algorithm requires two input parameters: scanning radius (eps) and minimum number of included points (minPts). The specific steps are: starting from any unvisited point, find all nearby points within eps (including eps); if the number of nearby points is not less than minPts, the current point and its nearby points form a cluster, and the starting point is marked as visited; then recursively, all points in the cluster that are not marked as visited are processed in the same way, thereby expanding the cluster; if the number of nearby points is less than minPts, the point is temporarily marked as a noise point; if the cluster is fully expanded, that is, all points in the cluster are marked as visited, and then the same algorithm is used to process the unvisited points.

[0035] Although the DBSCAN algorithm based on a single density can detect noise points without presetting the number of clusters, it cannot accurately distinguish the changing density if the data set has multiple densities, for example Figure 1 is a schematic diagram of data sets with different densities, through Figure 1It can be observed that there are three categories, and these three categories have different densities. The density of the category on the far left is relatively small, the density of the middle category is medium, and the density of the category on the far right is very high. The data clustering method of the prior art cannot accurately distinguish these three categories, that is, if a small radius is selected, it may cause the data on the left to be judged as outliers. If a larger radius is selected, the data of two different categories that originally belong to a higher density will be clustered into one category, which will lead to low clustering accuracy. With the advancement of technology, data collection has become easier and easier, resulting in larger and more complex databases, such as various types of trade transaction data, documents, gene expression data, etc. The dimensions (attributes) of the data can usually reach hundreds or thousands of dimensions, or even higher. Therefore, the present invention provides a method for data clustering, which can divide a data set into clusters of different densities based on multiple clustering radii (i.e., scanning radii).

[0036] Figure 2 is a schematic diagram of the main steps of the method for data clustering according to an embodiment of the present invention, such as Figure 2 As shown, the main steps of the data clustering method of the embodiment of the present invention may include:

[0037] Step S201: Determine the minimum clustering radius set of the data set to be clustered. In the present invention, firstly, the data set to be clustered is acquired, and then the minimum clustering radius set is determined according to the data set.

[0038] Step S202: Determine at least one clustering radius from the minimum clustering radius set. After obtaining the minimum clustering radius set in step S201, select at least one clustering radius from the set.

[0039] Step S203: Clustering the data set to be clustered according to at least one clustering radius. Clustering the data set according to the clustering radius obtained in step 202. In the present invention, DBSCAN clustering is performed on the data set based on multiple clustering radii.

[0040] In an embodiment of the present invention, obtaining a minimum clustering radius set of a data set to be clustered may include: for each data in the data set to be clustered, selecting multiple (in the present invention, multiple is a preset number, which may be one, and the specific value is determined according to the size of the quantity set) reference data from the data set to be clustered based on a data selection rule, then calculating multiple Manhattan distances between the data and the multiple reference data, and calculating the mean of the multiple Manhattan distances; generating a minimum clustering radius set according to the mean of the Manhattan distance corresponding to each data. In the present invention, assuming that there are n data in the data set A to be clustered, for each data a, first determine the m reference data of data a, then calculate the Manhattan distances between data a and the m reference data respectively, and then calculate the mean d of the Manhattan distances, then there are n Manhattan means for the data set A, constituting a minimum clustering radius set D. Considering that in the present invention, the m reference data of data a are the m data with the smallest Manhattan distance to data a, the set consisting of the mean of the Manhattan distance corresponding to each data is called the minimum clustering radius set in the present invention. Among them, for j-dimensional data, the Manhattan distance calculation method of two data x and y is as follows:

[0041] Where r=1.

[0042] If r in the above formula is set to 2, it is the calculation method of Euclidean distance, so Euclidean distance needs to calculate the sum of squares and square roots, which is a slower and more computationally complex calculation method. Manhattan distance only needs simple numerical addition and subtraction operations, and its calculation complexity is much lower than that of Euclidean distance, thus greatly reducing the calculation overhead and improving clustering performance and speed.

[0043] In an embodiment of the present invention, the data selection rule may include: at least one of a rule for selecting reference data based on distance and a rule for selecting reference data based on quantity. In the present invention, for data set A, the Manhattan distance between each pair is first calculated. Selecting reference data based on distance means, for a data a, selecting data from data set A whose Manhattan distance with data a is less than a preset distance value. Selecting reference data based on quantity means, for a data a, arranging the Manhattan distances between other data in data set A and data a from small to large, and then selecting a preset number of reference data from other data corresponding to the minimum distance value. In the embodiment of the present invention, reference data is selected from multiple angles of distance and quantity, so that adaptive adjustments can be made according to actual conditions, further improving the practicality and accuracy of the solution.

[0044] In an embodiment of the present invention, determining at least one clustering radius from a minimum clustering radius set may include: arranging the elements in the minimum clustering radius set in ascending or descending order; calculating the inflection point of the sorted minimum clustering radius set using an inflection point algorithm; and determining at least one clustering radius according to the element corresponding to the inflection point. In the present invention, the elements in the minimum clustering set D are arranged in order from large to small or from small to large, and the change point of the minimum distance is calculated by the inflection point calculation method, and then multiple clustering radii are selected for DBSCAN clustering. Among them, the inflection point, also known as the inflection point, refers to the point where the curve changes the upward or downward direction in mathematics. Intuitively speaking, the inflection point is the point where the tangent crosses the curve (i.e., the concave and convex dividing point of the curve). If the function of the curve graph has a second-order derivative at the inflection point, the second-order derivative has a different sign (from positive to negative or from negative to positive) or does not exist at the inflection point. The inflection point algorithm uses the inflection point obtained by calculating the second-order derivative as the "minimum distance change point".

[0045] Figure 3 FIG. 4 is a schematic diagram of determining a clustering radius according to a method for data clustering according to an embodiment of the present invention. Figure 3 As shown, taking two-dimensional data as an example, Figure 3 The left side shows the distribution of 1000 two-dimensional data sets. The minimum cluster radius set D is generated for the data set in the above way, and then the elements are arranged in reverse order to generate Figure 3 The distribution of the minimum clustering radius is shown on the right. The horizontal axis is the 1000 data numbers, and the vertical axis is the mean Manhattan distance from the corresponding data to its nearest 4 reference data (the number of reference data values ​​is 4). It can be seen from the curve that it contains three relatively flat distributions. There is an inflection point in front of each flat curve. The first ordinate of each flat distribution can be used as the value of the clustering radius. For the figure below, the inflection point algorithm can be used to obtain 3 selected clustering radius values: the value corresponding to the 65th data point is 1834, the value corresponding to the 443rd data point is 586, and the value corresponding to the 590th data point is 361.

[0046] In an embodiment of the present invention, clustering a dataset to be clustered according to at least one clustering radius may include: in the order from smallest to largest of the at least one clustering radius, clustering the target dataset based on each clustering radius to obtain the clusters corresponding to the clustering radius and the noise dataset. Wherein, the target dataset is the noise dataset obtained after the previous clustering, and the target dataset for the initial clustering is the dataset to be clustered. In the present invention, it is assumed that the dataset to be clustered is A, and three clustering radii d1, d2, and d3 are obtained, and d1 < d2 < d3. First, perform DBSCAN clustering on the dataset A with the clustering radius d1 as the scanning radius to obtain the clusters and the noise dataset c1 based on the clustering radius d1. Then, perform DBSCAN clustering on the noise dataset c1 with the clustering radius d2 to obtain the clusters and the noise dataset c2 based on the clustering radius d2. Finally, perform DBSCAN clustering on the noise dataset c2 with the clustering radius d3 to obtain the clusters and the noise dataset c3 based on the clustering radius d3. In this way, the clustering of the entire dataset A based on three densities is completed.

[0047] Figure 4 is a schematic diagram of the main process of the data clustering method according to an embodiment of the present invention, as Figure 4 shown, the main process of the data clustering method according to an embodiment of the present invention may include: Step S401, obtain the dataset A to be clustered; Step S402, for each data a in the dataset to be clustered, select m reference data corresponding to the data a, and calculate m Manhattan distances between the data a and its corresponding m reference data to obtain the mean d of the m Manhattan distances; Step S403, generate a set D of minimum clustering radii according to the mean d of the Manhattan distances corresponding to each data; Step S404, arrange the elements in the set D of minimum clustering radii in ascending or descending order to obtain the inflection points of the set D of minimum clustering radii; Step S405, obtain the clustering radius according to the element corresponding to the inflection point; Step S406, in the order from smallest to largest of the clustering radius, cluster the target dataset based on each clustering radius to obtain the clusters corresponding to the clustering radius and the noise dataset. Wherein, the target dataset in Step S406 is the noise dataset obtained after the previous clustering, and the target dataset for the initial clustering is the dataset A to be clustered.

[0048] It can be seen from the technical solution of data clustering according to the embodiment of the present invention that multiple clustering radii can be obtained, so that the data set can be divided into clusters of different densities based on multiple clustering radii, thereby improving the accuracy of clustering and reducing the complexity of calculation; in the embodiment of the present invention, the corresponding reference data is selected for each data, and then the minimum clustering radius set of the data set is determined by calculating the Manhattan distance between the data and the reference data, thereby greatly reducing the complexity of distance calculation under the premise of ensuring the accuracy of distance calculation; in the embodiment of the present invention, reference data is selected from multiple angles of distance and quantity, so that adaptive adjustments can be made according to actual conditions, further improving the practicality and accuracy of the solution; in the embodiment of the present invention, the inflection point algorithm is used to select the clustering radius from the minimum clustering radius set, so that different clustering radii can be determined, and then the data set can be divided into clusters of different densities; in the embodiment of the present invention, the data set is clustered in the order of clustering radius from small to large, so that clusters with very high density, medium density and small density can be selected from the data set in turn, further improving the accuracy of clustering.

[0049] Figure 5 FIG. 1 is a schematic diagram of the main modules of the data clustering device according to an embodiment of the present invention. Figure 5 As shown, the data clustering device 500 according to the embodiment of the present invention mainly includes the following modules: a first determination module 501 , a second determination module 502 and a clustering module 503 .

[0050] The first determination module 501 may be used to obtain a minimum clustering radius set of the data set to be clustered. The second determination module 502 may be used to determine at least one clustering radius from the minimum clustering radius set. The clustering module 503 may be used to cluster the data set to be clustered according to at least one clustering radius.

[0051] In an embodiment of the present invention, the first determination module 501 can also be used to: for each data in the data set to be clustered, select multiple reference data from the data set to be clustered based on the data selection rule, then calculate multiple Manhattan distances between the data and the multiple reference data, and calculate the mean of the multiple Manhattan distances; generate a minimum clustering radius set according to the mean of the Manhattan distance corresponding to each data.

[0052] In the embodiment of the present invention, the data selection rule may include at least one of a rule of selecting reference data according to distance and a rule of selecting reference data according to quantity.

[0053] In an embodiment of the present invention, the second determination module 502 can also be used to: arrange the elements in the minimum clustering radius set in ascending or descending order; calculate the inflection point of the sorted minimum clustering radius set using an inflection point algorithm; and determine at least one clustering radius based on the element corresponding to the inflection point.

[0054] In an embodiment of the present invention, the clustering module 503 can also be used to: cluster the target data set based on each clustering radius in the order of at least one clustering radius from small to large to obtain a cluster and a noise data set corresponding to the clustering radius, wherein the target data set is the noise data set obtained after the previous clustering, and the target data set of the initial clustering is the data set to be clustered.

[0055] From the above description, it can be seen that multiple clustering radii can be obtained, so that the data set can be divided into clusters of different densities based on multiple clustering radii, which improves the accuracy of clustering and reduces the complexity of calculation; in the embodiment of the present invention, the corresponding reference data is selected for each data, and then the minimum clustering radius set of the data set is determined by calculating the Manhattan distance between the data and the reference data, so that the complexity of distance calculation can be greatly reduced under the premise of ensuring the accuracy of distance calculation; in the embodiment of the present invention, reference data can be selected from multiple angles of distance and quantity, so that adaptive adjustments can be made according to actual conditions, further improving the practicality and accuracy of the solution; in the embodiment of the present invention, the inflection point algorithm is used to select the clustering radius from the minimum clustering radius set, so that different clustering radii can be determined, and then the data set can be divided into clusters of different densities; in the embodiment of the present invention, the data set is clustered in the order of clustering radius from small to large, so that clusters with very high density, medium density and small density can be selected from the data set in turn, further improving the accuracy of clustering.

[0056] Figure 6 An exemplary system architecture 600 is shown to which the method for data clustering or the apparatus for data clustering according to the embodiment of the present invention can be applied.

[0057] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0058] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0059] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0060] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0061] It should be noted that the data clustering method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data clustering device is generally disposed in the server 605 .

[0062] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0063] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0064] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0065] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0066] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0067] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0068] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0069] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a first determination module, a second determination module and a clustering module. The names of these modules do not, in some cases, constitute limitations on the modules themselves, for example, the first determination module may also be described as a "module for determining a minimum clustering radius set of a data set to be clustered".

[0070] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: determining a minimum clustering radius set of a data set to be clustered; determining at least one clustering radius from the minimum clustering radius set; and clustering the data set to be clustered according to at least one clustering radius.

[0071] According to the technical solution of the embodiment of the present invention, multiple clustering radii can be obtained, so that the data set can be divided into clusters of different densities based on multiple clustering radii, thereby improving the accuracy of clustering and reducing the complexity of calculation; in the embodiment of the present invention, the corresponding reference data is selected for each data, and then the minimum clustering radius set of the data set is determined by calculating the Manhattan distance between the data and the reference data, thereby greatly reducing the complexity of distance calculation while ensuring the accuracy of distance calculation; in the embodiment of the present invention, reference data can be selected from multiple angles of distance and quantity, so that adaptive adjustments can be made according to actual conditions, further improving the practicality and accuracy of the solution; in the embodiment of the present invention, the inflection point algorithm is used to select the clustering radius from the minimum clustering radius set, so that different clustering radii can be determined, and then the data set can be divided into clusters of different densities; in the embodiment of the present invention, the data set is clustered in the order of clustering radius from small to large, so that clusters with very high density, medium density and small density can be selected from the data set in turn, further improving the accuracy of clustering.

[0072] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data clustering, characterized in that: The server is used to receive the product information query request transmitted by the terminal device, analyze and process the data of the product information query request, and feed back the processing result to the terminal device. The analysis and processing process includes: Determining the minimum clustering radius set of the data set to be clustered, comprising: for each data in the data set to be clustered, selecting multiple reference data from the data set to be clustered based on a data selection rule, then calculating multiple Manhattan distances between each data and the multiple reference data, and calculating the mean of the multiple Manhattan distances; generating the minimum clustering radius set according to the mean of the Manhattan distance corresponding to each data; wherein the data set includes data of a product information query request, and the data is at least one of trade transaction data, documents, and gene expression data; the multiple reference data of each data are multiple data with the smallest Manhattan distance to each data; Determining at least one clustering radius from the minimum clustering radius set includes: arranging the elements in the minimum clustering radius set in descending order; calculating the inflection point of the sorted minimum clustering radius set using an inflection point algorithm; and determining at least one clustering radius according to the element corresponding to the inflection point; Clustering the data set to be clustered according to the at least one clustering radius includes: clustering the target data set based on each clustering radius in the order of the at least one clustering radius from small to large to obtain clusters and noise data sets corresponding to the clustering radius, wherein the target data set is the noise data set obtained after the previous clustering, and the target data set of the initial clustering is the data set to be clustered.

2. The method according to claim 1, characterized in that The data selection rule includes: at least one of a rule of selecting reference data based on distance and a rule of selecting reference data based on quantity, wherein the rule of selecting reference data based on distance includes: for data a in data set A, selecting data from data set A whose Manhattan distance to data a is less than a preset distance value; and The rule for selecting reference data based on quantity includes: for data a in data set A, arranging the Manhattan distances between other data in data set A and data a from small to large, and then selecting a preset number of data starting from other data corresponding to the minimum distance.

3. The method according to claim 1, characterized in that Determining at least one clustering radius from the minimum clustering radius set includes: Arrange the elements in the minimum cluster radius set in ascending order; Calculating the inflection point of the sorted minimum cluster radius set using an inflection point algorithm; At least one clustering radius is determined according to the element corresponding to the inflection point.

4. A data clustering device, characterized in that: The server is executed to receive the product information query request transmitted by the terminal device, analyze and process the data of the product information query request, and feed back the processing result to the terminal device, including: The first determination module is used to determine the minimum clustering radius set of the data set to be clustered, including: for each data in the data set to be clustered, multiple reference data are selected from the data set to be clustered based on the data selection rule, and then multiple Manhattan distances between each data and the multiple reference data are calculated, and the mean of the multiple Manhattan distances is calculated; the minimum clustering radius set is generated according to the mean of the Manhattan distance corresponding to each data; wherein the data set includes data of the product information query request, and the data is at least one of trade transaction data, documents, and gene expression data; the multiple reference data of each data are multiple data with the smallest Manhattan distance between each data; The second determination module is used to determine at least one clustering radius from the minimum clustering radius set, including: arranging the elements in the minimum clustering radius set in descending order; calculating the inflection point of the sorted minimum clustering radius set using an inflection point algorithm; and determining at least one clustering radius according to the element corresponding to the inflection point; A clustering module is used to cluster the data set to be clustered according to the at least one clustering radius, including: clustering the target data set based on each clustering radius in the order of the at least one clustering radius from small to large to obtain clusters and noise data sets corresponding to the clustering radius, wherein the target data set is the noise data set obtained after the previous clustering, and the target data set of the initial clustering is the data set to be clustered.

5. The device according to claim 4, characterized in that The data selection rule includes: at least one of a rule of selecting reference data based on distance and a rule of selecting reference data based on quantity, wherein the rule of selecting reference data based on distance includes: for data a in data set A, selecting data from data set A whose Manhattan distance to data a is less than a preset distance value; and The rule for selecting reference data based on quantity includes: for data a in data set A, arranging the Manhattan distances between other data in data set A and data a from small to large, and then selecting a preset number of data starting from other data corresponding to the minimum distance.

6. The device according to claim 4, characterized in that The second determining module is further used for: Arrange the elements in the minimum cluster radius set in ascending order; Calculating the inflection point of the sorted minimum cluster radius set using an inflection point algorithm; At least one clustering radius is determined according to the element corresponding to the inflection point.

7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Data clustering method based on dimensionality reduction and sampling

    CN105913077A

  • Urban business-circle cluster partition method based on self-adaptive DBSCAN density clustering

    CN107688955A