Data classification methods, devices and computing equipment

Abstract

This application provides a data classification method, apparatus, and computing device. The method includes: acquiring N data to be classified; determining the similarity between every two data in the N data; and, based on the similarity between every two data in the N data, identifying two data with a similarity greater than a first preset value as a relation pair, thereby obtaining M relation pairs; pre-clustering the N data according to the association and arrangement order of these M relation pairs to obtain P first connected components; then, obtaining Q second connected components according to the association between each of the P first connected components; and identifying the data within each of the Q second connected components as data of the same type. This solves the problem of high computational complexity and time consumption caused by excessive number of nodes and edges when calculating connected components.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data classification method, apparatus, and computing device. Background Technology

[0002] With the development of big data, it is necessary to classify the acquired big data. For example, in payment platforms, when merchants conduct transactions through the platform, they need to register, resulting in a large amount of merchant registration information. To facilitate merchant management, this registration information needs to be classified.

[0003] For example, the clique algorithm, which uses similarity calculation, can be used to classify data. Specifically, the similarity matrix of the data is calculated, and the similarity matrix is ​​used as input to construct an undirected graph. Connected components are searched in the undirected graph, and data within the same connected component are identified as data of the same type, thereby achieving data classification.

[0004] However, the current computation of connected components is time-consuming, resulting in low data classification efficiency. Summary of the Invention

[0005] This application provides a data classification method, apparatus, and computing device to reduce the computation time of connected components, thereby improving data classification efficiency.

[0006] In a first aspect, embodiments of this application provide a data classification method, including:

[0007] Obtain N data points to be classified, and determine the similarity between every two data points among the N data points;

[0008] Based on the similarity between any two data points in the N data points, two data points with a similarity greater than a first preset value are identified as a relation pair, resulting in M ​​relation pairs;

[0009] Based on the association and arrangement order among the M relation pairs, the N data are pre-clustered to obtain P first connected components;

[0010] Based on the correlation between the P first connected components, Q second connected components are obtained;

[0011] The data within each of the Q second connected components are determined to be of the same type.

[0012] Wherein, N, M, P, and Q are all positive integers, and Q is less than or equal to P.

[0013] Secondly, embodiments of this application provide a data classification apparatus, comprising:

[0014] The acquisition unit is used to acquire N data points to be classified.

[0015] A similarity determination unit is used to determine the similarity between every two data points in the N data points;

[0016] The relationship pair determination unit is used to determine two data pairs with a similarity greater than a first preset value as a relationship pair based on the similarity between every two data pairs in the N data pairs, thereby obtaining M relationship pairs;

[0017] The pre-clustering unit is used to pre-cluster the N data according to the association relationship and arrangement order between the M relation pairs to obtain P first connected components;

[0018] A connected component determination unit is used to obtain Q second connected components based on the correlation relationship between each of the P first connected components;

[0019] The classification unit is used to determine the data in each of the Q second connected components as data of the same type;

[0020] Wherein, N, M, P, and Q are all positive integers, and Q is less than or equal to P.

[0021] In some embodiments, the pre-clustering unit is specifically configured to, for the i-th relation pair in the arrangement of the M relation pairs, if the i-th relation pair contains first data that is the same as the first connected component of the current pre-cluster, then connect the i-th relation pair with the first data in the first connected component, where i is a positive integer less than or equal to M; if the i-th relation pair does not contain first data that is the same as the first connected component of the current pre-cluster, then use the data in the i-th relation pair as the starting node of the new first connected component to construct the new first connected component; if the two data in the i-th relation pair are located in two different first connected components, then determine the i-th relation pair as the relation pair between the two different first connected components.

[0022] In some embodiments, the connected component determination unit is further configured to determine two data points with a similarity less than or equal to the first preset value as independent nodes based on the similarity between every two data points in the N data points, thereby obtaining K independent nodes, where K is a positive integer; and to determine each of the independent nodes as a first connected component.

[0023] In some embodiments, the connected component determination unit is specifically configured to, for each of the P first connected components, if there is a relation pair between the first connected component and other first connected components, merge the first connected component and the other first connected components into a second connected component; if there is no relation pair between the first connected component and other first connected components, determine the first connected component as a second connected component, wherein the other first connected components are the first connected components other than the first connected component among the P first connected components.

[0024] In some embodiments, the connected component determination unit is specifically used to treat each of the P first connected components as a node, and the line connecting the relationship pair between two different first connected components as an edge, to perform connected component calculation and obtain the Q second connected components.

[0025] In some embodiments, the connected component determination unit is specifically used to treat each of the P first connected components as a node, the connection between two different first connected components as an edge, and use a generalized first search method or a depth-first search method to obtain the Q second connected components.

[0026] In some embodiments, the similarity determination unit is specifically configured to divide the N data into at least one first data group based on the attribute information of each data in the N data; send the data of each first data group to different computing devices for similarity calculation, and obtain the similarity between every two data in each first data group from the different computing devices.

[0027] In some embodiments, the similarity determination unit is specifically configured to, for each of the at least one first data group, if the amount of data in the first data group is greater than a second preset value, divide the data in the first data group into F data blocks according to a preset data block amount threshold, where F is a positive integer; and for every two data blocks in the F data blocks, send the two data blocks to a first computing device so that the first computing device can calculate the similarity between every two data blocks in the two data blocks.

[0028] In some embodiments, the similarity determination unit is specifically used to add 1 to the integer division result of the data amount in the first data group and the data amount threshold, and use this result as the number of data blocks corresponding to the first data group; and to divide the data in the first data group into F data blocks on an average basis according to the data amount in the first data group and the number of data blocks corresponding to the first data group.

[0029] In some embodiments, the similarity determination unit is specifically configured to obtain R combinations of pairwise data blocks based on different combinations of pairwise data blocks in the F data blocks, wherein the R combinations of pairwise data blocks include combinations of two identical data blocks and combinations of two different data blocks in the F data blocks, and R is a positive integer; for each pairwise data block combination in the R combinations of pairwise data blocks, the two data blocks in the pairwise data block combination are sent to the first computing device.

[0030] In some embodiments, the similarity determination unit is specifically used to obtain an FXF data block matrix based on different combinations of pairs of data blocks in the F data blocks; and to determine the combinations of pairs of data blocks located in the upper triangle of the FXF data block matrix as the R pairs of data blocks.

[0031] In some embodiments, the similarity determination unit is further configured to send the data in the first data group to a second computing device for similarity calculation if the amount of data in the first data group is less than or equal to the second preset value.

[0032] In some embodiments, if the attribute information of the data includes a naming pattern, the acquisition unit 11 is specifically used to perform word segmentation and part-of-speech tagging on each of the N data to obtain the naming pattern of each data; and to divide the data with the same naming pattern among the N data into a first data group.

[0033] Thirdly, embodiments of this application provide a computing device, including a processor and a memory;

[0034] The memory is used to store computer programs;

[0035] The processor is configured to execute the computer program to implement the method described in the first aspect above.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed by a computer, cause the computer to perform the method described in the first aspect.

[0037] Fifthly, embodiments of this application provide a computer program product comprising a computer program stored in a readable storage medium, wherein at least one processor of a computer can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the computer to perform the method described in the first aspect.

[0038] The data classification method, apparatus, and computing device provided in this application determine the similarity between every two data points out of N data points. Based on the similarity between every two data points out of N data points, two data points with a similarity greater than a first preset value are identified as a relation pair, resulting in M ​​relation pairs. According to the association and order of these M relation pairs, the N data points are pre-clustered to obtain P first connected components. Then, based on the association between each of the P first connected components, Q second connected components are obtained. Data within each of the Q second connected components is identified as data of the same type, thereby achieving data classification. Therefore, this application first performs pre-clustering during connected component calculation, and then performs connected component calculation based on the pre-clustering, thus solving the problem of high computational complexity and time consumption caused by an excessive number of nodes and edges. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 This is a schematic diagram of an optional structure of a distributed system applied to a blockchain system, as provided in an embodiment of this application.

[0041] Figure 2 A schematic diagram illustrating the principle of the clique algorithm for similarity calculation;

[0042] Figure 3 A flowchart illustrating a data classification method provided in an embodiment of this application;

[0043] Figure 4 A schematic diagram of a connected component;

[0044] Figure 5A This is a schematic diagram of the first connected component;

[0045] Figure 5B This is a schematic diagram of an undirected graph;

[0046] Figure 6 A flowchart illustrating a data classification method provided in another embodiment of this application;

[0047] Figure 7 This is a diagram illustrating the partitioning of the data set.

[0048] Figure 8 This is a schematic diagram illustrating the similarity calculation principle involved in the embodiments of this application;

[0049] Figure 9 A flowchart illustrating a data classification method provided in another embodiment of this application;

[0050] Figure 10 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0051] Figure 11 This is a diagram showing the comparison of the effects before and after optimization in Spark-based similarity calculation;

[0052] Figure 12 A diagram showing the effect comparison before and after optimization of connected components.

[0053] Figure 13 A schematic diagram of the structure of a data classification device provided in an embodiment of this application;

[0054] Figure 14 This is a block diagram of a computing device involved in an embodiment of this application. Detailed Implementation

[0055] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0056] It should be understood that, in the embodiments of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0057] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0058] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0059] To facilitate understanding of the embodiments of this application, the relevant concepts involved in the embodiments of this application will be briefly introduced as follows:

[0060] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0061] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0062] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0063] Machine learning can be divided into unsupervised learning and supervised learning, both of which are commonly used in industry. Regardless of whether it's unsupervised or supervised learning, industrial applications often face challenges such as large datasets and insufficient computing power. Currently, distributed systems are used to address these issues.

[0064] The system involved in the embodiments of this application can be a distributed system formed by connecting a client and multiple nodes (any form of computing device in the network, such as a server or user terminal) through network communication.

[0065] In some embodiments, a distributed system is used as an example, such as a blockchain system. See [link / reference] Figure 1 , Figure 1 This is an optional structural diagram of the distributed system 100 provided in this application embodiment applied to a blockchain system. It consists of multiple nodes (any form of computing device in the network, such as servers or user terminals) and clients, forming a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In the distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer.

[0066] In some embodiments, see Figure 1 The functions of each node in the blockchain system shown include:

[0067] 1) Routing: A basic function of nodes used to support communication between nodes.

[0068] In addition to routing capabilities, nodes can also have the following functions:

[0069] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block.

[0070] 3) A blockchain consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks contain the data submitted by the nodes in the blockchain system.

[0071] Figure 2 A schematic diagram illustrating the principle of the clique algorithm for similarity calculation, as shown below. Figure 2 As shown, the main steps include: first, calculating the similarity matrix of the data; then, using the similarity matrix as input for graph construction to calculate the connected components of the graph; and finally, analyzing and applying the algorithm results.

[0072] However, existing clique algorithms based on similarity calculation have high computational complexity and long processing time when calculating connected components.

[0073] To address the aforementioned technical problems, this application determines the similarity between every two data points out of N data points. Based on this similarity, two data points with a similarity greater than a first preset value are identified as a relation pair, resulting in M ​​relation pairs. According to the association and order of these M relation pairs, the N data points are pre-clustered to obtain P first connected components. Then, based on the association between the P first connected components, Q second connected components are obtained. Data within each of the Q second connected components is identified as belonging to the same data type, thus achieving data classification. Therefore, this application first performs pre-clustering during connected component calculation, and then performs connected component calculation based on the pre-clustering, thereby solving the problem of high computational complexity and time consumption caused by an excessive number of nodes and edges.

[0074] Example 1

[0075] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0076] Figure 3 This is a schematic flowchart illustrating a data classification method provided in an embodiment of this application. Figure 3 As shown, the method in this application embodiment includes:

[0077] S301. Obtain N data points to be classified and determine the similarity between every two data points, where N is a positive integer.

[0078] The execution subject of this application embodiment is a device with data classification function, such as a data classification device. In some embodiments, the data classification device is a computing device, such as... Figure 1 In some embodiments, the data classification device described above is a unit with data processing capabilities in a computing device, such as a processor in the computing device. This application embodiment is described using a computing device as an example of the execution subject.

[0079] The embodiments of this application can be used in unsupervised models or deep learning models, for example, in the offline training process or online computation process of the above models.

[0080] This application does not restrict the above data; it can be data from any scenario. For example, taking merchant registration as an example, the above data can be the merchant's registration information.

[0081] Similarity calculation is fundamental to many machine learning algorithms, such as KNN (K-Nearest Neighbor), K-Means clustering, and DBSCAN (Density-based spatial clustering of applications with noise). For a set D of entities (e.g., users or merchants), containing N data points (also called N elements), the similarity between any two elements in set D is calculated.

[0082] In some embodiments, the similarity between pairs of elements is the distance between pairs of elements.

[0083] In some embodiments, when the amount of data is large, Cartesian product can be used for distributed similarity calculation.

[0084] This application does not impose any restrictions on the specific method for calculating the similarity between pairs of elements.

[0085] S302. Based on the similarity between any two data points in the N data points, determine the two data points with a similarity greater than the first preset value as a relation pair, and obtain M relation pairs.

[0086] Step S301 above can determine the similarity between every two data points in N data points. Based on the similarity between two data points, this step can determine which two data points can form a relation pair. Specifically, two data points in N data points with a similarity greater than a first preset value are determined as a relation pair, thereby obtaining M relation pairs, where M is a positive integer.

[0087] For example, the above N data points include data A, data B, and data C. The similarity between data A and data B is 0.8, the similarity between data A and data C is 0.7, and the similarity between data B and data C is 0.4. Assuming a first preset value of 0.6, data A and data B form a relation pair (A, B), data A and data C form a relation pair (A, C), while data B and data C cannot form a relation pair and are separate data nodes.

[0088] It should be noted that the above-mentioned first preset value of 0.6 is an example. The value of the first preset value includes, but is not limited to, 0.6. This application does not limit the specific value of the first preset value, but determines it according to actual needs.

[0089] S303. Based on the association and arrangement order among the M relation pairs, pre-cluster the N data to obtain P first connected components.

[0090] First, we will introduce the relevant concepts of connected components.

[0091] In graph theory, an undirected graph G is defined as a finite set of nonempty vertices V(G) and a finite set of edges E(G), denoted by the following formula (1):

[0092] G=(V(G),E(G)) (1)

[0093] Each element of E(G) is an unordered pair of vertices in V(G), called an edge of G.

[0094] Vertex connectivity: In an undirected graph G, if there is a path from vertex vi to vertex vj, then vertex vi and vertex vj are said to be connected.

[0095] Connected graph: In an undirected graph G, if any two distinct vertices in V(G) are connected (i.e., there is a path between them), then G is called a connected graph.

[0096] Connected components: In graph theory, a maximal connected subgraph of an undirected graph G is called a connected component. In a connected component, any two vertices are connected by a path. Every connected graph has only one connected component, which is the connected graph itself. Disconnected undirected graphs consist of multiple connected components. For example... Figure 4 As shown, graph G consists of three maximal connected subgraphs. Any two vertices in each of the three connected components are mutually connected. There are no paths connecting the connected components to each other.

[0097] The methods for finding connected components of a computational graph involved in the embodiments of this application include, but are not limited to, breadth-first search (BFS) or depth-first search (DFS). Taking the depth-first search algorithm as an example, the algorithm logic for finding connected components of an undirected graph is as follows:

[0098]

[0099] Depth-First Search (DFS) starts from a vertex v in graph G, visits any of its adjacent vertices w1, and then starts from w1, visits any unvisited adjacent vertices, until all adjacent vertices have been visited. Then, it backtracks one step to a previously visited vertex and checks if there are any unvisited adjacent vertices. This process is repeated until all vertices in the undirected graph have been visited. DFS can thus capture all connected components of an undirected graph.

[0100] As mentioned above, current methods for obtaining connected components involve first constructing an undirected graph, then performing a connected component search within that graph to obtain its connected components. For example, the time complexity of a Depth-First Search (DFS) algorithm for obtaining connected components is O(V+E), where V is the number of nodes in the graph and E is the number of edges. When the number of edges is excessive, the overall complexity increases significantly, greatly increasing computation time. To address this technical problem, this application optimizes the process of obtaining connected components by pre-clustering the undirected graph during construction to obtain one or more first connected components. Based on these first connected components, the final second connected components are then obtained.

[0101] When constructing an undirected graph, M relation pairs are input one by one in a permutation order. The permutation order of the M relation pairs can also be understood as the input order of the M relation pairs. During the input process, based on the association relationships and permutation order of the input M relation pairs, the N data points are pre-clustered to obtain P first connected components.

[0102] The pre-clustering in this application involves connecting the input relation pairs with an association relationship according to the order of the relation pairs to form a first connected component. A first connected component can be understood as a cluster.

[0103] For example Figure 5AAs shown, M=16, and the 16 relation pairs are arranged in the following order: (a1, a2), (a2, a3), (a3, a4), (a4, a5), (a4, a6), (a6, a7), (a6, a8), (a8, a9), (a8, a10), (a12, a13), (a13, a14), (a13, a15), (a14, a15), (a11, a10), (a16, a17), (a16, a18). These 16 relation pairs are input into the wireless graph construction model in the order listed. The relation pairs (a1, a2), (a2, a3), (a3, a4), (a4, a5), (a4, a6), (a6, a7), (a6, a8), (a8, a9), and (a8, a10) are related and can be connected to form... Figure 5A The first connected component 1 is shown. When the relation pair (a12, a13) is input, there is no association between the relation pair (a12, a13) and the relation pairs in the first connected component 1. At this time, the construction of the first connected component 1 is completed. The relation pair (a12, a13) is used as the starting node to start the construction of a new first connected component 2. The relation pair (a12, a13) has a specific association with the subsequently input relation pairs (a13, a14), (a13, a15), and (a14, a15) and can be connected to each other to form the first connected component 2. When the relation pair (a11, a10) is input, one element a11 of the relation pair is located in the first connected component 2, and the other element a10 is located in the first connected component 1. In this way, there is a relation pair (a11, a10) between the first connected component 1 and the first connected component 2. When the relation pair (a16, a17) is input, there is no association between this relation pair (a16, a17) and the relation pairs in the first connected component 2. At this time, the construction of the first connected component 2 is completed. The relation pair (a16, a17) is used as the starting node to start the construction of a new first connected component 3. This relation pair (a16, a17) has a specific association with the subsequently input relation pair (a16, a18) and can be connected to each other to form the first connected component 3.

[0104] Furthermore, the process of constructing the first connected component for each of the M relation pairs is the same. The above S303 will be explained below using any one of the M relation pairs as an example.

[0105] In some embodiments, for the i-th relation pair in the order of M relation pairs, if the i-th relation pair contains the same first data as the first connected component of the current pre-cluster, then the i-th relation pair is connected to the first data in the first connected component, where i is a positive integer less than or equal to M; if the i-th relation pair does not contain the same first data as the first connected component of the current pre-cluster, then the data in the i-th relation pair is used as the starting node of the new first connected component to construct a new first connected component; if the two data in the i-th relation pair are located in two different first connected components, then the i-th relation pair is determined as the relation pair between the two different first connected components.

[0106] Continue to refer to Figure 5A In the example shown, assuming the i-th relation pair is (a8, a10) as described above, and the first connected component of the current pre-cluster is the first connected component 1, it is determined whether there is a first data in the i-th relation pair (a8, a10) that is the same as the first connected component 1. After determining that there is a first data a8 in the i-th relation pair (a8, a10) that is the same as the first connected component 1, the relation pair (a8, a10) is connected to the first data a8 in the first connected component 1, that is, the data a10 is connected to the first data a8.

[0107] Assuming the i-th relation pair is (a12, a13) as described above, and the first connected component of the current pre-cluster is the first connected component 1, determine whether there is any first data in the i-th relation pair (a12, a13) that is the same as the first connected component 1. If it is determined that there is no first data in the i-th relation pair (a12, a13) that is the same as the first connected component 1, then the relation pair (a12, a13) is used as the starting node of the new first connected component 2 to construct the new first connected component 2.

[0108] Suppose the i-th relation pair is (a10, a11), where data a10 is located in the first connected component 1 and data a11 is located in the first connected component 2. Then the relation pair (a10, a11) is determined as the relation pair between the first connected component 1 and the first connected component 2.

[0109] Following the method described above, perform the above method on each of the M relation pairs to obtain one or more first connected components.

[0110] The first connected component formed by the above method includes at least two data points.

[0111] In some embodiments, such as Figure 5A As shown, the first connected component 4 also includes a data, for example, the first connected component 4 includes data a19.

[0112] Specifically, based on the similarity between any two data points in the N data points, two data points with a similarity less than or equal to a first preset value are identified as independent nodes, resulting in K independent nodes, where K is a positive integer; each independent node is identified as a first connected component.

[0113] In other words, based on the similarity between pairs of data in N data points, M relation pairs and K independent nodes are obtained. Following the method described above, one or more first connected components can be obtained from the M relation pairs, each containing at least two data points. Additionally, each of the K independent nodes can be designated as a first connected component. Finally, P first connected components are obtained from the N data points, each containing at least one data point.

[0114] S304. Based on the correlation between the first connected components in the P first connected components, obtain the Q second connected components.

[0115] This application pre-clusters N data points to obtain P first connected components, and then, based on the correlation between the P first connected components, obtains Q second connected components. Compared to... Figure 5B As shown, compared to searching for connected components in an undirected graph after its construction, this application calculates connected components during the construction process of the undirected graph, reducing the computational complexity of connected components, saving computation time, and improving computational efficiency.

[0116] The methods for obtaining Q second connected components based on the correlation relationships between the P first connected components in S304 above include, but are not limited to, the following:

[0117] Method 1, S304 above includes: for each of the P first connected components, if there is a relation pair between the first connected component and other first connected components, the first connected component is merged with other first connected components into a second connected component; if there is no relation pair between the first connected component and other first connected components, the first connected component is determined as a second connected component, and the other first connected components are the first connected components other than the first connected component among the P first connected components.

[0118] For example, Figure 5A As shown, there is a specific relationship pair (a11, a10) between the first connected component 1 and the first connected component 2. Therefore, the first connected component 1 and the first connected component 2 can be merged into a second connected component 1. However, there is no relationship pair between the first connected component 3 and both the first connected component 1 and the first connected component 2. Therefore, the first connected component 3 is determined as the second connected component 2.

[0119] Method 2: Treat each of the P first connected components as a node, and the line connecting the relationship between two different first connected components as an edge, and calculate the connected components to obtain Q second connected components.

[0120] For example, each of the P first connected components is treated as a node, and the line connecting the relationship between two different first connected components is treated as an edge. Inputting these into the connected component calculation model yields Q second connected components, where Q is a positive integer less than or equal to P.

[0121] In one example, each of the P first connected components is treated as a node, and the line connecting the relationship between two different first connected components is treated as an edge. Using a generalized first search method or a depth-first search method, Q second connected components are obtained.

[0122] For example Figure 5A As shown, the first connected component 1, the second connected component 2, the first connected component 3, and the second connected component 4 are each treated as a node, and the line connecting the first connected component 1 and the second connected component 2 is treated as an edge. Finally, three second connected components are calculated. That is, this application transforms the computational workload of the original 19 nodes and 17 edges into the computational workload of 4 nodes and 1 edge, significantly reducing the computational cost of the algorithm and thus improving the computational efficiency of connected components.

[0123] S305. Determine that the data in each of the Q second connected components is of the same type.

[0124] As can be seen from the above, the relation pairs in the second connected component are related. Therefore, the data in a second connected component are of the same type. Thus, by calculating the Q second connected components of N data, the N data can be divided into Q types.

[0125] In this embodiment, during connected component calculation, N data points are pre-clustered based on the association relationships and arrangement order between relation pairs to obtain P first connected components. Then, based on the association relationships between the P first connected components, Q second connected components are obtained. Data within each of the Q second connected components is classified as the same type, thus achieving the classification of the N data points. Therefore, this application first performs pre-clustering before calculating connected components, thereby solving the problem of excessive computation time caused by too many nodes and edges, and improving the efficiency of data classification.

[0126] Example 2

[0127] The following section provides a detailed explanation of the similarity between any two data points among the N data points determined in S301 above.

[0128] Figure 6 A flowchart illustrating a data classification method provided in another embodiment of this application is shown below. Figure 6 As shown, S301 above includes:

[0129] S401. Based on the attribute information of each of the N data, divide the N data into at least one first data group.

[0130] S402. Send the data of each first data group to different computing devices for similarity calculation, and obtain the similarity between every two data in each first data group from different computing devices.

[0131] For a set of business data D, the computational complexity for calculating the similarity (e.g., distance) between any two data points in D is O(N) = N. 2 The number of data points in set D is |D| = N. Therefore, as the set N grows, the computational complexity increases exponentially.

[0132] In real-world business scenarios, the number of data sets often reaches millions or even tens of millions. Therefore, in actual algorithm implementation, distributed frameworks such as Spark are used to first group the data based on its attribute information. For example, based on the attribute information of each of the N data sets, the N data sets are divided into at least one first data set.

[0133] In some embodiments, the data includes merchant registration information, which includes the merchant's name. In this case, the data attribute information includes a naming pattern, which can be understood as the naming style of the merchant's name. In this case, in step S401 above, grouping the N data points according to the data attribute information can involve performing word segmentation and part-of-speech tagging on each of the N data points to obtain the naming pattern for each data point; and grouping data points with the same naming pattern into a first data group.

[0134] In this context, the data in each first data group are not similar to each other.

[0135] The data from each first data group is distributed to different computing devices for similarity calculation. For example, the data from different first data groups are distributed to... Figure 1 Similarity is calculated on the different nodes shown.

[0136] In one example, such as Figure 7As shown, N data points are divided into three first data groups, denoted as first data group 1, first data group 2, and first data group 3. First data group 1 is sent to computing device 1, which calculates the similarity between every two data points within first data group 1. First data group 2 is sent to computing device 2, which calculates the similarity between every two data points within first data group 2. First data group 3 is sent to computing device 3, which calculates the similarity between every two data points within first data group 3.

[0137] This application can greatly improve the computational efficiency of similarity through grouped distributed computing, but it may suffer from data skew.

[0138] Data skew is a common problem in distributed data processing. In distributed processing, under normal circumstances, data is distributed to each machine node (e.g., ...). Figure 1 Ideally, the amount of data processed on each node in a distributed system should be roughly the same, so that the processing time for each node is not significantly different, and distributed processing can achieve optimal performance. However, in practice, various reasons often lead to uneven distribution of data to each machine node for processing. When some nodes receive significantly more data than other nodes, this situation is called data skew.

[0139] When data skew occurs in distributed processing, the completion time of the entire task will be determined by the node that processes the most data. The node processing the most data has a large computational load and a long computation time, which increases the overall task execution time, while other nodes have already finished processing the distributed data, resulting in idle and unusable resources.

[0140] In the process of calculating similarity, distributed parallel computing through grouping can effectively improve the overall computational efficiency. However, in reality, some groups often have more element data than others, leading to data skew in the processing. This application addresses this data skew by proposing a distributed optimization method based on a distribution threshold, which can effectively avoid data skew and ensure the overall execution efficiency of the task.

[0141] In some embodiments, S402 above includes the following S402-1 to S402-3:

[0142] S402-1. For each of the first data groups in at least one first data group, if the amount of data in the first data group is greater than a second preset value, then according to the preset data volume threshold in the data block, the data in the first data group is divided into F data blocks, where F is a positive integer.

[0143] S402-2. For every two data blocks in the F data blocks, send the two data blocks to the first computing device so that the first computing device can calculate the similarity between every two data blocks.

[0144] S402-3. If the amount of data in the first data group is less than or equal to the second preset value, then the data in the first data group is sent to the second computing device for similarity calculation.

[0145] The process of calculating the similarity of data in each first data group is the same in this application. Taking one first data group as an example.

[0146] The decision to further divide the first data group is based on its data volume. Specifically, if the data volume in the first data group exceeds a preset data block volume threshold S, the data in the first data group is divided into F data blocks. Each pair of data blocks is then sent to different first computing devices to calculate the similarity between any two data points in the two data blocks. If the data volume in the first data group is less than or equal to the data volume threshold S, the data in the first data group is sent to a second computing device for similarity calculation.

[0147] In one possible implementation, S402-1 above, which divides the data in the first data group into F data blocks according to a preset data volume threshold, includes the following steps S402-11 and S402-12:

[0148] S402-11. Add 1 to the result of dividing the data quantity in the first data group by the data quantity threshold, and use it as the number of data blocks corresponding to the first data group;

[0149] S402-12. Based on the amount of data in the first data group and the number of data blocks corresponding to the first data group, divide the data in the first data group into F data blocks on an average basis.

[0150] For example, the number of data blocks F corresponding to the first data group is calculated according to the following formula (2):

[0151] F=N1 / / S+1 (2)

[0152] Where N1 is the amount of data in the first data group, S is the preset data volume threshold in the data block, and " / / " means integer division.

[0153] Based on the amount of data N1 in the first data group and the number of data blocks F corresponding to the first data group, the data in the first data group is divided into F data blocks on an average basis. For example, Figure 8 As shown, the first data group 3 is divided into 3 data blocks.

[0154] At this point, in S402-2 above, for every two data blocks out of the F data blocks, two data blocks are sent to the first computing device, including S402-21 and S402-22:

[0155] S402-21. Based on the different combinations of pairs of data blocks in F data blocks, obtain R combinations of pairs of data blocks, where the R combinations of pairs of data blocks include combinations of two identical data blocks in the F data blocks and combinations of two different data blocks, and R is a positive integer.

[0156] S402-22. For each pair of data blocks in the R pairs of data blocks, send the two data blocks in the pair of data blocks to the first computing device.

[0157] For example, suppose the above F data blocks include data block 1, data block 2 and data block 3, and the R pairs of data blocks include: (data block 1, data block 1), (data block 1, data block 2), (data block 1, data block 3), (data block 2, data block 3), (data block 2, data block 2), (data block 3, data block 3). (Data block 1, Data block 1) is sent to the first computing device 1, which calculates the similarity between each pair of data in data block 1; (Data block 1, Data block 2) is sent to the first computing device 2, which calculates the similarity between each pair of data in data block 1 and data block 2; (Data block 1, Data block 3) is sent to the first computing device 3, which calculates the similarity between each pair of data in data block 1 and data block 3; (Data block 2, Data block 3) is sent to the first computing device 4, which calculates the similarity between each pair of data in data block 2 and data block 3; (Data block 2, Data block 2) is sent to the first computing device 5, which calculates the similarity between each pair of data in data block 2; (Data block 3, Data block 3) is sent to the first computing device 6, which calculates the similarity between each pair of data in data block 3.

[0158] In one possible implementation, S402-21 includes: obtaining an FXF data block matrix based on different combinations of pairs of data blocks from the F data blocks; determining R combinations of pairs of data blocks located in the upper triangular part of the FXF data block matrix; and sending two data blocks from each of the R combinations of pairs of data blocks to the first computing device.

[0159] For example Figure 8 As shown, assuming F is 3, the first data group is divided into 3 data blocks. Different combinations of each pair of data blocks result in a 3x3 data block matrix as shown. Figure 8 As shown, the pairwise data blocks located in the upper triangle of this 3x3 data block matrix are combined into six pairs: (data block 1, data block 1), (data block 1, data block 2), (data block 1, data block 3), (data block 2, data block 3), (data block 2, data block 2), and (data block 3, data block 3). The two data blocks in each pairwise data block combination are distributed to different first computing devices for similarity calculation. For example, (data block 1, data block 1) is sent to first computing device 1, which calculates the similarity between the pairwise data in data block 1 and data block 2; (data block 1, data block 2) is sent to first computing device 2, which calculates the similarity between the pairwise data in data block 1 and data block 2. A computing device 2 calculates the similarity between pairwise data in data block 1 and data block 2; sends (data block 1, data block 3) to a first computing device 3, so that the first computing device 3 calculates the similarity between pairwise data in data block 1 and data block 3; sends (data block 2, data block 3) to a first computing device 4, so that the first computing device 4 calculates the similarity between pairwise data in data block 2 and data block 3; sends (data block 2, data block 2) to a first computing device 5, so that the first computing device 5 calculates the similarity between pairwise data in data block 2; sends (data block 3, data block 3) to a first computing device 6, so that the first computing device 6 calculates the similarity between pairwise data in data block 3.

[0160] After the similarity calculation is completed according to the above method, the steps S302 to S305 above are executed to calculate the connected components and realize the classification of data. The specific process of calculating the connected components is described in S302 to S304 above, and will not be repeated here.

[0161] Example 3

[0162] Figure 9 A flowchart illustrating a data classification method provided in another embodiment of this application is shown below. Figure 9 As shown, the method includes:

[0163] S701. Obtain N data points to be classified.

[0164] Figure 10 This is a schematic diagram of an application scenario of an embodiment of this application. This embodiment of the application can be applied to a merchant name similarity model in a certain center. By optimizing the offline model training scheme, the offline training performance can be greatly improved, and the problem of big data computing and computing power can be solved.

[0165] like Figure 10As shown, when a merchant conducts a transaction through this payment platform, they first need to complete registration, i.e., the merchant uploads registration information to the payment platform. This registration information includes basic information such as the merchant's name. The payment platform sends the merchant's registration information to the risk control center, which performs policy interception, such as determining whether the merchant is a legitimate merchant and whether the merchant is already registered. For example, the merchant's registration information is matched against historical registered merchants. If the merchant's registration information exists in historical groups, it means that the merchant is already registered and will not be registered again. If the merchant is not found in historical groups, the merchant is determined to be a new merchant, and the new merchant's registration information is sent to the backend computing device. Since multiple merchants register information with the payment platform, the backend computing device can obtain the registration information of multiple merchants, for example, N merchants. The backend computing device executes the method of this embodiment to complete the merchant classification.

[0166] As can be seen from the above, in Figure 10 In the application scenario shown, the registration information of N merchants can be understood as N data to be categorized.

[0167] S702. Perform word segmentation and part-of-speech tagging on each of the N data points to obtain the naming pattern for each data point.

[0168] S703. Group the data with the same naming pattern among the N data into a first data group.

[0169] As mentioned above, merchant registration information includes merchant names. Different types of merchant names may have different naming patterns. Based on this, the computing device performs word segmentation and part-of-speech tagging on each of the N data (e.g., merchant names) to obtain the naming pattern for each data (merchant name). Data with the same naming pattern are grouped into a first data group, thus obtaining one or more first data groups.

[0170] S704. For each of the at least one first data group, determine whether the amount of data in the first data group is greater than a second preset value. If the amount of data in the first data group is greater than the second preset value, then execute S705 to S708. If the amount of data in the first data group is less than the second preset value, then execute S709.

[0171] S705. Based on the preset data volume threshold within the data block, divide the data in the first data group into F data blocks.

[0172] S706. Based on the different combinations of pairs of data blocks in the F data blocks, obtain the FXF data block matrix.

[0173] S707. Combine the pairs of data blocks located in the upper triangle of the FXF data block matrix to determine R pairs of data blocks.

[0174] S708. For each pair of data blocks in the R pairs of data blocks, send the two data blocks in the pair of data blocks to the first computing device so that the first computing device can calculate the similarity between each pair of data in the two data blocks.

[0175] For each of at least one first data group, if the amount of data in the first data group is greater than a second preset value, the data in the first data group is divided into F data blocks according to a preset data block size threshold S. For example, the data in the first data group is divided into F data blocks on average. Based on different combinations of pairs of data blocks in the F data blocks, an FXF data block matrix is ​​obtained. The pairs of data blocks located in the upper triangle of the FXF data block matrix are combined to determine R pairs of data blocks. For each pair of data blocks in the R pairs of data blocks, the two data blocks in the pair are sent to a first computing device so that the first computing device can calculate the similarity between every two data points in the two data blocks. This solves the data skew problem, improves the calculation efficiency of data similarity, balances computing resources, and reduces waste of computing resources.

[0176] S709. If the amount of data in the first data group is less than or equal to the second preset value, then the data in the first data group is sent to the second computing device for similarity calculation.

[0177] If the amount of data in the first data group is less than the second preset value, there is no need to divide the data in the first data group. Instead, the amount of data in the first data group is directly sent to the corresponding second computing device for similarity calculation.

[0178] Based on S705 to S709 above, after calculating the similarity between every two data points in N data points, the following steps are performed to calculate connected components based on the similarity.

[0179] S710. Based on the similarity between any two data points in the N data points, determine the two data points with a similarity greater than a first preset value as a relation pair, and obtain M relation pairs.

[0180] S711. For the i-th relation pair in the arrangement of M relation pairs, determine whether there is a first data in the i-th relation pair that is the same as the first connected component of the current pre-cluster. If there is a first data in the i-th relation pair that is the same as the first connected component of the current pre-cluster, execute S712. If there is no first data in the i-th relation pair that is the same as the first connected component of the current pre-cluster, execute S713. Here, i is a positive integer less than or equal to M.

[0181] S712. Connect the i-th relation pair with the first data in the first connected component.

[0182] S713. Construct a new first connected component by using the data in the i-th relation pair as the starting node of the new first connected component.

[0183] S714 determines whether i is equal to M. If i is not equal to M, then i = i + 1, and continue to execute steps S711 to S713. If i is equal to M, then execute S715.

[0184] Through the steps S710 to S714 above, P first connected components are obtained based on the similarity between pairs of data.

[0185] S715. Treat each of the P first connected components as a node, and the line connecting the relationship between two different first connected components as an edge, and perform connected component calculation to obtain Q second connected components.

[0186] For example, each of the P first connected components is treated as a node, and the line connecting the relationship between two different first connected components is treated as an edge. These are input into the connected component calculation model, which outputs Q second connected components.

[0187] S716. Determine that the data in each of the Q second connected components is of the same type.

[0188] In some embodiments, depending on the size of the data within the first data group, the method for calculating connected components can be divided into connected component calculation based on similarity calculation and connected component calculation based on similarity results.

[0189] 1. Connected component calculation based on similarity calculation

[0190] For similarity calculations within the first data group where the data volume is relatively small, single-node calculation can be used. An exemplary connected component optimization calculation logic is shown below:

[0191]

[0192] Based on the logic code above, it can be seen that for similarity calculations within the first data group where the amount of data is relatively small, a single-node calculation can be used.

[0193] 2. Calculation of connected components based on similarity results

[0194] For similarity calculations within the first data group that involve a large amount of data, a Cartesian product can be used for distributed similarity calculation. Based on this, the calculated similarity is used as input to an algorithm for optimizing connected components. An example of the connected component optimization logic is shown below:

[0195]

[0196] Based on the above logic code, it can be seen that for similarity calculations within the first data group with a large amount of data, Cartesian product can be used for distributed similarity calculation.

[0197] The following are the beneficial effects of the technical solution of this application.

[0198] Figure 11 This is a diagram comparing the effects of optimization before and after in Spark-based similarity calculation. Figure 11 As can be seen, the technical solution of this application embodiment is an optimized technical solution. By adopting the technical solution of this application, the time can be greatly saved when calculating similarity. For example, for a data volume of 300,000, it takes 180 minutes to calculate similarity using the original technology, while it takes only 20 minutes using the technical solution of this application.

[0199] Figure 12 To illustrate the effect comparison before and after optimization of connected components, the diagram is provided by... Figure 12 It is understood that the technical solution of this application embodiment is an optimized technical solution. By adopting the technical solution of this application, the number of nodes and edges can be reduced when calculating connected components, thereby saving the calculation time of connected components. For example, for the first set of experiments, the existing technology needs to process 23,408 nodes and 7,560,995 edges when calculating connected components, taking 17.3 seconds (s), while using the technical solution of this application to calculate connected components requires processing 9,752 nodes and 548 edges, taking 32.1 milliseconds (ms).

[0200] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0201] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0202] Example 4

[0203] The above text combined Figures 3 to 12 The method embodiments of this application are described in detail below, in conjunction with... Figures 13 to 14 The following describes in detail the device embodiments of this application.

[0204] Figure 13 This is a schematic diagram of a data classification device provided in an embodiment of this application. The device can be a computing device or a component of a computing device (e.g., an integrated circuit, a chip, etc.). Figure 13 As shown, the data classification device 10 may include:

[0205] Acquisition unit 11 is used to acquire N data points to be classified;

[0206] The similarity determination unit 12 is used to determine the similarity between every two data points in the N data points;

[0207] The relationship pair determination unit 13 is used to determine two data pairs with a similarity greater than a first preset value as a relationship pair based on the similarity between every two data pairs in the N data pairs, thereby obtaining M relationship pairs;

[0208] The pre-clustering unit 14 is used to pre-cluster the N data according to the association relationship and arrangement order between the M relation pairs to obtain P first connected components;

[0209] The connected component determination unit 15 is used to obtain Q second connected components based on the correlation relationship between each of the P first connected components;

[0210] Classification unit 16 is used to determine the data in each of the Q second connected components as data of the same type;

[0211] Wherein, N, M, P, and Q are all positive integers, and Q is less than or equal to P.

[0212] In some embodiments, the pre-clustering unit 14 is specifically used for the i-th relation pair in the arrangement order of the M relation pairs. If the i-th relation pair contains the same first data as the first connected component of the current pre-cluster, then the i-th relation pair is connected to the first data in the first connected component. If the i-th relation pair does not contain the same first data as the first connected component of the current pre-cluster, then the data in the i-th relation pair is used as the starting node of the new first connected component to construct the new first connected component, where i is a positive integer less than or equal to M.

[0213] In some embodiments, the connected component determination unit 15 is further configured to determine two data points with a similarity less than or equal to the first preset value as independent nodes based on the similarity between every two data points in the N data points, thereby obtaining K independent nodes, where K is a positive integer; and to determine each of the independent nodes as a first connected component.

[0214] In some embodiments, the connected component determination unit 15 is specifically configured to, for each of the P first connected components, if there is a relation pair between the first connected component and other first connected components, merge the first connected component and the other first connected components into a second connected component; if there is no relation pair between the first connected component and other first connected components, determine the first connected component as a second connected component, wherein the other first connected components are the first connected components other than the first connected component among the P first connected components.

[0215] In some embodiments, the connected component determination unit 15 is specifically used to treat each of the P first connected components as a node, and the line connecting the relationship pair between two different first connected components as an edge, to perform connected component calculation and obtain the Q second connected components.

[0216] In some embodiments, the connected component determination unit 15 is specifically used to treat each of the P first connected components as a node, the connection between two different first connected components as an edge, and use a generalized first search method or a depth-first search method to obtain the Q second connected components.

[0217] In some embodiments, the similarity determination unit 12 is specifically configured to divide the N data into at least one first data group according to the attribute information of each data in the N data; send the data of each first data group to different computing devices for similarity calculation, and obtain the similarity between every two data in each first data group from the different computing devices.

[0218] In some embodiments, the similarity determination unit 12 is specifically configured to, for each of the at least one first data group, if the amount of data in the first data group is greater than a second preset value, divide the data in the first data group into F data blocks according to a preset data block amount threshold, where F is a positive integer; and for every two data blocks in the F data blocks, send the two data blocks to a first computing device so that the first computing device can calculate the similarity between every two data blocks in the two data blocks.

[0219] In some embodiments, the similarity determination unit 12 is specifically used to add 1 to the integer division result of the data amount in the first data group and the data amount threshold, and use it as the number of data blocks corresponding to the first data group; according to the data amount in the first data group and the number of data blocks corresponding to the first data group, the data in the first data group is divided into F data blocks on an average basis.

[0220] In some embodiments, the similarity determination unit 12 is specifically configured to obtain R combinations of pairwise data blocks based on different combinations of pairwise data blocks in the F data blocks, wherein the R combinations of pairwise data blocks include combinations of two identical data blocks and combinations of two different data blocks in the F data blocks, and R is a positive integer; for each pairwise data block combination in the R combinations of pairwise data blocks, the two data blocks in the pairwise data block combination are sent to the first computing device.

[0221] In some embodiments, the similarity determination unit 12 is specifically used to obtain an FXF data block matrix based on different combinations of pairs of data blocks in the F data blocks; and to determine the combinations of pairs of data blocks located in the upper triangle of the FXF data block matrix as the R pairs of data blocks.

[0222] In some embodiments, the similarity determination unit 12 is further configured to send the data in the first data group to a second computing device for similarity calculation if the amount of data in the first data group is less than or equal to the second preset value.

[0223] In some embodiments, if the attribute information of the data includes a naming pattern, the acquisition unit 11 is specifically used to perform word segmentation and part-of-speech tagging on each of the N data to obtain the naming pattern of each data; and to divide the data with the same naming pattern among the N data into a first data group.

[0224] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 13The apparatus shown can perform the embodiments of the above-described methods, and the foregoing and other operations and / or functions of each module in the apparatus are respectively for implementing the method embodiments corresponding to the encoder. For the sake of brevity, they will not be described in detail here.

[0225] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0226] Example 5

[0227] Figure 14 This is a block diagram of a computing device involved in an embodiment of this application. The computing device is used to execute the data classification method described in the above embodiments. For details, please refer to the description in the above method embodiments.

[0228] Figure 14 The computing device 200 shown includes a memory 201, a processor 202, and a communication interface 203. The memory 201, processor 202, and communication interface 203 are communicatively connected to each other. For example, the memory 201, processor 202, and communication interface 203 can be connected via a network. Alternatively, the computing device 200 may also include a bus 204. The memory 201, processor 202, and communication interface 203 are communicatively connected to each other via the bus 204. Figure 14 The computing device 200 is a computer that uses a memory 201, a processor 202, and a communication interface 203 to communicate with each other via a bus 204.

[0229] The memory 201 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 201 can store programs, and when the program stored in the memory 201 is executed by the processor 202, the processor 202 and the communication interface 203 are used to execute the above methods.

[0230] The processor 202 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits.

[0231] The processor 202 can also be an integrated circuit chip with signal processing capabilities. In implementation, the method of this application can be completed through integrated logic circuits in the hardware of the processor 202 or through software instructions. The processor 202 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The software module can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 201, and the processor 202 reads information from memory 201 and, in conjunction with its hardware, completes the method of the embodiments of this application.

[0232] Communication interface 203 uses transceiver modules, such as, but not limited to, transceivers, to enable communication between computing device 200 and other devices or communication networks. For example, a dataset can be obtained through communication interface 203.

[0233] When the computing device 200 includes a bus 204, the bus 204 may include a path for transmitting information between various components of the computing device 200 (e.g., memory 201, processor 202, communication interface 203).

[0234] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0235] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0236] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0237] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0239] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0240] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data classification method, characterized in that, include: Obtain N data points to be classified, and divide the N data points into at least one first data group according to the attribute information of each data point. The data in each first data group is not similar to each other. The data includes the merchant's registration information. For each of the at least one first data group, if the amount of data in the first data group is greater than a second preset value, then according to the data amount threshold, the data in the first data group is divided into F data blocks, and the amount of data included in the data block does not exceed the data amount threshold. For every two data blocks in the F data blocks, the two data blocks are sent to the first computing device so that the first computing device can calculate the similarity between every two data blocks. If the amount of data in the first data group is less than or equal to the second preset value, then the first data group is sent to the second computing device for similarity calculation. Based on the similarity between any two data points in the N data points, two data points with a similarity greater than a first preset value are identified as a relation pair, resulting in M ​​relation pairs; Based on the order of the M relation pairs, the relation pairs with association are connected together to obtain P first connected components; Each of the P first connected components is treated as a node, and the line connecting the relationship between two different first connected components is treated as an edge. Connectivity component calculation is performed to obtain Q second connected components. The data within each of the Q second connected components are determined to be of the same type. Where N, F, M, P, and Q are all positive integers, and Q is less than or equal to P.

2. The method according to claim 1, characterized in that, The step of connecting the relation pairs with association relationships together according to the order of the M relation pairs to obtain P first connected components includes: For the i-th relation pair in the order of the M relation pairs, if the i-th relation pair contains the same first data as the first connected component of the current pre-cluster, then the i-th relation pair is connected to the first data in the first connected component, where i is a positive integer less than or equal to M. If there is no first data in the i-th relation pair that is the same as the first connected component of the current pre-cluster, then the data in the i-th relation pair is used as the starting node of the new first connected component to construct the new first connected component; If the two data in the i-th relation pair are located in two different first connected components, then the i-th relation pair is determined as the relation pair between the two different first connected components.

3. The method according to claim 2, characterized in that, The method further includes: Based on the similarity between every two data points in the N data points, two data points with a similarity less than or equal to the first preset value are determined as independent nodes, resulting in K independent nodes, where K is a positive integer. Each of the independent nodes is identified as a first connected component.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: For each of the P first connected components, if there is a relation pair between the first connected component and other first connected components, the first connected component is merged with the other first connected components into a second connected component. If there is no relation pair between the first connected component and other first connected components, the first connected component is determined as a second connected component, and the other first connected components are the first connected components other than the first connected component among the P first connected components.

5. The method according to claim 1, characterized in that, The step of treating each of the P first connected components as a node and the line connecting the relationships between two different first connected components as an edge, and calculating the connected components to obtain the Q second connected components, includes: Treat each of the P first connected components as a node, and the line connecting the relationship between two different first connected components as an edge. Use a generalized first search method or a depth-first search method to obtain the Q second connected components.

6. The method according to claim 1, characterized in that, The step of dividing the data within the first data group into F data blocks based on the data volume threshold includes: The result of dividing the amount of data in the first data group by the data amount threshold is incremented by 1, and this result is used as the number of data blocks corresponding to the first data group. Based on the amount of data in the first data group and the number of data blocks corresponding to the first data group, the data in the first data group is divided into F data blocks on an average basis.

7. The method according to claim 1, characterized in that, For every two data blocks out of the F data blocks, sending the two data blocks to the first computing device includes: Based on the different combinations of pairs of data blocks in the F data blocks, R combinations of pairs of data blocks are obtained. The R combinations of pairs of data blocks include combinations of two identical data blocks in the F data blocks and combinations of two different data blocks, where R is a positive integer. For each of the R pairs of data blocks, send the two data blocks in the pair of data blocks to the first computing device.

8. The method according to claim 7, characterized in that, The step of obtaining R combinations of pairwise data blocks based on different combinations of pairwise data blocks from the F data blocks includes: Based on the different combinations of each pair of data blocks in the F data blocks, an F×F data block matrix is ​​obtained; The pairs of data blocks located in the upper triangle of the F×F data block matrix are combined to determine the R pairs of data blocks.

9. The method according to claim 1, characterized in that, If the attribute information of the data includes a naming pattern, then dividing the N data into at least one first data group based on the attribute information of each of the N data includes: Perform word segmentation and part-of-speech tagging on each of the N data points to obtain the naming pattern for each data point; Group the data with the same naming pattern among the N data into a first data group.

10. A data classification device, characterized in that, include: The acquisition unit is used to acquire N data items to be classified, including the merchant's registration information; A similarity determination unit is used to divide the N data into at least one first data group based on the attribute information of each data in the N data; For each of the at least one first data group, if the amount of data in the first data group is greater than a second preset value, then according to the data amount threshold, the data in the first data group is divided into F data blocks, and the amount of data included in the data block does not exceed the data amount threshold; for every two data blocks in the F data blocks, the two data blocks are sent to a first computing device so that the first computing device can calculate the similarity between every two data in the two data blocks; If the amount of data in the first data group is less than or equal to the second preset value, then the first data group is sent to the second computing device for similarity calculation. The relationship pair determination unit is used to determine two data pairs with a similarity greater than a first preset value as a relationship pair based on the similarity between every two data pairs in the N data pairs, thereby obtaining M relationship pairs; The pre-clustering unit is used to connect the relation pairs with the relationship according to the arrangement order of the M relation pairs to obtain P first connected components; The connected component determination unit is used to treat each of the P first connected components as a node, and the line connecting the relationship between two different first connected components as an edge, to calculate the connected components and obtain Q second connected components. The classification unit is used to determine the data in each of the Q second connected components as data of the same type. Where N, F, M, P, and Q are all positive integers, and Q is less than or equal to P.

11. A computing device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 9.

Images