Black-grey production gang identification method, device and equipment and storage medium

Abstract

The invention discloses a black-grey production gang identification method and device, equipment and a storage medium. The black-grey production gang identification method comprises the following steps: acquiring business data in a black-grey production scene; preprocessing the service data to obtain a plurality of cache data blocks and a plurality of parallel partition data sets; calculating a similarity distance between the business data in each partition data set and the business data in each cache data block; determining a temporary cluster based on a preset neighborhood radius threshold value, a preset neighbor number threshold value and the similarity distance; re-partitioning the temporary clustering clusters, and gradually reducing the number of partitions until all the temporary clustering clusters with reachable density are merged into the same partition to obtain a final clustering cluster; and identifying the target object corresponding to each piece of business data in the same final cluster as a member in the same black-grey production gang. According to the method, large-scale black-grey production business data can be efficiently processed, black-grey production gangs can be accurately identified, the business safety level can be improved, and black-grey production illegal activities can be attacked.

Description

Technical Field

[0001] The present invention relates to the field of business security, and in particular to a method, device, equipment and storage medium for identifying black and gray production gangs. Background Art

[0002] In the field of business security, gang identification technology plays a key role in identifying black and gray industries. Through gang identification technology, black and gray industries can be detected and analyzed, helping business security personnel to better identify black and gray industry gangs, which helps to improve business security levels and crack down on illegal activities of black and gray industries, and provide strong support for risk management and prevention.

[0003] The existing black and gray gang identification technology is usually based on the clustering of business data, and the identification of black and gray gangs is achieved through data analysis to mine the behavior patterns and correlations between gangs and key roles within gangs. However, as the methods of black and gray gangs become increasingly complex and covert, the amount of data that needs to be analyzed and mined and the computational complexity are gradually increasing. Existing data clustering usually consumes a lot of memory resources, computing resources and time resources, and has low processing efficiency and insufficient accuracy, making it difficult to apply to the identification of black and gray gangs. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for identifying black and gray production gangs, aiming to solve the technical problem of low efficiency in the existing black and gray production gang identification and processing.

[0005] A first aspect of the present invention provides a method for identifying a black and gray production group, the method comprising:

[0006] Obtain business data in black and gray industry scenarios;

[0007] Preprocessing the business data to obtain multiple cache data blocks and multiple parallel partition data sets;

[0008] Calculating the similarity distance between the service data in each partition data set and the service data in each cache data block;

[0009] Determine a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and the similarity distance;

[0010] Repartitioning the temporary clusters, and gradually reducing the number of partitions until all the temporary clusters with reachable densities are merged into the same partition to obtain final clusters;

[0011] The target objects corresponding to the business data in the same final cluster are identified as members of the same black and gray production gang.

[0012] Optionally, in a first implementation of the first aspect of the present invention, preprocessing the service data to obtain multiple cache data blocks and multiple parallel partition data sets includes:

[0013] Initializing the business data and aggregating the business data into a business data set;

[0014] Distributed storage is performed on the business data set to obtain a plurality of partitioned data sets;

[0015] The business data set is divided into cache blocks to obtain a plurality of cache data blocks.

[0016] Optionally, in a second implementation of the first aspect of the present invention, the calculating the similarity distance between the business data in each partitioned data set and the business data in each cache data block includes:

[0017] Broadcasting each of the cache data blocks so that the partitions where each of the partition data sets is located receive the service data in each of the cache data blocks;

[0018] The similarity distance between all the business data in each partition data set and all the business data in each cache data block is calculated based on a preset similarity distance function.

[0019] Optionally, in a third implementation of the first aspect of the present invention, each of the business data includes a plurality of feature data with different weights;

[0020] The calculating the similarity distance between all the business data in each partition data set and all the business data in each cache data block based on a preset similarity distance function includes:

[0021] The sum of the weights of the same feature data between the business data is calculated, and the sum of the weights is used as the similarity distance.

[0022] Optionally, in a fourth implementation of the first aspect of the present invention, determining a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and the similarity distance includes:

[0023] Aggregating a plurality of the similarity distances to obtain a similarity distance data set;

[0024] Based on the preset neighborhood radius threshold and the similarity distance data set, the business data whose similarity distance with each business data is not greater than the preset neighborhood radius threshold is used as neighboring business data;

[0025] The business data whose number of neighbor business data is not less than the preset neighbor number threshold is used as core business data, and the core business data and the corresponding neighbor business data are used as a temporary cluster corresponding to the core business data.

[0026] Optionally, in a fifth implementation of the first aspect of the present invention, repartitioning the temporary clusters and gradually reducing the number of partitions until all the temporary clusters with reachable densities are merged into the same partition to obtain the final clusters, includes:

[0027] Aggregating the plurality of temporary clusters to obtain a temporary cluster data set;

[0028] Distribute and store the temporary clustering data set into a plurality of partitions, so that each partition includes at least one temporary cluster in the temporary clustering data set;

[0029] Merging the temporary clusters with density reachable in each of the partitions to obtain intermediate clusters in each partition;

[0030] When the number of partitions including the intermediate cluster is not 1, merging the intermediate clusters in at least two partitions into the same partition; or

[0031] Aggregating the intermediate clusters in each of the partitions to obtain an intermediate cluster data set, and distributing and storing the intermediate cluster data set in a smaller number of partitions;

[0032] Merging the temporary clusters that are density-reachable in each of the new partitions to update the intermediate clusters in each partition;

[0033] When the number of partitions including the intermediate cluster is 1, the intermediate cluster is used as the final cluster.

[0034] Optionally, in a sixth implementation of the first aspect of the present invention, the preprocessing of the service data to obtain a plurality of cache data blocks and a plurality of parallel partition data sets further includes:

[0035] Assigning a unique identifier to each of the business data;

[0036] The step of distributing and storing the temporary cluster data set into a plurality of partitions comprises:

[0037] Distribute and store the temporary clusters of the business data having at least one identical unique identifier in the same partition;

[0038] The distributing and storing the intermediate clustering data set into a smaller number of partitions includes:

[0039] The intermediate clusters of the business data having at least one identical unique identifier are distributed and stored in the same partition.

[0040] The second aspect of the present invention further provides a device for identifying illegal and gray production groups, the device comprising:

[0041] The acquisition module is used to obtain business data in the black and gray industry scenarios;

[0042] A preprocessing module, used for preprocessing the business data to obtain multiple cache data blocks and multiple parallel partition data sets;

[0043] A calculation module, used for calculating the similarity distance between the business data in each partition data set and the business data in each cache data block;

[0044] A first clustering module, configured to determine a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and the similarity distance;

[0045] A second clustering module is used to repartition the temporary clusters and gradually reduce the number of partitions until all the temporary clusters with reachable densities are merged into the same partition to obtain final clusters;

[0046] The identification module is used to identify the target objects corresponding to the business data in the same final cluster as members of the same black and gray production gang.

[0047] The third aspect of the present invention also provides a computer device, which includes: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the method for identifying black and gray production gangs as described above.

[0048] A fourth aspect of the present invention further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the method for identifying black and gray production gangs as described above is implemented.

[0049] The embodiment of the present invention provides a method, device, equipment and storage medium for identifying black and gray production gangs. First, business data in the black and gray production scenario is obtained; then the business data is preprocessed to obtain multiple cache data blocks and multiple parallel partition data sets; then the similarity distance between the business data in each partition data set and the business data in each cache data block is calculated; based on the preset neighborhood radius threshold, the preset neighbor number threshold and the similarity distance, a temporary clustering cluster is determined; finally, the temporary clustering cluster is repartitioned, and the number of partitions is gradually reduced until all density-reachable temporary clustering clusters are merged into the same partition to obtain the final clustering cluster; the target object corresponding to each business data in the same final clustering cluster can be identified as a member of the same black and gray production gang. Based on the black and gray production gang identification method, the complexity of black and gray production data processing and the memory consumption of computing nodes are reduced by distributed computing, and large-scale black and gray production business data can be processed in parallel and efficiently, and black and gray production gangs can be identified more accurately and reliably, which is helpful to improve the business security level and combat illegal black and gray production activities, and provide strong support for risk management and security prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of a flow chart of an embodiment of a method for identifying black and gray production groups in an embodiment of the present invention;

[0051] Figure 2 for Figure 1 A schematic diagram of a flow chart of an embodiment of Embodiment 102;

[0052] Figure 3 for Figure 1 A schematic diagram of a flow chart of an embodiment of Embodiment 103;

[0053] Figure 4 for Figure 1 A schematic diagram of a flow chart of an embodiment of Embodiment 104;

[0054] Figure 5 For Spark-based implementation Figure 3 and Figure 4 A schematic diagram of the architecture of an embodiment;

[0055] Figure 6 for Figure 1 A schematic diagram of a flow chart of an embodiment of Embodiment 105;

[0056] Figure 7 For Spark-based implementation Figure 6 A schematic diagram of the architecture of an embodiment;

[0057] Figure 8 This is a schematic diagram of functional modules of an embodiment of a device for identifying black and gray production groups in an embodiment of the present invention;

[0058] Fig. 9 The figure is a functional module diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] For ease of understanding, the specific process of the method for identifying black and gray production groups in the embodiment of the present invention is described below. Figure 1 , Figure 1 This is a flow chart of a first embodiment of a method for identifying a black and gray production group in an embodiment of the present invention. In this embodiment, the method for identifying a black and gray production group includes:

[0061] 101. Obtain business data in black and gray industry scenarios;

[0062] In this embodiment, black and gray industries refer to a type of illegal cybercrime activities, including hacker attacks, online fraud, and illegal transactions, etc. These activities usually involve the use of illegal means to obtain profits, posing a threat to normal business activities and social order. Business data in the black and gray industry scenario refers to various data records suspected of black and gray industries. Such business data may include rich information, such as user identity information, registration information, login or access device information, interactive object information, and IP address (Internet Protocol Address) information. These business data can be obtained through user registration archives, user behavior logs, network transaction records, and social media lists of target objects suspected of black and gray industries.

[0063] In this embodiment, these business data are usually stored in different business systems or databases for different black and gray production scenarios, and can be obtained through web crawlers, API (Application Programming Interface) calls and database queries. For example, these business data can be pre-processed and stored in Hive tables, and Hive tables can be seamlessly integrated with Spark. Using Spark, business data in Hive tables can be quickly read for subsequent processing.

[0064] 102. Preprocess the business data to obtain multiple cache data blocks and multiple parallel partition data sets;

[0065] In this embodiment, preprocessing refers to a pre-processing method for processing data into the format required for subsequent steps. The preprocessing of business data in this embodiment is intended to divide the business data into multiple cache data blocks, and distribute the business data to multiple partitions, so as to obtain a partitioned data set for subsequent parallel processing. Among them, all cache data blocks share the memory space of a computing node, and each partitioned data set occupies the memory space of a computing node, so as to realize distributed partitioned storage of data. The memory space occupied by the cache data block can be divided into multiple sub-memory spaces, so that each cache data block corresponds to the sub-memory space one by one, so that the cache data blocks can be processed one by one or in batches.

[0066] Optionally, in one embodiment, see Figure 2 , pre-process the business data to obtain multiple cache data blocks and multiple parallel partition data sets, including:

[0067] 1021. Initialize business data and aggregate the business data into a business data set;

[0068] 1022. Distribute and store the business data set to obtain multiple partitioned data sets;

[0069] 1023. Divide the business data set into cache blocks to obtain multiple cache data blocks.

[0070] In this optional embodiment, during the initialization of business data, business data in the black and gray production scenarios obtained from various business systems or databases are aggregated to form a unified business data set. This data set contains the business data of all target objects suspected of black and gray production, providing a basis for subsequent analysis and identification.

[0071] In this optional embodiment, the business data set is processed in a distributed storage manner to obtain partitioned data blocks. Distributed storage can disperse large data sets to multiple computing nodes, thereby achieving high concurrent reading and writing of data and improving the parallelism and scalability of data processing. By distributing the business data set, we obtain multiple partitioned data sets, each of which contains a part of the business data, and each partitioned data set is located on an independent computing node, which is convenient for subsequent parallel processing.

[0072] In this optional embodiment, while obtaining the partitioned data set, the business data set is also divided into cache blocks. Cache block division is the process of dividing a large data set into multiple smaller cache data blocks, each of which contains a portion of the business data. The size of the cache data block can be adjusted according to the memory size of the computing node and the needs of data processing. Through cache block division, the business data can be dispersed into multiple cache data blocks, thereby realizing block-by-block or batch processing of the business data, which can further reduce the amount and complexity of single processing in the subsequent data processing process, and further improve the efficiency and accuracy of data processing. After the cache data block division is completed, each cache data block corresponds to a sub-memory space, and these sub-memory spaces together constitute the memory space of the computing node. In this way, we can realize the processing of cache data blocks one by one or in batches, thereby making full use of the memory space of the computing node and improving the efficiency of data processing.

[0073] In this optional embodiment, the process of initializing business data can process the business data into a data format that can be processed by Spark, and aggregate it so that Spark can be used to process the corresponding business data later. For example, the distributed data set Dataset of the Spark platform can be used. First, the business data configuration creates a SparkConf object to set the configuration properties of the Spark application, such as the application name, cluster deployment, etc., and then creates a SparkSession object through SparkSession.builder() as the entry point of Spark SQL, which can be used to perform DataFrame and SQL operations, configure Hive support, and finally obtain a SparkSession instance. Then call the sql() method of the object to read data from the hive table and create a distributed data set. Once the creation is completed, this distributed data set can be operated in parallel, that is, datasource = Spark.sql(sql), and the data set Dataset is obtained. Each business data in the Dataset can be converted into a corresponding Log object through the map() function to obtain the data source Dataset. <log>As a custom class, the Log object can include business data and detailed data and features in the business data.

[0074] In this optional embodiment, Spark can also be used to divide the partitioned data set and cached data blocks. <log>After the data source is obtained, the corresponding distributed storage and cache block division can be performed through repation operations. For example, using Dataset <log>The .repatition(n) operation divides the data source into n partitions. Each partition has an independent dataset. The datasets in each partition occupy the resources of a computing node and can be computed in parallel. Similarly, the dataset can be used to <log>The .repatition(m) operation divides the data source into m cache data blocks according to the data volume. All cache data blocks can share the memory space of a computing node, which is convenient for processing block by block or partition by partition.

[0075] Optionally, in one embodiment, the business data is preprocessed to obtain a plurality of cache data blocks and a plurality of parallel partition data sets, and further includes: assigning a unique identifier to each business data.

[0076] In this optional embodiment, the business data is distinguished by assigning a unique identifier. Since each business data may include information of multiple features, the information of corresponding features in different business data may be the same or similar, or some unique features are too long or easily confused with data of other features, and thus are usually not convenient for direct distinction, a unique identifier may be assigned to each business data to facilitate the distinction of the business data, and also facilitate the distinction and merging of clusters containing the same business data in a series of subsequent clustering processes of the business data.

[0077] In this optional embodiment, the unique identifier can be a sequential number, a code based on a specific feature, or a hash value that combines multiple factors. The unique identifier can be assigned through Spark. For example, the business data can be arranged in rows and then

[0078] The functions.monotonically_increasing_id().alias("row_id") operation is used to add an auto-incrementing row number to each row of data, so that each business data corresponds to a unique row number in sequence. The setting of the auto-incrementing row number makes it easy to distinguish the size of the unique identifier, which is conducive to the subsequent response distinction and merging processing based on the unique identifier.

[0079] 103. Calculate the similarity distance between the business data in each partition data set and the business data in each cache data block;

[0080] In this embodiment, the similarity distance is an indicator to measure the similarity between two business data. By calculating the similarity distance, business data with similar characteristics can be preliminarily identified. These data may be generated by the same black and gray production group, which provides an important basis for subsequent cluster analysis and group identification. Among them, the similarity distance can be calculated based on different similarity distance calculation functions or similarity evaluation models, such as Euclidean distance, cosine similarity and Manhattan distance algorithms. In the calculation process, the characteristics of the business data are used as the independent variables of the corresponding function or the input of the corresponding model, so as to obtain the similarity distance.

[0081] In this embodiment, after the partitioning and memory space division mentioned above, the business data in the partitioned data set and the business data in each cache data block are calculated, and then the similarity distance between each business data in the partitioned data set and all the business data in the partition and all the data in the cache data block can be determined. For example, 500 business data are divided into 20 partitioned data sets and 10 cache data blocks by an average division method. Then, each partitioned data set includes 25 business data, and each cache data block includes 50 business data. After the calculation, there are 200 similarity distance calculation results, and each calculation result contains 1250 groups of similarity distances. The similarity distance results can be aggregated later, and the similarity distance calculated between the same two business data is excluded, so that the similarity distance between each business data and other business data can be determined.

[0082] In this embodiment, the calculation process is performed in the partitions, so the memory space of the cache data blocks storing a large amount of data does not need to participate in the calculation. It only needs to be able to provide the cache data blocks in the sub-memory space to each partition one by one or in batches. The corresponding calculations are distributed to each partition and performed in parallel. Each partition calculates a smaller amount of data, which has lower requirements on the computing power of the computing nodes, and the complexity of the calculation is also reduced, which can be calculated more efficiently, stably and reliably. After the calculation is completed, the calculated similarity distance can be clustered accordingly. The clustering process usually only involves simple data transmission and elimination of duplicate data, and the memory consumption is low.

[0083] Optionally, in one embodiment, see Figure 3 , calculate the similarity distance between the business data in each partition data set and the business data in each cache data block, including:

[0084] 1031. Broadcast each cached data block so that the partition where each partitioned data set is located receives the business data in each cached data block;

[0085] 1032. Calculate the similarity distance between all business data in each partition data set and the business data in each cache data block based on a preset similarity distance function.

[0086] In this optional embodiment, the purpose of broadcasting each cache data block is to ensure that each partition data set can access the business data in all cache data blocks. Since the cache data blocks may be stored on different computing nodes, and the partition data sets are also scattered on various computing nodes, it is necessary to send the business data in the cache data blocks to the computing nodes where each partition is located through a broadcast mechanism, so that each partition can obtain the business data in the complete cache data blocks, providing a data basis for the calculation of similarity distance.

[0087] In this optional embodiment, the preset similarity distance function can be defined based on the characteristics of the business data, or it can be an existing similarity evaluation model selected according to a specific application scenario. The function accepts two business data as input and outputs the similarity distance value between them. Generally, the smaller the value, the more similar the two business data are; the larger the value, the greater the difference between them. The calculation process is performed in parallel on the computing nodes where each partition is located, which can effectively improve the calculation efficiency.

[0088] In this optional embodiment, the broadcast and calculation process can also be implemented based on Spark. <log>The .collect() method pulls cached data blocks one by one or in batches to the Spark Driver process to schedule the cached data blocks, then creates broadcast variables to broadcast the cached data blocks, and receives the broadcasts in each partition so that each partition gets a copy of the same cached data block. The mapPartition() function is called once for each business data in the partitioned dataset Dataset of each partition to calculate the similarity distance with each business data in the cached data block Dataset.

[0089] Optionally, in some embodiments, the business data includes multiple feature data with different weights, and the similarity distance between all business data in each partitioned data set and the business data in each cache data block is calculated based on a preset similarity distance function, including: calculating the sum of the weights of the same feature data between each of the business data, and taking the sum of the weights as the similarity distance.

[0090] In an optional embodiment, the business data may include multiple feature data with different weights, which reflect multiple aspects of the business data. For example, the business data in a certain black and gray production scenario includes four main feature data: user account, registration time, mobile phone number, and device name. Among them, the registration time of user A is a1, the mobile phone number is b1, and the device name is c1, the registration time of user B is a1, the mobile phone number is b1, and the device name is c1, and the registration time of user C is a1, the mobile phone number is b1, and the device name is c1. If the weight of the registration time is set to 0.2, the weight of the mobile phone number is set to 0.6, and the weight of the device name is set to 0.3, then the similarity distance between user A and user B, that is, the sum of the weights, is 0.2+0.6+0.3=1.1, and the similarity distance between user C and user A and user B is 0.2+0.6=0.8. The calculation method of the sum of weights fully considers the importance of different features in the business data, making the calculation of the similarity distance more reasonable and accurate. The calculation complexity of this summation method is also low and has good interpretability, which is conducive to further improving the efficiency of the calculation and provides strong support for subsequent gang identification and analysis.

[0091] 104. Determine a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold, and a similarity distance;

[0092] In this embodiment, after determining the similarity distance, the closeness between each business data can be evaluated by presetting two parameters, namely, a neighborhood radius threshold and a preset neighborhood point number threshold. The closeness between business data can be indicated by the close similarity distance, and the presence of multiple business data with close similarity distances indicates that the multiple business data are closely connected, thereby reflecting the group characteristics of the data. When there are many business data with a close distance to one business data, it can usually be said that the member corresponding to the business data is a core member of the group, and the business data can be regarded as a core business data.

[0093] In this embodiment, in order to measure the two characteristics of the judgment condition of close similarity distance and the large number of points in close distance, the gang can be judged by the preset neighborhood radius threshold and the preset neighbor number threshold. There is a density-direct relationship between the core business data and other business data in the temporary cluster, that is, starting from a core business data, other business data can be directly reached within the preset neighborhood radius threshold. For each core business data, it can be formed into a temporary cluster with all neighbor business data that meet the similarity distance conditions. The determined temporary cluster is the determined black and gray production gang with different core business data as the core, and each business data that meets the conditions of the preset neighborhood radius threshold and the preset neighbor number threshold corresponds to a corresponding temporary cluster with the business data as the core business data.

[0094] Optionally, in one embodiment, see Figure 4 , based on a preset neighborhood radius threshold, a preset neighbor number threshold and a similarity distance, a temporary clustering cluster is determined, including:

[0095] 1041. Aggregate multiple similarity distances to obtain a similarity distance data set;

[0096] 1042. Based on a preset neighborhood radius threshold and a similarity distance data set, the business data whose similarity distance to each business data is not greater than the preset neighborhood radius threshold is used as neighboring business data;

[0097] 1043. The business data whose number of neighbor business data is not less than a preset neighbor number threshold is used as core business data, and the core business data and the corresponding neighbor business data are used as temporary clusters corresponding to the core business data.

[0098] In this optional embodiment, similarity aggregation is to unify the results of the calculated similarity distance into a similarity distance data set, which can quickly determine the similarity distance between every two business data based on the similarity distance data set, which is conducive to the subsequent judgment and clustering of relationships such as density directness and density reachability between business data.

[0099] In this optional embodiment, refer to Figure 5 , the aggregation of multiple similarity distances can also be achieved based on Spark. For example, in the previous steps, the distance similarity of the business data between n partitioned data sets (Patition) and m cached data blocks (Buffer) has been calculated, so m*n intermediate similarity results can be obtained. Each partition includes m results, and each result block includes the product of the number of business data in the corresponding partitioned data set and the number of business data in the cached data block. Based on these m*n results, the calculation results in each partition can be merged through the mapPartition operation and then aggregated through the join() function. The results of each partition can be merged to obtain n data sets Dataset <tempres>1 to Dataset <tempres>n, or extract the partition results one by one to obtain m data sets Dataset <tempres>1 to Dataset <tempres>m, and then the unique similarity distance dataset Dataset can be obtained through the join() operation <tempres>.

[0100] In this optional embodiment, after aggregating multiple similarity distances to obtain a similarity distance data set, the business data whose similarity distance to each business data is not greater than the preset neighborhood radius threshold can be screened out as neighboring business data according to the preset neighborhood radius threshold and the similarity distance data set. This step actually defines the concept of a "neighborhood", that is, all business data whose similarity distance to a certain business data is within a certain range are regarded as neighbors of the business data, indicating that the relationship between the business data is relatively close.

[0101] In this optional embodiment, the business data whose number of neighbor business data is not less than the preset neighbor number threshold is used as the core business data. The core business data here can be understood as the core members of the gang. There are enough neighbor business data around the core members, and there is a strong correlation between the core business data and these neighbor business data. By using the core business data and the corresponding neighbor business data as a temporary cluster with the core business data as the core, this temporary cluster is the black and gray production gang with the members corresponding to the core business data as the core. For example, there are multiple business data 1, 2, 3, 4, 5, among which the similarity distance between 1 and 2, 3, 4, 5 is less than the preset neighborhood radius threshold, the similarity distance between 2 and 1, 3 is less than the preset neighborhood radius threshold, and the similarity distance between 2 and 4, 5 is greater than the preset neighborhood radius threshold. Therefore, the neighbor business data of 1 are 2, 3, 4 and 5, that is, the number of corresponding neighbor business data is 4, and the neighbor business data of 2 are 1 and 3, that is, the number of corresponding neighbor business data is 2. If the preset neighbor number threshold is 3, then it can be determined that the obtained temporary clustering cluster takes 1 as the core sample data, and the sample data included are 1, 2, 3, 4, 5.

[0102] In this optional embodiment, the process of forming temporary clusters based on the preset neighborhood radius threshold, the preset neighbor number threshold and the similarity distance data set can also be quickly implemented based on Spark. For example, in the only Dataset obtained above <tempres>On the basis of, the neighboring business data of each business data can be determined by grouping, and the number of corresponding neighboring business data can be counted. When the number of neighboring business data is not less than the preset neighbor number threshold through comparison, a temporary clustering cluster Dataset is generated based on the business data and its neighboring business data <cluster>, and can maintain its core point list accordingly.

[0103] 105. Repartition the temporary clusters, and gradually reduce the number of partitions until all temporary clusters with reachable density are merged into the same partition to obtain the final clusters;

[0104] In this embodiment, after determining the temporary clusters corresponding to multiple different business data as core business data, the members of the same black and gray production gang can be further associated and confirmed to ensure that the final clusters can be more accurate and in line with the actual characteristics of the black and gray production gangs.

[0105] In this embodiment, the association between clusters is determined based on the density reachability relationship. Density reachability means that starting from one business data, another business data can be reached through a series of density-directly reachable business data. If two business data are density-reachable, they can be considered to belong to the same black and gray production gang. For example, a partition includes eight business data, 1, 2, 3, 4, 5, 6, 7, and 8, of which 4 and 5 can be used as core business data. The temporary cluster corresponding to 4 includes the business data 1, 2, 3, 4, 5, and 7, and the temporary cluster corresponding to 5 includes the business data 3, 4, 5, 6, and 7. Both temporary clusters include four business data, 3, 4, 5, and 7. The temporary cluster corresponding to 4 does not include business data 6, and the business data corresponding to 5 does not include business data 1 and 2. Then 6 is the density reachable point of the temporary cluster corresponding to 4, and 1 and 2 are the density reachable points of the temporary cluster corresponding to 5. On this basis, the temporary cluster corresponding to 4 and the temporary cluster corresponding to 5 can be merged, and the resulting cluster includes the seven business data 1, 2, 3, 4, 5, 6, and 7.

[0106] In this embodiment, the density reachable point can be used to expand the scope of business data covered by the temporary cluster. The density reachable point corresponding to the core point is usually related to other business data in the temporary cluster where the core point is located. Therefore, the density reachable business data in the temporary cluster and the members corresponding to the density reachable business data of the business data in the temporary cluster usually belong to the same black and gray production group. By continuously judging all business data through the relationship between density direct access and density reachable, a final cluster containing at least one black and gray production group can be obtained in the end to ensure that the members corresponding to the business data in each final cluster identified are in the same black and gray production group.

[0107] In this embodiment, repartitioning is an iterative process that is performed multiple times. Each iteration will adjust the partitions according to the current partition situation and the situation of the clusters in each partition, so as to continuously divide each business data and its corresponding density-direct and density-reachable business data into the same final cluster according to the relationship between density-direct and density-reachable. As the iteration proceeds, the number of partitions will gradually decrease, and at the same time, the business data in each partition will gradually become more compact and consistent. Finally, when all density-reachable temporary clusters are merged into the same partition, the final cluster can be obtained. The final cluster has one or more clusters, corresponding to one or more black and gray production groups.

[0108] Optionally, in one embodiment, see Figure 6 , repartition the temporary clusters, and gradually reduce the number of partitions until all density-reachable temporary clusters are merged into the same partition to obtain the final clusters, including:

[0109] 1051. Aggregate multiple temporary clusters to obtain a temporary cluster data set;

[0110] 1052. Distribute and store the temporary clustering data set into a plurality of partitions, so that each partition includes at least one temporary clustering cluster in the temporary clustering data set;

[0111] 1053. Merge the temporary clusters that are density-reachable in each partition to obtain the intermediate clusters in each partition;

[0112] 1054. When the number of partitions including the intermediate cluster is not 1, merge the intermediate clusters in at least two partitions into the same partition; or

[0113] 1055. Aggregate the intermediate clusters in each partition to obtain an intermediate cluster data set, and distribute and store the intermediate cluster data set into a smaller number of partitions;

[0114] 1056. Merge the temporary clusters that are density-reachable in each new partition to update the intermediate clusters in each partition;

[0115] 1057. When the number of partitions including the intermediate cluster is 1, the intermediate cluster is used as the final cluster.

[0116] In this optional embodiment, multiple temporary clusters can be aggregated first to facilitate partitioning of the temporary clusters. Since the temporary clusters are optionally obtained based on a unique similarity distance data set in the aforementioned process, each temporary cluster may be pre-aggregated under the aforementioned different processing to obtain a corresponding temporary cluster data set. After obtaining the temporary clusters, the temporary clusters can be distributed and stored in multiple partitions, so that each partition includes at least one temporary cluster in the temporary cluster data set, so as to further merge the temporary clusters to more deeply determine the scope of the black and gray production gangs. The merging of temporary clusters within a partition is based on density reachability. If the business data between two clusters has a density reachability relationship, then it can be considered that the two clusters belong to the same black and gray production gang. After the merger, at least one intermediate cluster will be formed in each partition. The intermediate cluster is closer to the real black and gray production gang structure than the aforementioned temporary cluster. If the number of intermediate clusters is greater than 1, it usually indicates that the business data in the partition involves at least two black and gray production gangs or that there are no key core members who can connect the relationships between some of the black and gray production gangs.

[0117] In this optional embodiment, different numbers of partitions are handled differently. When the number of partitions including the intermediate clusters is greater than 1, it means that there are still multiple possible related members of the black and gray production gangs that have not been merged into the corresponding black and gray production gangs. Therefore, it is necessary to gradually reduce the number of partitions and gradually merge the clusters. When the number of partitions is 1, and the density-direct and density-reachable business data have been assigned to the corresponding clusters, it means that all members of the black and gray production gangs have been found, and the members and structure of the black and gray production gangs can be determined based on the final clusters obtained. Among them, when the number of partitions is not 1, there are two main ways to merge partitions. One is to merge at least two partitions that have obtained the intermediate clusters into the same partition to gradually reduce the number of partitions. The other is to aggregate the intermediate clusters and divide them into a smaller number of partitions to reduce the number of partitions, and then perform a new round of merging operations until all business data are merged and clustered into one partition.

[0118] In an alternative embodiment, see Figure 7 , the process of multiple iterations of repartitioning and merging can also be implemented based on Spark. The temporary clusters can be repartitioned through the rePartition() operation to obtain repartitioned data, and each partition includes at least one temporary cluster. After that, the temporary clusters in each partition can be merged through the mapPartion() operation to obtain intermediate clusters. After obtaining the intermediate clusters, the number of partitions is continuously reduced, and the clusters are merged iteratively for multiple times. Finally, the clusters are merged within one partition to obtain the final cluster. In the example in this figure, the temporary clusters are partitioned into k partitions. In the process of reducing the partitions, the partitions are halved and merged, and this process is repeated until the final cluster Dataset is obtained. <cluster>.

[0119] Optionally, in one embodiment, a temporary clustering data set is distributed and stored in multiple partitions, including: temporary clustering clusters of business data with at least one identical unique identifier are distributed and stored in the same partition; and intermediate clustering data sets are distributed and stored in a smaller number of partitions, including: intermediate clustering clusters of business data with at least one identical unique identifier are distributed and stored in the same partition.

[0120] In this optional embodiment, in the case of assigning unique identifiers to business data as mentioned above, temporary clusters or intermediate clusters with at least one identical unique identifier can be divided into the same partition. In the case of business data with the same unique identifier, it usually indicates that the same business data exists in the corresponding two clusters, and the probability of density being reached is greater. The judgment that the unique identifier is the same can also be used to compare more business data in the clusters, thereby improving the effectiveness of partition merging. The business data in the clusters can also be sorted according to some sorting rules before comparison. For example, the business data in each cluster can be pre-arranged in numerical or symbolic order. When the business data has the same smallest unique identifier or the same largest unique identifier, the corresponding clusters are divided into the same partition. In this process, the hash function can also be used to determine the hash parameters of the partition according to the unique identifier, which is conducive to more efficient and rapid merging of the corresponding intermediate clusters.

[0121] 106. Identify the target objects corresponding to the business data in the same final cluster as members of the same black and gray production gang.

[0122] In this embodiment, the final clusters can be obtained through the aforementioned series of calculations and iterative processes. The business data in each final cluster is the business data generated by the same black and gray production group. Therefore, by corresponding the business data in each final cluster to the actual target object, it can be determined which target objects are members of the same black and gray production group. The target object may be a specific user, account, device or IP address, etc. The target object can be obtained from some specific information in the business data, or it can be determined by using the mapping relationship between business data and target objects in a database or business system. The target objects are related to each other through business data and form a corresponding group structure. These target objects are members of the same black and gray production group.

[0123] The method provided in this embodiment reduces the complexity of black and gray product data processing and the memory consumption of computing nodes through distributed computing, can realize parallel and efficient processing of large-scale black and gray product business data, can more accurately and reliably identify black and gray product gangs, and help improve the business security level. On the basis of efficiently and accurately identifying black and gray product gangs in this embodiment, the identified black and gray product gangs can be further processed, such as establishing a real-time monitoring system for each member or core member, providing the black and gray product gang identification results to the risk control system, and providing the black and gray product gang identification results to the data analysis team for analysis to establish gang models and gang identification models, etc., which is helpful for subsequent crackdowns on illegal black and gray product activities and provides strong support for risk management and security prevention. In addition, the execution of this method can be realized by simply obtaining business data and using Spark operations, which is easy to deploy and control execution, and can further improve the convenience and flexibility of black and gray product gang identification.

[0124] In order to execute the corresponding steps in the above method embodiment and each optional implementation, the following provides an implementation method of a black and gray production group identification device. Figure 8 , Figure 8 This is a functional module diagram of an embodiment of a device for identifying a black and gray production group in an embodiment of the present invention. In this embodiment, the device for identifying a black and gray production group is applied to a computer device, and the device for identifying a black and gray production group includes:

[0125] An acquisition module 201 is used to acquire business data in the black and gray industry scenario;

[0126] A preprocessing module 202 is used to preprocess the business data to obtain multiple cache data blocks and multiple parallel partition data sets;

[0127] A calculation module 203, used to calculate the similarity distance between the business data in each partition data set and the business data in each cache data block;

[0128] A first clustering module 204 is used to determine a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and a similarity distance;

[0129] The second clustering module 205 is used to re-partition the temporary clusters and gradually reduce the number of partitions until all temporary clusters with reachable density are merged into the same partition to obtain the final clusters;

[0130] The identification module 206 is used to identify the target objects corresponding to the business data in the same final cluster as members of the same black and gray production gang.

[0131] Optionally, in one embodiment, the preprocessing module 202 is specifically used to: initialize business data and aggregate the business data into a business data set; distribute the business data set to obtain multiple partitioned data sets; divide the business data set into cache blocks to obtain multiple cache data blocks.

[0132] Optionally, in one embodiment, the calculation module 203 is specifically used to: broadcast each cache data block so that the partition where each partitioned data set is located receives the business data in each cache data block; calculate the similarity distance between all business data in each partitioned data set and all business data in each cache data block based on a preset similarity distance function.

[0133] Optionally, in one embodiment, the business data acquired by the acquisition module 201 includes multiple feature data with different weights; the calculation module 203 is specifically used to: calculate the sum of weights of the same feature data between each business data, and use the sum of weights as the similarity distance.

[0134] Optionally, in one embodiment, the first clustering module 204 is specifically used to: aggregate multiple similarity distances to obtain a similarity distance data set; based on a preset neighborhood radius threshold and a similarity distance data set, use business data whose similarity distance with each business data is not greater than a preset neighborhood radius threshold as neighbor business data; use business data whose number of neighbor business data is not less than a preset neighbor number threshold as core business data, and use the core business data and the corresponding neighbor business data as temporary clustering clusters corresponding to the core business data.

[0135] Optionally, in one embodiment, the second clustering module 205 is specifically used to: aggregate multiple temporary clusters to obtain a temporary cluster data set; store the temporary cluster data set in a distributed manner in multiple partitions so that each partition includes at least one temporary cluster in the temporary cluster data set; merge the density-reachable temporary cluster clusters in each partition to obtain intermediate cluster clusters in each partition; when the number of partitions including intermediate cluster clusters is not 1, merge the intermediate cluster clusters in at least two partitions into the same partition; or aggregate the intermediate cluster clusters in each partition to obtain an intermediate cluster data set, and store the intermediate cluster data set in a distributed manner in a smaller number of partitions; merge the density-reachable temporary cluster clusters in each new partition to update the intermediate cluster clusters in each partition; when the number of partitions including intermediate cluster clusters is 1, use the intermediate cluster cluster as the final cluster cluster.

[0136] Optionally, in one embodiment, the preprocessing module 202 is specifically used to: assign a unique identifier to each business data; the second clustering module 205 is specifically used to: distribute and store temporary clusters of business data with at least one identical unique identifier in the same partition, and distribute and store intermediate clusters of business data with at least one identical unique identifier in the same partition.

[0137] Since the embodiments of the device part correspond to the embodiments of the above-mentioned method, please refer to the above-mentioned method embodiments for the introduction of the black and gray production gang identification device provided in the embodiments of the present invention. The embodiments of the present invention will not be repeated here, and it has the same beneficial effects as the above-mentioned black and gray production gang identification method.

[0138] The present invention also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the method for identifying black and gray production gangs as described above. Fig. 9 3 is a functional module diagram of a computer device provided by an embodiment of the present invention. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 300. Furthermore, the processor 310 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0139] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Fig. 9 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0140] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the method for identifying black and gray production gangs as described above.

[0141] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. If the integrated module or unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0142] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / cluster> < / cluster> < / tempres> < / tempres> < / tempres> < / tempres> < / tempres> < / tempres> < / log> < / log> < / log> < / log> < / log>

Claims

1. A method for identifying black and gray production groups, characterized in that: The method for identifying black and gray production groups includes: Obtain business data in black and gray industry scenarios; Preprocessing the business data to obtain multiple cache data blocks and multiple parallel partition data sets; Calculating the similarity distance between the service data in each partition data set and the service data in each cache data block; Determine a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and the similarity distance; Repartitioning the temporary clusters, and gradually reducing the number of partitions until all the temporary clusters with reachable densities are merged into the same partition to obtain final clusters; The target objects corresponding to the business data in the same final cluster are identified as members of the same black and gray production gang.

2. The method for identifying black and gray production groups according to claim 1 is characterized in that: The preprocessing of the business data to obtain a plurality of cache data blocks and a plurality of parallel partition data sets includes: Initializing the business data and aggregating the business data into a business data set; Distributed storage is performed on the business data set to obtain a plurality of partitioned data sets; The business data set is divided into cache blocks to obtain a plurality of cache data blocks.

3. The method for identifying black and gray production groups according to claim 1 is characterized in that: The calculating the similarity distance between the service data in each partition data set and the service data in each cache data block includes: Broadcasting each of the cache data blocks so that the partitions where each of the partition data sets is located receive the service data in each of the cache data blocks; The similarity distance between all the business data in each partition data set and all the business data in each cache data block is calculated based on a preset similarity distance function.

4. The method for identifying black and gray production groups according to claim 3 is characterized in that: Each of the business data includes a plurality of feature data with different weights; The calculating the similarity distance between all the business data in each partition data set and all the business data in each cache data block based on a preset similarity distance function includes: The sum of the weights of the same feature data between the business data is calculated, and the sum of the weights is used as the similarity distance.

5. The method for identifying black and gray production groups according to claim 1 is characterized in that: The determining of a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and the similarity distance includes: Aggregating a plurality of the similarity distances to obtain a similarity distance data set; Based on the preset neighborhood radius threshold and the similarity distance data set, the business data whose similarity distance with each business data is not greater than the preset neighborhood radius threshold is used as neighboring business data; The business data whose number of neighbor business data is not less than the preset neighbor number threshold is used as core business data, and the core business data and the corresponding neighbor business data are used as a temporary cluster corresponding to the core business data.

6. The method for identifying black and gray production groups according to claim 5 is characterized in that: The repartitioning of the temporary clusters and gradually reducing the number of partitions until all the temporary clusters with reachable densities are merged into the same partition to obtain final clusters includes: Aggregating the plurality of temporary clusters to obtain a temporary cluster data set; Distribute and store the temporary clustering data set into a plurality of partitions, so that each partition includes at least one temporary cluster in the temporary clustering data set; Merging the temporary clusters with density reachable in each of the partitions to obtain intermediate clusters in each partition; When the number of partitions including the intermediate cluster is not 1, merging the intermediate clusters in at least two partitions into the same partition; or Aggregating the intermediate clusters in each of the partitions to obtain an intermediate cluster data set, and distributing and storing the intermediate cluster data set in a smaller number of partitions; Merging the temporary clusters that are density-reachable in each of the new partitions to update the intermediate clusters in each partition; When the number of partitions including the intermediate cluster is 1, the intermediate cluster is used as the final cluster.

7. The method for identifying black and gray production groups according to claim 6 is characterized in that: The preprocessing of the business data to obtain a plurality of cache data blocks and a plurality of parallel partition data sets also includes: Assigning a unique identifier to each of the business data; The step of distributing and storing the temporary cluster data set into a plurality of partitions comprises: Distribute and store the temporary clusters of the business data having at least one identical unique identifier in the same partition; The distributing and storing the intermediate clustering data set into a smaller number of partitions includes: The intermediate clusters of the business data having at least one identical unique identifier are distributed and stored in the same partition.

8. A device for identifying black and gray production groups, characterized in that: The black and gray production group identification device includes: The acquisition module is used to obtain business data in the black and gray industry scenarios; A preprocessing module, used for preprocessing the business data to obtain multiple cache data blocks and multiple parallel partition data sets; A calculation module, used for calculating the similarity distance between the business data in each partition data set and the business data in each cache data block; A first clustering module, configured to determine a temporary cluster based on a preset neighborhood radius threshold, a preset neighbor number threshold and the similarity distance; A second clustering module is used to repartition the temporary clusters and gradually reduce the number of partitions until all the temporary clusters with reachable densities are merged into the same partition to obtain final clusters; The identification module is used to identify the target objects corresponding to the business data in the same final cluster as members of the same black and gray production gang.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the computer device executes the method for identifying black and gray production gangs as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the method for identifying black and gray production gangs as described in any one of claims 1-7 is implemented.

Images