Hotspot feature extraction method and device, equipment and medium

Abstract

The invention discloses a hotspot feature extraction method and device, equipment and a medium. The hotspot feature extraction method comprises the steps that real-time storage consumption is carried out on business data through a message storage queue so as to optimize the processability of the business data, and the business data are generated by a website, an application or running equipment in real time; performing clustering consumption on the message storage queue through a clustering algorithm to identify gang clustering data in the service data; performing frequent item mining on the gang clustering data to extract a frequent item set in the gang clustering data; and storing the frequent item set in the Elasticsearch, and executing measurement aggregation based on the Elasticsearch so as to extract the hot spot features. By means of the mode, real-time extraction of the hotspot features can be accurately and efficiently carried out in a large amount of data.

Description

Technical Field

[0001] The present application relates to the field of business data security, and in particular to a hotspot feature extraction method, device, equipment and medium. Background Art

[0002] In the field of business security, group identification technology plays a key role in identifying black and gray industries. By analyzing the behavioral patterns and interactive relationships between groups, group identification technology can reveal the coordinated individual networks in malicious activities. This helps to identify the main players, understand the relationship between different entities, and detect the organizational structure of the group. In this way, group identification technology can effectively reveal the key roles and operating methods in the black and gray industry network, thereby helping to strengthen risk management and combat illegal activities.

[0003] After clustering, frequent item extraction is usually performed to generate a large number of frequent item sets. As the number of groups and the size of data grow, the amount of data that needs to be manually analyzed increases significantly. This results in a large amount of manpower and time required to process and analyze this data. Manual analysis is slow in processing large amounts of data and it is difficult to respond to emerging threats in a timely manner. At the same time, manual analysis of data processing methods makes it difficult to obtain a global perspective from a large amount of detailed data, and it is easy to miss some important patterns and associations. Summary of the invention

[0004] The present application mainly provides a hotspot feature extraction method, device, equipment and medium to solve the problem of difficulty in extracting hotspot features from large-scale data.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a hot spot feature extraction method, including: real-time storage and consumption of business data through a message storage queue to optimize the processability of the business data, wherein the business data is generated in real time by a website, application or running device; clustering and consuming the message storage queue through a clustering algorithm to identify group clustering data in the business data; frequent item mining on the group clustering data to extract frequent item sets in the group clustering data; storing frequent item sets in Elasticsearch, and performing metric aggregation based on Elasticsearch to extract hot spot features.

[0006] In some embodiments, the method of performing Elasticsearch-based metric aggregation to extract hotspot features includes: creating and initializing a data structure; traversing the rule fields of each of the frequent itemsets in the Elasticsearch, and storing the records returned by each of the frequent itemsets during the traversal in the data structure; returning the records stored in the data structure, and integrating them into the hotspot features output by the Elasticsearch.

[0007] In some embodiments, the creation and initialization of the data structure includes: creating and initializing a first hash mapping structure, setting keywords based on the first hash mapping structure, each keyword corresponding to multiple frequent item sets; traversing the rule field of each frequent item set in the Elasticsearch, and storing the records returned by each frequent item set during the traversal process in the data structure, including: traversing the rule field of each frequent item set, and performing a non-empty check on each rule field; and updating the rule data corresponding to each frequent item set stored in the keyword according to the check results returned by each rule field.

[0008] In some embodiments, the updating of the rule data corresponding to each of the frequent item sets stored in the keyword according to the check results returned by each of the rule fields includes: in response to the check result of the rule field being not empty, traversing each rule item in the rule field that is not empty, and performing a non-empty check on each of the rule items; in response to the check result of the rule item being not empty and the corresponding rule item already existing in the keyword, updating the quantity value of the rule item corresponding to the frequent item set in the keyword; or, in response to the check result of the rule item being not empty and the corresponding rule item not existing in the keyword, adding the rule item corresponding to the frequent item set in the keyword and setting the quantity value corresponding to the rule item.

[0009] In some embodiments, returning the records stored in the data structure and integrating them into the hot spot features output by the Elasticsearch includes: returning the rule data of each keyword in the first hash mapping structure to a preset second hash mapping structure; in response to returning new rule data in the second hash mapping structure, updating the corresponding rule data stored in the second hash mapping structure based on the new rule data; and obtaining the hot spot features in the updated rule data.

[0010] In some embodiments, the corresponding rule data stored in the second hash mapping structure is updated based on the new rule data, including: traversing the rule data stored in the second hash mapping structure one by one based on each new rule; in response to the existence of the same rule items in the second hash mapping structure, updating the quantity value of the rule items; or, in response to the absence of the same rule items in the second hash mapping structure, adding the corresponding rule items in the second hash mapping structure and setting the quantity value corresponding to the rule items.

[0011] In some embodiments, storing frequent item sets in Elasticsearch and performing metric aggregation based on Elasticsearch to extract hot spot features also includes: obtaining a preset query method and a preset metric aggregation operation; filtering the frequent item sets for performing metric aggregation based on the Elasticsearch based on preset filtering conditions; performing metric aggregation based on the Elasticsearch on the filtered frequent item sets according to the query method and the metric aggregation operation, and obtaining rule items output by the Elasticsearch and the quantity values ​​corresponding to each of the rule items; selecting the rule items whose corresponding quantity values ​​meet the preset conditions as the hot spot features.

[0012] To solve the above technical problems, another technical solution adopted in the present application is: to provide a hot spot feature extraction device, the hot spot feature extraction device comprising: a data processing module, used to store and consume business data in real time through a message storage queue to optimize the processability of the business data, the business data is generated in real time by a website, application or running device; a group clustering algorithm module, used to cluster and consume the message storage queue through a clustering algorithm to identify group clustering data in the business data; a frequent item mining module, used to perform frequent item mining on the group clustering data to extract frequent item sets in the group clustering data; a hot spot feature extraction module, used to store frequent item sets in Elasticsearch, and perform metric aggregation based on Elasticsearch to extract hot spot features.

[0013] The present application also provides a computer device, which includes: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the hotspot feature extraction method as described above.

[0014] The present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the hotspot feature extraction method as described above is implemented.

[0015] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a method, device, equipment and medium for extracting hotspot features. Business data is stored and consumed in real time through a message storage queue to optimize the processability of business data. Business data is generated in real time by a website, application or running device; using a message storage queue can efficiently and stably process a large amount of business data and realize real-time data transmission. Clustering and consuming the message storage queue through a clustering algorithm to identify group clustering data in the business data; using a suitable clustering algorithm to process the data, quickly discover the group pattern in the business, and provide a basis for subsequent frequent item extraction. Frequent item mining is performed on the group clustering data to extract frequent item sets in the group clustering data; extracting the co-occurrence pattern of frequent items is conducive to an in-depth understanding of the behavior of the group. Frequent item sets are stored in Elasticsearch, and metric aggregation based on Elasticsearch is performed to extract hotspot features. By leveraging Elasticsearch's excellent indexing and search capabilities, it is possible to efficiently process and retrieve a large number of complex frequent item sets and provide almost real-time analysis, meeting the needs of real-time feature extraction and accurately and efficiently extracting hotspot features from large amounts of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flow chart of an embodiment of a hotspot feature extraction method provided by the present application; Figure 2 Yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 400 in the method shown; Figure 3 Yes Figure 2 A schematic diagram of a flow chart of an embodiment of the method shown; Figure 4 Yes Figure 3 A schematic diagram of a flow chart of an embodiment of step 422 in the method shown; Figure 5 Yes Figure 2 A schematic diagram of a flow chart of an embodiment of step 430 in the method shown; Figure 6 Yes Figure 5 A schematic diagram of a flow chart of an embodiment of step 432 in the method shown; Figure 7 Yes Figure 1A flow chart of another embodiment of step 400 in the method shown; Figure 8 It is a structural schematic diagram of an embodiment of a hotspot feature extraction device in an embodiment of the present invention; Fig. 9 It is a structural diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0019] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a hotspot feature extraction method provided by the present application. The hotspot feature extraction method based on gang rules includes: Step 100: Real-time storage and consumption of business data is performed through a message storage queue to optimize the processability of the business data, where the business data is generated in real time by a website, application, or running device.

[0021] In this embodiment, Kafka is used as a message storage queue for illustration. First, Kafka is used as a real-time data stream processing platform to consume business data in real time. Kafka is a distributed, partitionable, multi-copy and persistent real-time messaging system that can process large-scale real-time data generated by websites, applications and running devices, and provide real-time data input and output capabilities.

[0022] Specifically, use Kafka consumers to consume business data from the Kafka cluster in real time. Configure parameters such as the number of consumer groups and automatic offset submission according to business needs. Configure the deserializer corresponding to the producer on the consumer side to correctly parse the data. Process the consumed data according to business needs, such as real-time analysis, real-time calculation, and real-time push. In the process of storage and consumption, the consumer is the business data collection program, and the consumption method is to store the collected business data in the message queue.

[0023] Using message storage queues can efficiently and stably process large amounts of business data and achieve real-time data transmission. Build a real-time data processing platform based on message storage queues to achieve real-time consumption and processing of business data generated by websites, applications, or running devices.

[0024] Clustering consumption and business data collection are performed asynchronously through the message storage queue as a buffer. After the business data is stored in the message queue, the collection work is continued. Compared with the business data collection program directly handing over the business data to the clustering program after collecting it, there is an additional message queue, thus realizing asynchronous consumption of business data, without waiting for the collected business data to complete clustering consumption before the next collection. This architecture not only improves the efficiency and real-time performance of data processing, but also enhances the scalability and fault tolerance of the system.

[0025] Step 200: Identify group cluster data in the consumed business data by using a clustering algorithm. Cluster consumption is performed on the message storage queue by using a clustering algorithm to identify group cluster data in the business data.

[0026] After obtaining the consumed business data from the message storage queue, the data is clustered and consumed in real time through a suitable clustering algorithm to quickly discover the group pattern in the business. In the cluster consumption process, the consumed party is the message storage queue, the consumer is the clustering algorithm, and the consumption method is group clustering.

[0027] Optionally, after obtaining the business data, a suitable clustering algorithm is selected according to the business data characteristics and clustering requirements, such as K-means, DBSCAN, hierarchical clustering, spectral clustering or density-based clustering.

[0028] Use stream processing frameworks such as Apache Flink and Spark Streaming to process real-time business data. Analyze the characteristics of each gang in the clustering results, such as transaction frequency, transaction amount, geographical distribution, etc. Identify common behavior patterns among gang members, such as frequent transactions and fund transfers.

[0029] The gang clustering data obtained by the clustering algorithm is organized into a transaction form, where each transaction contains a series of behaviors or feature items of the gang members.

[0030] A gang pattern usually refers to a collection of entities that show close associations or similar behaviors in the data. These entities can be users, transactions, devices, or any other business-related objects.

[0031] Apply the identified gang patterns to the risk monitoring system, or adjust business strategies based on gang characteristics and behavior patterns, such as adjusting transaction limits, strengthening identity authentication, etc., to provide a basis for subsequent frequent item extraction.

[0032] Step 300: Perform frequent item mining on the group clustering data to extract frequent item sets in the group clustering data.

[0033] Set the items to be mined. These items can be specific behaviors (such as transferring money, logging in, purchasing goods, etc.) or features (such as geographic location, device type, etc.). Select an appropriate frequent item mining algorithm to mine frequent items and obtain frequent item sets to reflect the specific behavior patterns or features of the gang.

[0034] Optionally, the FP-Growth algorithm is used to perform frequent item mining on the gang clustering data to extract frequent item sets in the gang clustering data.

[0035] The FP-Growth algorithm is used to mine frequent items from the obtained data. FP-Growth is an efficient frequent item set mining algorithm that uses a compact data structure, Frequent Pattern Tree (FP-tree), to extract the co-occurrence pattern of frequent items, which is conducive to a deep understanding of the behavior patterns of gangs.

[0036] Scan the transaction data set, count the frequency of occurrence of each item, and determine a minimum support threshold. Keep items whose frequency is greater than or equal to the minimum support threshold, and build a header table to store these frequent items and their occurrence counts. Traverse the transaction data set, sort each transaction (from high to low frequency), and build an FP-Tree.

[0037] Starting from the bottom of the header table, perform the following operations for each frequent item: find all prefix paths of the frequent item from the FP-Tree; build a conditional FP-Tree that only contains prefix paths related to the current frequent item; recursively mine frequent item sets in the conditional FP-Tree until the conditional FP-Tree contains only one item or is empty.

[0038] Finally, the mined frequent items are combined into frequent item sets to reflect the specific behavior patterns or characteristics of the gang.

[0039] Step 400: Store frequent itemsets in Elasticsearch, and perform Elasticsearch-based metric aggregation to extract hotspot features.

[0040] Each frequent item set mined is stored as a document in Elasticsearch. Each document contains metadata such as the item set identifier, item set content, and support. An index is created in Elasticsearch to store these documents. After the frequent item sets are stored in Elasticsearch, the aggregation function of Elasticsearch can be used to extract hot features, analyze the aggregation results, and find hot features.

[0041] Among them, hot spot features represent certain patterns or trends of gang behavior. In this embodiment, hot spot features refer to features that appear frequently in the data set. The extracted hot spot features are applied to scenarios such as risk monitoring, strategy optimization, and gang identification. For example, if the support for certain combinations of items is abnormally high, it may indicate some illegal activity or fraud. By finding hot spots or common patterns, it is helpful to understand and interpret the data more effectively, and then adjust the model and strategy to better meet business or experimental needs.

[0042] Elasticsearch is an open source search engine based on the Lucene library. It has distributed multi-user capabilities and is more suitable for large-scale indexing and searching in cloud environments. With its excellent indexing and search capabilities, Elasticsearch can efficiently process and retrieve a large number of complex frequent item sets and provide almost real-time analysis, meeting the needs of real-time feature extraction.

[0043] Elasticsearch's script aggregation allows you to use supported scripting languages ​​to set custom aggregation logic, such as how to calculate aggregation results at the single document, shard level, and cluster level. The results of the aggregation query will contain one or more hot item combinations and their support, which are used to analyze the common characteristics of gang behavior and further filter and locate key hot spots.

[0044] The problem of information overload can be effectively solved by extracting hotspot features. When processing a large amount of gang data, the amount of information calculation may be very huge. The hotspot feature extraction of gang rules can quickly extract key information from a large amount of rule data to help understand and process this data. At the same time, in the decision-making process, by extracting hotspot features, problems can be better processed in batches, and a basis can be provided to adjust existing models and analyze new trends. By extracting hotspot features in gangs, we can have a deeper understanding of the behavior patterns of gangs, find out their common characteristics, and even discover some of their hidden behaviors. By tracking and predicting hotspot features, we can automatically issue early warnings to gangs that exhibit specific behaviors, thereby helping to identify and prevent possible risks in a timely manner.

[0045] Further, see Figure 2 , step 400 comprises: Step 410: Create and initialize data structures.

[0046] The script initScript that creates and initializes the data structure is executed at the beginning of the aggregation execution and is executed and executed only once for each aggregation task to initialize the data structure that needs to be used in subsequent steps. This usually includes creating an empty collection, map, or counter to store and accumulate data in subsequent stages.

[0047] Further, see Figure 3 , step 410 includes: Step 411: Create and initialize a first hash mapping structure, set keywords based on the first hash mapping structure, and each keyword corresponds to multiple frequent item sets.

[0048] Create a hash map structure named state.keys in initScript to store rule items and their occurrence counts. Here, state is a keyword representing a global state object used to share data between different script stages.

[0049] Step 420: traverse the rule fields of each frequent itemset in Elasticsearch, and store the records returned by each frequent itemset during the traversal process in a data structure.

[0050] Each document record returned by the query is processed through the mapping script mapScript. Specifically, specific calculations or operations are performed on each document, and the results are stored in the data structure that has been initialized by the initScript script, including extracting field values ​​in the document and performing calculations or conversions based on business logic.

[0051] See also Figure 4Further, step 420 includes: Step 421: traverse the rule fields of each frequent item set and perform a non-empty check on each rule field.

[0052] In mapScript, iterate over the ruleList field of each document and update the number of occurrences of the rule item in state.keys.

[0053] Check the rule field for null. If the rule field is not null, enter the next loop and traverse each rule item in the rule field.

[0054] The rule field is a list or array containing a series of rule items.

[0055] Step 422: According to the check results returned by each rule field, the rule data corresponding to each frequent item set stored in the keyword is updated.

[0056] Traverse each rule item in the rule field, that is, each rule, and count the number of times each rule appears in the state.keys counter hash map structure. After executing this mapScript, the keyword state stores each rule and its number of occurrences after the current document traversal is completed. By finally summarizing the keyword state of all documents, you can know the total number of occurrences of each rule in all documents, that is, their frequency.

[0057] See also Figure 5 Specifically, step 422 includes: Step 4221: In response to the check result of the rule field being not empty, each rule item in the rule field that is not empty is traversed, and a non-empty check is performed on each rule item.

[0058] Check the rule field for non-empty. If the rule field is not empty, enter the next loop, traverse each rule item in the rule field, and check each rule item for non-empty.

[0059] Step 4222: In response to the check result of the rule item being not empty and the corresponding rule item already existing in the keyword, the quantity value of the rule item corresponding to the frequent item set in the keyword is updated.

[0060] If the rule item is not empty, check whether the keyword state contains this rule item. If the rule item already exists in the keyword state, take out the current quantity value of this rule item and add one, then put this rule and the new quantity value back to the keyword state again.

[0061] Or, step 4223: in response to the check result of the rule item being not empty, and the corresponding rule item not existing in the keyword, a rule item corresponding to the frequent item set is added to the keyword, and a quantity value corresponding to the rule item is set.

[0062] If the rule item is not found in the keyword state, it means that the rule item appears for the first time. Then add this item to the keyword state and set the number value of the rule item that appears for the first time to 1.

[0063] After this process, the keyword state stores all the rules in each document and the number of rules corresponding to each rule.

[0064] Step 430: Return the records stored in the data structure and integrate them into hot spot features.

[0065] The combineScript runs on each shard to process the intermediate results of the mapScript stage and form the final result of the shard, which is used to summarize, filter, or perform other operations on the output of the mapScript stage at the shard level.

[0066] Since mapScript is executed once for each shard, the keyword state saves all the rules and their frequencies on the current shard. combineScript is executed once on each shard and returns the intermediate result state.keys of mapScript. Elasticsearch can then collect the state.keys results of all shards globally.

[0067] After the results of the combine phase of all shards are ready, the reduce script reduceScript is run once to form the global final result. Specifically, the combineScript results of all shards are integrated to generate a global aggregate result, including merging, sorting, or performing other statistical and analytical operations on the shard results.

[0068] In Elasticsearch, shards are the basic unit for storing and querying data. When frequent itemsets are stored in Elasticsearch, they are distributed on different shards. In the aggregation analysis phase, Elasticsearch uses the sharding mechanism to process query and aggregation requests in parallel, improving processing efficiency. The data on each shard can be independently operated with mapScript, combineScript, and reduceScript, and finally the reduceScript integrates the results of all shards to form a global final result.

[0069] Through custom mapScript and reduceScript scripts, you can freely set your own data processing logic and result calculation method. In the mapScript script, the original data is processed and converted; in the reduceScript script, the results of the mapScript stage are integrated and the final count result is returned. Custom scripts enable complex processing and calculation of data, not just limited to the aggregation types that come with Elasticsearch.

[0070] Further, see Figure 6 , step 430 includes: Step 431: Return the rule data of each keyword in the first hash mapping structure to a preset second hash mapping structure.

[0071] Create a new second hash map structure and name it finalCount to save the collection of statistical results in all shards.

[0072] Step 432: In response to the new rule data being returned in the second hash mapping structure, the corresponding rule data stored in the second hash mapping structure is updated based on the new rule data.

[0073] In the combineScript phase, the state.keys results of all shards are collected into the states list. The script iterates over each element in the keyword states through the outer loop statement, where each element is a keyword state result, that is, all the rules and their frequencies of a shard. Then, in the inner loop, it iterates over all the rule items in each keyword.

[0074] The reduceScript combines the results of all shards and performs a global frequency count to get the total number of occurrences of each rule in all documents. finalCount is this final result, which is returned as the result of the entire script measurement.

[0075] Further, see Figure 7 , step 432 includes: Step 4321: Traverse the rule data stored in the second hash map structure one by one based on each new rule.

[0076] The script iterates through each element in the keyword states through the outer loop statement, where each element is a keyword state result, that is, all the rules and their frequencies of a shard. Then, in the inner loop, it iterates through all the rule items in each keyword.

[0077] Step 4322: In response to the existence of the same rule item in the second hash mapping structure, the quantity value of the rule item is updated.

[0078] If the rule item is already included in finalCount, add the original frequency to the frequency of the rule item in the current shard and put it back into finalCount.

[0079] Or, step 4323: in response to the absence of the same rule item in the second hash mapping structure, a corresponding rule item is added to the second hash mapping structure, and a quantity value corresponding to the rule item is set.

[0080] If the rule is not included in finalCount, the rule and the frequency of the rule in the current shard are directly put into finalCount.

[0081] Use reduceScript to merge the state.keys results of all shards and calculate the total number of occurrences of each rule in all documents.

[0082] Step 433: Obtain hotspot features in the updated rule data.

[0083] According to the statistical results and business requirements, rule items whose content or quantity values ​​meet the conditions are selected as hot features.

[0084] Using Elasticsearch's scripted metric aggregation query to extract hotspot features is conducive to executing customized complex metrics, making the results of data analysis more intuitive and practical. Through Elasticsearch's distributed and multi-threaded processing capabilities, the efficiency of using scripted aggregation queries to process data will be significantly improved, making it more suitable for scenarios that process large amounts of data.

[0085] Optionally, see Figure 7 , another embodiment of step 400 includes: Step 401: Obtain a preset query method and a preset metric aggregation operation.

[0086] Without any preconditions, we first perform global hotspot feature extraction on the data by executing the Elasticsearch script metric aggregation task. In this step, through the custom script, Elasticsearch will traverse all documents that meet the query conditions, extract and count the hotspot features of each frequent item set corresponding to the document, and generate a global hotspot feature and their frequency list. Users can select one or more hotspot features from the generated hotspot feature list as new preconditions for the next round of script metric aggregation.

[0087] Specifically, initialize the SearchSourceBuilder object to construct the Elasticsearch search request. In this object, you can define the query conditions, sorting, data return fields and other information. Create a ScriptedMetricAggregationBuilder object named scriptedMetricAgg. This object is used to define the scripted metric aggregation operation in Elasticsearch, where initScript defines the initialization script, mapScript is the processing script, combineScript is the shard local merge script, and reduceScript is used to globally merge the processing results.

[0088] Among them, SearchSourceBuilder is the core class used to build search requests in Elasticsearch, which is used to define query conditions, sorting, returned fields, etc.

[0089] For example, users can select the most frequent features or the features that have the greatest impact on the business. Based on the selected hot features as preconditions, the Elasticsearch script metric aggregation is executed again. This time, the script metric aggregation objects are only those documents that contain the selected preconditions (that is, one or more hot features). In this way, hot features can be further extracted and counted under these specific conditions.

[0090] The process of drilling down the characteristic features is to repeat the above two steps continuously. After each round of script measurement aggregation, some new hot features are selected as new preconditions for the next round of script measurement aggregation. Each new round of measurement aggregation will deeply analyze the hot features under more detailed and specific conditions. The extraction and analysis of hot features at different levels and under different conditions realizes the drill-down operation of data analysis, which can deeply explore hot features and meet the data analysis needs at different levels.

[0091] Step 402: Filter frequent itemsets for performing Elasticsearch-based metric aggregation based on preset filtering conditions.

[0092] Call the createQueryBuilder(search) method to create a pre-query. This query can construct an Elasticsearch query that meets specific conditions based on the input search object. These conditions are the pre-filter conditions.

[0093] createQueryBuilder(search) is a custom method that accepts a search criteria object and returns an Elasticsearch query (a query in QueryBuilders).

[0094] Step 403: perform Elasticsearch-based metric aggregation on the filtered frequent itemsets according to the query mode and metric aggregation operation, and obtain the rule items output by Elasticsearch and the quantity values ​​corresponding to each rule item.

[0095] Add the defined script metric aggregation to searchSourceBuilder and close the acquisition of source documents, then set the number of documents returned to 0 to obtain the statistical aggregation results. Use the Elasticsearch client to execute the search request and process the aggregated results of the drill-down analysis.

[0096] For a specific method of executing the metric aggregation, reference may be made to the embodiment of step 400 above.

[0097] Step 404: Select rule items whose corresponding quantity values ​​meet preset conditions as hot spot features.

[0098] Construct an Elasticsearch query based on specific preconditions, and then generate a script metric aggregation. This metric aggregation can count the frequency of occurrence of each specific rule in all documents that meet the preconditions, thereby extracting the most popular (most frequent) features from a set of documents that meet certain conditions.

[0099] The hotspot feature drill-down analysis solution based on Elasticsearch script metric aggregation creates pre-filter conditions based on business needs, and then executes script metric aggregation to extract hotspot features in the data set that meet the conditions. After the first metric aggregation, users can select new hotspot features as pre-conditions and then execute the next round of metric aggregation. This cycle continues until the user's analysis needs are met.

[0100] Through drill-down analysis, data can be analyzed and understood layer by layer from macro to micro, providing in-depth insights into the characteristics, behavioral patterns, and influencing factors of gang activities. By screening and analyzing data associated with specific conditions, such as a hot spot feature, the behavior of the gang can be understood and predicted more accurately, thereby improving the accuracy and efficiency of decisions. Through in-depth understanding and comparison of hot spot features at all levels, business personnel can be guided to formulate or optimize strategies in a targeted manner and improve work efficiency. In the process of in-depth analysis and comparison, some previously overlooked hot spot features or patterns may be discovered, which may provide business personnel with new optimization directions.

[0101] The above describes the hotspot feature extraction method in the embodiment of the present invention. The following describes the hotspot feature extraction device in the embodiment of the present invention. Figure 8 , an embodiment of the hotspot feature extraction device in the embodiment of the present invention includes: The data processing module 501 is used to store and consume business data in real time through a message storage queue. The business data is generated in real time by a website, application or running device.

[0102] The group clustering algorithm module 502 is used to perform clustering consumption on the message storage queue through a clustering algorithm to identify group clustering data in the business data.

[0103] The frequent item mining module 503 is used to perform frequent item mining on the group clustering data to extract frequent item sets in the group clustering data.

[0104] The hotspot feature extraction module 504 is used to store frequent item sets in Elasticsearch and perform metric aggregation based on Elasticsearch to extract hotspot features.

[0105] above Figure 8 The hotspot feature extraction device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the computer device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0106] Fig. 9 : is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. The computer device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the computer device 500.

[0107] The computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Fig. 9 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0108] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the hotspot feature extraction method in the above-mentioned embodiments.

[0109] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the hotspot feature extraction method.

[0110] Different from the prior art, this application uses Elasticsearch script aggregation analysis to extract important hot spot features in real time, quickly and effectively find common features, thereby screening and locating key hot spots, which is conducive to further deepening the understanding of the behavior of related gangs. By setting the query of Elasticsearch, the hot spot feature data is drilled down and analyzed, so that the analysis can be freely switched between different levels to flexibly meet diverse business needs. The hot spot feature extraction method provided by this application can support the real-time processing of large amounts of data, accurately and efficiently complete the extraction of hot spot features, and support flexible definition, which is convenient for expanding application scenarios.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0113] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hotspot feature extraction method, characterized in that: include: Real-time storage and consumption of business data through message storage queues to optimize the processability of the business data, where the business data is generated in real time by websites, applications or running devices; Clustering the message storage queue using a clustering algorithm to identify group clustering data in the business data; Performing frequent item mining on the group clustering data to extract frequent item sets in the group clustering data; Store frequent itemsets in Elasticsearch and perform Elasticsearch-based metric aggregation to extract hotspot features.

2. The hotspot feature extraction method according to claim 1, characterized in that: The Elasticsearch-based metric aggregation is performed to extract hotspot features, including: Create and initialize data structures; Traversing the rule fields of each of the frequent itemsets in the Elasticsearch, and storing the records returned by each of the frequent itemsets during the traversal process in the data structure; The records stored in the data structure are returned and integrated into the hot spot features.

3. The hotspot feature extraction method according to claim 2, characterized in that: The creating and initializing data structure includes: Creating and initializing a first hash mapping structure, setting keywords based on the first hash mapping structure, each of the keywords corresponding to a plurality of the frequent item sets; The traversing the rule field of each of the frequent itemsets in the Elasticsearch and storing the records returned by each of the frequent itemsets during the traversal in the data structure includes: Traversing the rule fields of each of the frequent itemsets, and performing a non-empty check on each of the rule fields; According to the inspection results returned by the rule fields, the rule data corresponding to the frequent itemsets stored in the keyword are updated.

4. The hotspot feature extraction method according to claim 3, characterized in that: The updating of the rule data corresponding to each of the frequent itemsets stored in the keyword according to the inspection results returned by each of the rule fields includes: In response to the check result of the rule field being not empty, traversing each rule item in the rule field that is not empty, and performing a non-empty check on each rule item; In response to the check result of the rule item being not empty and the corresponding rule item already exists in the keyword, updating the quantity value of the rule item corresponding to the frequent item set in the keyword; Or, in response to the check result of the rule item being not empty and the corresponding rule item not existing in the keyword, the rule item corresponding to the frequent item set is added to the keyword, and the quantity value corresponding to the rule item is set.

5. The hotspot feature extraction method according to claim 3, characterized in that: The returning the records stored in the data structure and integrating them into the hot spot features includes: Returning the rule data of each of the keywords in the first hash mapping structure to a preset second hash mapping structure; In response to returning new rule data in the second hash mapping structure, updating corresponding rule data stored in the second hash mapping structure based on the new rule data; The hotspot feature is obtained from the updated rule data.

6. The hotspot feature extraction method according to claim 5, characterized in that: The updating of the corresponding rule data stored in the second hash map structure based on the new rule data includes: Traversing the rule data stored in the second hash map structure one by one based on each new rule; In response to the existence of the same rule item in the second hash mapping structure, updating the quantity value of the rule item; Or, in response to the absence of the same rule item in the second hash mapping structure, the corresponding rule item is added to the second hash mapping structure, and the quantity value corresponding to the rule item is set.

7. The hotspot feature extraction method according to any one of claims 1 to 6, characterized in that: The method of storing frequent itemsets in Elasticsearch and performing metric aggregation based on Elasticsearch to extract hotspot features also includes: Get the preset query method and preset metric aggregation operation; Filtering the frequent itemsets for performing metric aggregation based on the Elasticsearch based on a preset filtering condition; Performing Elasticsearch-based metric aggregation on the filtered frequent item sets according to the query mode and the metric aggregation operation, and obtaining rule items output by Elasticsearch and quantity values ​​corresponding to each rule item; The rule item whose corresponding quantity value meets the preset condition is selected as the hot spot feature.

8. A hotspot feature extraction device, characterized in that: The hotspot feature extraction device comprises: A data processing module is used to store and consume business data in real time through a message storage queue. The business data is generated in real time by a website, application or running device; A group clustering algorithm module, used for clustering consumption of the message storage queue by using a clustering algorithm to identify group clustering data in the business data; A frequent item mining module, used for performing frequent item mining on the group clustering data to extract frequent item sets in the group clustering data; The hotspot feature extraction module is used to store frequent itemsets in Elasticsearch and perform Elasticsearch-based metric aggregation to extract hotspot features.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the hotspot feature extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the hotspot feature extraction method according to any one of claims 1 to 7 is implemented.

Images