Intelligence analysis method, device, equipment and readable storage medium

Abstract

The present application discloses an intelligence analysis method, apparatus, device and readable storage medium. The method includes: obtaining a database access link, caching the intelligence information in the corresponding database in the memory according to the database access link; the intelligence information includes the intelligence model name, field and intelligence data; filtering the source data in the source database according to the intelligence model name and field in the memory; colliding the source data in the filtered source database with the intelligence data in the memory, and if a collision occurs, outputting the collided intelligence data and the corresponding data in the source database. The technical solution disclosed in the present application improves the efficiency of intelligence analysis, reduces the delay of intelligence analysis, and improves the performance of intelligence analysis by caching the intelligence information in the database in the memory and performing intelligence analysis based on the intelligence information cached in the memory, and filtering the source data in the source database according to the intelligence model name and field cached in the memory.

Description

Technical Field

[0001] The present application relates to the field of intelligence analysis technology, and more specifically, to an intelligence analysis method, apparatus, device, and readable storage medium. Background Art

[0002] Currently, when performing intelligence data analysis in massive data scenarios, a request is sent to a database containing information such as the name of the stored intelligence model for each data source, so that the data source is collided with the information stored in the database to achieve intelligence analysis. However, due to the huge amount of data in the data source, when performing intelligence analysis on these data sources, requests will be frequently sent to the database, which will consume a lot of IO and time, putting considerable pressure on the database.

[0003] In summary, how to improve intelligence analysis performance is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an intelligence analysis method, apparatus, device and readable storage medium for improving intelligence analysis performance.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] An intelligence analysis method comprising:

[0007] Obtaining a database access link, and caching intelligence information in the corresponding database in a memory according to the database access link; the intelligence information includes an intelligence model name, fields, and intelligence data;

[0008] Filtering source data in the source database according to the intelligence model name and field in the memory;

[0009] The source data in the filtered source database is collided with the intelligence data in the memory. If a collision occurs, the collided intelligence data and the corresponding data in the source database are output.

[0010] Preferably, before colliding the filtered source data in the source database with the intelligence data in the memory, the method further includes:

[0011] It is determined whether the data value corresponding to the field in the filtered source database is empty. If so, the source data corresponding to the data value is deleted to obtain the remaining data.

[0012] Preferably, after deleting the source data corresponding to the data value, the method further includes:

[0013] storing the remaining data in a cuckoo filter;

[0014] The source data in the filtered source database is collided with the intelligence data in the memory, including:

[0015] The remaining data is collided with the intelligence data in the memory using the cuckoo filter.

[0016] Preferably, it also includes:

[0017] If no collision occurs, the source data in the filtered source database is collided with the intelligence data in the database.

[0018] Preferably, obtaining a database access link and caching intelligence information in a corresponding database in a memory according to the database access link includes:

[0019] Obtain a redis access link, and cache the corresponding intelligence information in redis in the memory according to the redis access link.

[0020] Preferably, it also includes:

[0021] Logs are collected in advance, and data in the logs are parsed into a JSON format to obtain the source data, and the source data is stored in the source database.

[0022] An intelligence analysis device, comprising:

[0023] An acquisition module is used to obtain a database access link and cache intelligence information in the corresponding database in the memory according to the database access link; the intelligence information includes an intelligence model name, fields and intelligence data;

[0024] A filtering module, configured to filter source data in a source database according to the intelligence model name and field in the memory;

[0025] The first collision module is used to collide the source data in the filtered source database with the intelligence data in the memory, and if a collision occurs, output the collided intelligence data and the corresponding data in the source database.

[0026] Preferably, it also includes:

[0027] The deletion module is used to determine whether the data value corresponding to the field in the filtered source database is empty before colliding the source data in the filtered source database with the intelligence data in the memory. If so, the source data corresponding to the data value is deleted to obtain the remaining data.

[0028] An intelligence analysis device comprising:

[0029] Memory for storing computer programs;

[0030] A processor is configured to implement the steps of any of the above-mentioned intelligence analysis methods when executing the computer program.

[0031] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned intelligence analysis methods.

[0032] The present application provides an intelligence analysis method, apparatus, device and readable storage medium, wherein the method includes: obtaining a database access link, and caching the intelligence information in the corresponding database in the memory according to the database access link; the intelligence information includes the intelligence model name, field and intelligence data; filtering the source data in the source database according to the intelligence model name and field in the memory; colliding the source data in the filtered source database with the intelligence data in the memory, and if a collision occurs, outputting the collided intelligence data and the corresponding data in the source database.

[0033] The above-mentioned technical solution disclosed in the present application caches the intelligence information in the database in the memory, and performs intelligence analysis on the source data in the source database based on the intelligence information cached in the memory, without having to read the intelligence information from the database every time. Therefore, the efficiency of intelligence analysis can be improved, the delay of intelligence analysis can be reduced, and the performance of intelligence analysis can be improved. In addition, the present application reduces the amount of data in the source database by filtering the source data in the source database according to the intelligence model name and field cached in the memory when performing intelligence analysis, thereby shortening the intelligence analysis time, reducing the delay of intelligence analysis, and improving the performance of intelligence analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0035] Figure 1 A flowchart of an intelligence analysis method provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the structure of an intelligence analysis device provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of the structure of an intelligence analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The core of this application is to provide an intelligence analysis method, device, equipment and readable storage medium for improving intelligence analysis performance.

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] See also Figure 1 , which shows a flow chart of an intelligence analysis method provided by an embodiment of the present application. The intelligence analysis method provided by an embodiment of the present application may include:

[0041] S11: Obtain a database access link, and cache the intelligence information in the corresponding database in the memory according to the database access link; the intelligence information includes the intelligence model name, fields, and intelligence data.

[0042] When performing intelligence analysis, the intelligence analysis engine can be initialized when it is started. Specifically, the database access link, data dictionary and other information corresponding to the database used to store the intelligence information used in the intelligence analysis can be obtained to prepare for subsequent steps.

[0043] On the basis of the above, the database corresponding to the database access link can be accessed according to the database access link obtained during the initialization process, and all the intelligence information in the corresponding database can be cached in the memory, wherein the intelligence information mentioned here specifically includes the intelligence model name, field (specifically, the field to be collided with intelligence) and the intelligence data corresponding to the intelligence model. It should be noted that when caching intelligence information in the memory, it can be cached.<ruleid+key,jsondata> , where ruleid is the name of the intelligence model, key is the field to be collided with intelligence, and jsondata is the intelligence data.

[0044] By caching all intelligence information in the database in memory, there is no need to send requests to the corresponding database every time intelligence analysis is performed, thereby reducing IO consumption and reducing the pressure on the database for intelligence analysis. In addition, by caching intelligence information in memory, the intelligence information can be read quickly and efficiently, thereby reducing the delay of intelligence analysis and improving intelligence analysis performance.

[0045] S12: Filter the source data in the source database according to the intelligence model name and field in the memory.

[0046] Based on step S11, the source data in the source database can be filtered according to the intelligence model name and field in the intelligence information cached in the memory. The source data mentioned here specifically refers to the data to be analyzed for intelligence, and the source of these source data can be from various types of data, such as audit data, traffic, session data, etc. The source database is the database that stores the source data. By storing the source data in the source database, it is convenient to analyze each source data together, thereby improving the convenience of intelligence analysis of the source data.

[0047] When filtering, the intelligence model names in the source data stored in the source database can be compared with the intelligence model names cached in the memory. If there is an intelligence model name corresponding to the intelligence model name in the source database in the memory, the intelligence model name in the source database is retained; if the source data in the source database contains one or some intelligence model names, and there is no corresponding intelligence model name in the memory, the above-mentioned intelligence model name in the source database (that is, there is no corresponding intelligence model name in the memory) and the corresponding source data can be filtered out. That is, if there is no one or some intelligence model name in the memory but these intelligence model names are in the source database, it means that there is no data in the memory that matches these intelligence models in the source database. Therefore, the above-mentioned intelligence models in the source database can be filtered out to reduce the workload of intelligence analysis.

[0048] Based on the above, the source data in the filtered source database can be filtered again according to the fields cached in the memory (specifically, according to the names of the fields) (i.e., a second filtering). If there is a field in the memory that corresponds to a field in the source database, the field in the source database and its corresponding source data are retained. If there is no field in the memory that corresponds to a field in the source database, the field in the source database and its corresponding source data are deleted. This reduces the amount of source data in the source database through filtering, thereby reducing the workload of intelligence analysis and improving the efficiency of intelligence analysis.

[0049] S13: The source data in the filtered source database is collided with the intelligence data in the memory. If a collision occurs, the collided intelligence data and the corresponding data in the source database are output.

[0050] After the above two filterings, the source data in the filtered source database (that is, the remaining source data in the filtered source database) can be collided with the intelligence data cached in the memory (specifically, it can be collided with the intelligence data in the cache in the memory). If the source data in the filtered source database collides with the intelligence data cached in the memory, the collided intelligence data and the data corresponding to the collided intelligence data in the source database are output. In addition, the intelligence model corresponding to the collided intelligence data can also be output, so that relevant personnel can promptly obtain the intelligence analysis results based on the output results.

[0051] The above-mentioned technical solution disclosed in the present application caches the intelligence information in the database in the memory, and performs intelligence analysis on the source data in the source database based on the intelligence information cached in the memory, without having to read the intelligence information from the database every time. Therefore, the efficiency of intelligence analysis can be improved, the delay of intelligence analysis can be reduced, and the performance of intelligence analysis can be improved. In addition, the present application reduces the amount of data in the source database by filtering the source data in the source database according to the intelligence model name and field cached in the memory when performing intelligence analysis, thereby shortening the intelligence analysis time, reducing the delay of intelligence analysis, and improving the performance of intelligence analysis.

[0052] An intelligence analysis method provided in an embodiment of the present application may further include, before colliding the filtered source data in the source database with the intelligence data in the memory:

[0053] Determine whether the data value corresponding to the field in the filtered source database is empty. If so, delete the source data corresponding to the data value to obtain the remaining data.

[0054] In the present application, before colliding the source data in the filtered source database with the intelligence data in the memory, the source data in the source database may be filtered a third time. Specifically, after the two filtering steps, it is determined whether the data value in the filtered source database corresponding to the field cached in the memory is empty. In other words, it is determined whether the data value of the field to be collided with intelligence in the remaining source data in the filtered source database is empty. If the data value is not empty, the source data is retained. If the data value is empty, the source data corresponding to the data value is deleted to obtain the remaining data.

[0055] The third filtering can further reduce the amount of data in the source database and avoid collisions with source data whose values are empty, thereby improving intelligence analysis efficiency, reducing intelligence analysis delays, and improving intelligence analysis performance.

[0056] An intelligence analysis method provided in an embodiment of the present application may further include, after deleting the source data corresponding to the data value:

[0057] Save the remaining data in a cuckoo filter;

[0058] Colliding the filtered source data in the source database with the intelligence data in memory can include:

[0059] Use the cuckoo filter to collide the remaining data with the intelligence data in memory.

[0060] In the present application, after the remaining data in the source database is obtained through three filtrations, the remaining data can be stored in a cuckoo filter, that is, the results after three filtrations in the source database are stored in the cuckoo filter.

[0061] The cuckoo filter is derived from the cuckoo hash algorithm. The cuckoo hash table consists of two tables, each with its own hash function. When new data is inserted, it calculates the corresponding locations in the two tables. The data will be stored in one of these two locations (Table 1 or Table 2). Once a location in one table is found to be occupied, the original data in that location is removed, and the removed data is then sent to the other table to find its corresponding location. By continuously removing data, all data eventually finds its destination. However, data may continue to be removed, eventually forming a loop. If one data item cannot find its destination, this indicates that the cuckoo hash table has reached its limit and requires optimization of the hash algorithm or expansion of the hash table. The cuckoo filter only stores the element's fingerprint information (a few bits, similar to a Bloom filter). Since it does not store all the data information, misjudgment is possible. Because the Cuckoo Filter needs to recalculate the hash value of the original data in another table when rejecting data, the hash algorithm is designed to combine two hash functions into one. The candidate position in the first table is Hash(x), and the candidate position in the second table is Hash(x) ⊕ hash(fingerprint(x)). Specifically, the position in the first table is XORed with the hash value of the stored fingerprint. This allows the fingerprint value to be XORed with the hash value of the original position to calculate the position in the other table. Therefore, the Cuckoo Filter can perform hash operations and perform collisions based on the hash values obtained from these operations, thereby improving the efficiency and performance of intelligence analysis.

[0062] On the basis of storing the remaining data in the cuckoo filter, when colliding the source data in the filtered source database with the intelligence data in the memory, the cuckoo filter can be used to collide the remaining data with the intelligence data in the memory. Specifically, the cuckoo filter can be used to perform a hash operation on the remaining data to obtain a hash value, and then the obtained hash value can be used to collide with the intelligence data cached in the memory, thereby improving the collision efficiency.

[0063] It should be noted that when using the Cuckoo filter for intelligence analysis, the intelligence information in the corresponding database is cached in the memory. The intelligence model name (ruleid) and field in the corresponding database can also be loaded into the Cuckoo filter, and a Map is returned after encapsulation (it is a function in the code, which stores key and value type data, and the Map is cached in the memory). Its type is Map <String,Tuple2<CuckooFilter <charsequence>,String>>, where the Map key is ruleid, Tuple2 contains the name of the Cuckoo filter and the field to be matched, and String represents a string. Accordingly, the three filters mentioned above can be implemented based on the Map cached in memory and the Cuckoo filter.

[0064] An intelligence analysis method provided in an embodiment of the present application may further include:

[0065] If no collision occurs, the source data in the filtered source database will be collided with the intelligence data in the database.

[0066] In the present application, when the source data in the filtered source database is collided with the intelligence data in the memory, if the source data in the filtered source database does not collide with the intelligence data cached in the memory, the source data in the filtered source database can be collided with the intelligence data in the database, instead of colliding based on the intelligence data cached in the memory, so as to improve the reliability and accuracy of intelligence analysis.

[0067] An intelligence analysis method provided in an embodiment of the present application, which obtains a database access link and caches intelligence information in a corresponding database in a memory according to the database access link, may include:

[0068] Get the redis access link and cache the corresponding intelligence information in redis in memory according to the redis access link.

[0069] In this application, redis can be used to store intelligence information used in intelligence analysis. Accordingly, a redis access link can be obtained, and the corresponding intelligence information in redis can be cached in memory according to the redis access link.

[0070] Since Redis stores data in the form of key-value, storing intelligence information in Redis can facilitate the intelligence analysis process.

[0071] In addition, while storing intelligence information in Redis, it can also be stored in Elasticsearch to facilitate subsequent interface queries and tracing.

[0072] An intelligence analysis method provided in an embodiment of the present application may further include:

[0073] Collect logs in advance, parse the data in the logs into JSON format to obtain source data, and store the source data in the source database.

[0074] In this application, logs to be analyzed for intelligence (such as syslog) can be collected in advance, and the data in the logs can be parsed into a standard json format (of course, the data can also be parsed into other data formats) to obtain the source data, and the source data can be stored in the source database to facilitate intelligence analysis of the data in the source database.

[0075] The present application also provides an intelligence analysis device. Figure 2 , which shows a schematic structural diagram of an intelligence analysis device provided in an embodiment of the present application, which may include:

[0076] The acquisition module 21 is used to obtain a database access link and cache the intelligence information in the corresponding database in the memory according to the database access link; the intelligence information may include the intelligence model name, field and intelligence data;

[0077] The filtering module 22 is used to filter the source data in the source database according to the intelligence model name and field in the memory;

[0078] The first collision module 23 is used to collide the filtered source data in the source database with the intelligence data in the memory, and if a collision occurs, output the collided intelligence data and the corresponding data in the source database.

[0079] An intelligence analysis device provided in an embodiment of the present application may further include:

[0080] The deletion module is used to determine whether the data value corresponding to the field in the filtered source database is empty before colliding the source data in the filtered source database with the intelligence data in the memory. If so, the source data corresponding to the data value is deleted to obtain the remaining data.

[0081] An intelligence analysis device provided in an embodiment of the present application may further include:

[0082] A saving module is used to save the remaining data in the cuckoo filter after deleting the source data corresponding to the data value;

[0083] The first collision module 23 may include:

[0084] The collision unit is used to collide the remaining data with the intelligence data in the memory using the Cuckoo filter.

[0085] An intelligence analysis device provided in an embodiment of the present application may further include:

[0086] The second collision module is used to collide the source data in the filtered source database with the intelligence data in the database if no collision occurs.

[0087] An intelligence analysis device provided in an embodiment of the present application may include an acquisition module 21 that:

[0088] The acquisition unit is used to obtain the redis access link and cache the corresponding intelligence information in the redis in the memory according to the redis access link.

[0089] An intelligence analysis device provided in an embodiment of the present application may further include:

[0090] The collection module is used to collect logs in advance, parse the data in the logs into JSON format to obtain source data, and store the source data in the source database.

[0091] The present application also provides an intelligence analysis device. Figure 3 , which shows a schematic structural diagram of an intelligence analysis device provided in an embodiment of the present application, which may include:

[0092] Memory 31, for storing computer programs;

[0093] The processor 32 is configured to execute the computer program stored in the memory 31 and implement the following steps:

[0094] Obtain a database access link and cache the intelligence information in the corresponding database in memory according to the database access link; the intelligence information includes the intelligence model name, fields and intelligence data; filter the source data in the source database according to the intelligence model name and fields in memory; collide the source data in the filtered source database with the intelligence data in memory, and if a collision occurs, output the collided intelligence data and the corresponding data in the source database.

[0095] The present application also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the following steps can be implemented:

[0096] Obtain a database access link and cache the intelligence information in the corresponding database in memory according to the database access link; the intelligence information includes the intelligence model name, fields and intelligence data; filter the source data in the source database according to the intelligence model name and fields in memory; collide the source data in the filtered source database with the intelligence data in memory, and if a collision occurs, output the collided intelligence data and the corresponding data in the source database.

[0097] The readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0098] For the description of the relevant parts of the intelligence analysis device, equipment and readable storage medium provided in this application, please refer to the detailed description of the corresponding parts of the intelligence analysis method provided in the embodiment of this application, and will not be repeated here.

[0099] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device that includes a series of elements is inherent to the elements. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements. In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0100] The above description of the disclosed embodiments will enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.< / charsequence>

Claims

1. An intelligence analysis method, characterized in that: include: Obtaining a database access link, and caching intelligence information in the corresponding database in a memory according to the database access link; the intelligence information includes an intelligence model name, fields, and intelligence data; Filtering source data in the source database according to the intelligence model name and field in the memory; The source data in the filtered source database is collided with the intelligence data in the memory. If a collision occurs, the collided intelligence data and the corresponding data in the source database are output; wherein: The filtering of the source data in the source database according to the intelligence model name and field in the memory includes: The intelligence model names in the source data stored in the source database are compared with the intelligence model names cached in the memory. If the memory contains an intelligence model name corresponding to the intelligence model name in the source database, the intelligence model name in the source database and the corresponding source data are retained; if the source data in the source database contains one or more intelligence model names, and the memory does not contain the corresponding intelligence model name, the intelligence model name in the source database and the corresponding source data are filtered out; the source data in the source database filtered by the intelligence model name is filtered again according to the fields cached in the memory. If the memory contains a field corresponding to the field in the source database filtered by the intelligence model name, the field in the source database and the corresponding source data are retained. If the memory does not contain a field corresponding to the field in the source database filtered by the intelligence model name, the field in the source database and the corresponding source data are deleted.

2. The intelligence analysis method according to claim 1, characterized in that: Before colliding the filtered source data in the source database with the intelligence data in the memory, the method further includes: It is determined whether the data value corresponding to the field in the filtered source database is empty. If so, the source data corresponding to the data value is deleted to obtain the remaining data.

3. The intelligence analysis method according to claim 2, characterized in that: After deleting the source data corresponding to the data value, the method further includes: storing the remaining data in a cuckoo filter; The source data in the filtered source database is collided with the intelligence data in the memory, including: The remaining data is collided with the intelligence data in the memory using the cuckoo filter.

4. The intelligence analysis method according to any one of claims 1 to 3, characterized in that: Also includes: If no collision occurs, the source data in the filtered source database is collided with the intelligence data in the database.

5. The intelligence analysis method according to claim 4, characterized in that: Obtaining a database access link and caching intelligence information in a corresponding database in memory according to the database access link, including: Obtain a redis access link, and cache the corresponding intelligence information in redis in the memory according to the redis access link.

6. The intelligence analysis method according to claim 4, characterized in that: Also includes: Logs are collected in advance, and data in the logs are parsed into a JSON format to obtain the source data, and the source data is stored in the source database.

7. An intelligence analysis device, characterized in that: include: An acquisition module is used to obtain a database access link and cache intelligence information in the corresponding database in the memory according to the database access link; the intelligence information includes an intelligence model name, fields and intelligence data; A filtering module, configured to filter source data in a source database according to the intelligence model name and field in the memory; The first collision module is used to collide the source data in the filtered source database with the intelligence data in the memory, and if a collision occurs, output the collided intelligence data and the corresponding data in the source database, wherein: The filtering module is specifically configured to compare the intelligence model name in the source data stored in the source database with the intelligence model name cached in the memory, and if the memory contains an intelligence model name corresponding to the intelligence model name in the source database, retain the intelligence model name in the source database and the corresponding source data; If the source data in the source database contains one or more intelligence model names, and there is no corresponding intelligence model name in the memory, the intelligence model name in the source database and the corresponding source data are filtered out; the source data in the source database after being filtered by the intelligence model name is filtered again according to the fields cached in the memory; if there is a field in the memory corresponding to the field in the source database after being filtered by the intelligence model name, the field in the source database and the corresponding source data are retained; if there is no field in the memory corresponding to the field in the source database after being filtered by the intelligence model name, the field in the source database and the corresponding source data are deleted.

8. The information analysis device according to claim 7, characterized in that Also includes: The deletion module is used to determine whether the data value corresponding to the field in the filtered source database is empty before colliding the source data in the filtered source database with the intelligence data in the memory. If so, the source data corresponding to the data value is deleted to obtain the remaining data.

9. An intelligence analysis device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the intelligence analysis method according to any one of claims 1 to 6 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the intelligence analysis method according to any one of claims 1 to 6 are implemented.

Images