Data processing method and apparatus in intelligence collision process, and device

Abstract

The present application relates to the technical field of security, and discloses a data processing method and apparatus in an intelligence collision process, and a device, capable of obtaining different intelligence information by means of matching in a data processing process in the intelligence collision process, thereby improving the accuracy of security detection. The method comprises: obtaining a historical collision result collected by a local cache cluster, the historical collision result recording hit information of intelligence data on different parameter fields in a historical collision verification process; in response to an intelligence collision instruction and on the basis of the historical collision result, querying whether collision-pending intelligence data has undergone collision; when said intelligence data has not undergone collision, calling an intelligence interface to perform collision verification on different parameter fields in said intelligence data, to obtain hit information of the different parameter fields in said intelligence data; and updating the hit information of the different parameter fields in said intelligence data to the local cache cluster as a collision result of the intelligence data.

Description

Data processing method, device and equipment during intelligence collision process

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 13, 2023, with application number 202311182371.1 and application name “Data processing method, device and equipment in intelligence collision process”, the entire contents of which are incorporated by reference into the application. Technical Field

[0002] The present application relates to the field of security technology, and in particular to a method, device and equipment for processing data during an intelligence collision process. Background Art

[0003] With the continuous development of internet technology and new lifestyles, massive amounts of data have been generated. With the continuous integration of networks, data, and the public, big data technology has emerged and is widely used in intelligence management. In this field, data management tools can be used to collect various intelligence. Intelligence collision services can then compare local intelligence with known threat intelligence libraries, effectively identifying potential threats hidden in local intelligence. This helps users promptly identify and respond to various network threats, improving their ability to prevent cybersecurity.

[0004] In related technologies, intelligence collision services use raw data to directly collide and compare with the original intelligence interface. The collision function operator combines with other rule operators to detect anomalies and generate alerts when anomaly rules are met. However, when the data volume is large, the intelligence collision service generates a large number of request interface calls. Frequent request interface calls increase the response latency of the intelligence collision service, lengthening the interface call time, resulting in a large data backlog and affecting the data processing capacity during the intelligence collision process.

[0005] Application Contents

[0006] In view of this, the present application provides a data processing method, device and equipment during the intelligence collision process. The main purpose is to solve the problem that the interface call time in the intelligence collision service process is prolonged in the existing technology, resulting in a large amount of data backlog, affecting the data processing capability during the intelligence collision process.

[0007] According to a first aspect of the present application, a method for processing data in an intelligence collision process is provided, comprising:

[0008] Obtaining historical collision results collected by the local cache cluster, wherein the historical collision results record hit information of intelligence data on different parameter fields during the historical collision verification process;

[0009] In response to the intelligence collision instruction, querying the pending collision intelligence data based on the historical collision results to determine whether the collision has occurred;

[0010] If the pending collision intelligence data has not experienced a collision, calling an intelligence interface to perform collision verification on different parameter fields in the pending collision intelligence data to obtain hit information of different parameter fields in the pending collision intelligence data;

[0011] The hit information of different parameter fields in the intelligence data to be collided is used as the collision result of the intelligence data and updated to the local cache cluster.

[0012] Furthermore, before obtaining the historical collision results collected by the local cache cluster, the method further includes:

[0013] Start cache preheating monitoring, traverse and read the original data to be collided, assemble the intelligence data that has been collided in the original data to be collided, and insert it into the local cache cluster. The original data to be collided is the original data extracted for real-time consumption.

[0014] Furthermore, the starting of cache preheating monitoring, traversing and reading the original data to be collided, assembling the intelligence data that has been collided in the original data to be collided and inserting it into the local cache cluster, includes:

[0015] Start cache preheating monitoring, traverse and read the original data to be collided, and put the original data to be collided into the first queue;

[0016] Extracting collision elements from the raw data to be collided by a parsing thread, performing collision preprocessing on the collision elements, and assembling the unhit collision elements into an intelligence collection object and placing it into a second queue;

[0017] Pull the intelligence collection object from the second queue through the collision thread, use the intelligence collection object as the data to be collided to batch call the collision interface to obtain the collision result of the data to be collided, and store the collision result in the local cache cluster.

[0018] Furthermore, in response to the intelligence collision instruction, querying the pending collision intelligence data based on the historical collision results to determine whether the collision has occurred includes:

[0019] In response to the intelligence collision instruction, extracting the pending collision intelligence data from the raw data according to a preset configuration;

[0020] parsing the valid parameter fields in the pending collision intelligence data, and querying whether there is hit information of the valid parameter fields in the historical collision results according to the valid parameter fields;

[0021] If so, it is determined that the pending collision intelligence data has undergone a collision; otherwise, it is determined that the pending collision intelligence data has not undergone a collision.

[0022] Furthermore, after responding to the intelligence collision instruction and querying the potential collision intelligence data based on the historical collision results to determine whether a collision has occurred, the method further includes:

[0023] If the pending collision intelligence data has experienced a collision, then extracting hit information of the valid parameter field from the historical cache result according to the valid parameter field in the pending collision intelligence data;

[0024] The collision result of the intelligence data is determined according to the hit information of the valid parameter field.

[0025] Furthermore, after updating the hit information of different parameter fields in the intelligence data to be collided as the collision result of the intelligence data into the local cache cluster, the method further includes:

[0026] Determine whether the parameter field hit by the intelligence data satisfies the alarm detection rule based on the historical collision results in the local cache cluster;

[0027] If so, an alarm message is generated based on the parameter fields that meet the intelligence hit in the alarm detection rules.

[0028] Furthermore, the alarm detection rule records at least one intelligence hit judgment logic, and the judgment of whether the parameter field hit by the intelligence data satisfies the alarm detection rule based on the historical collision results in the local cache cluster specifically includes:

[0029] Determining the parameter fields hit during the historical collision verification process based on the historical collision results in the local cache cluster;

[0030] The judgment logic of the at least one intelligence hit is used to determine whether the parameter field hit during the historical collision verification process meets the alarm detection rules.

[0031] According to a second aspect of the present application, there is provided a data processing device for an intelligence collision process, comprising:

[0032] An acquisition unit is used to acquire historical collision results collected by a local cache cluster, wherein the historical collision results record hit information of intelligence data on different parameter fields during the historical collision verification process;

[0033] A query unit, configured to respond to an intelligence collision instruction and query, based on the historical collision results, whether the pending collision intelligence data has experienced a collision;

[0034] a verification unit, configured to, if the pending collision intelligence data has not experienced a collision, call an intelligence interface to perform collision verification on different parameter fields in the pending collision intelligence data, and obtain hit information of different parameter fields in the pending collision intelligence data;

[0035] The updating unit is configured to update the hit information of different parameter fields in the intelligence data to be collided into the local cache cluster as the collision result of the intelligence data.

[0036] Furthermore, the device further comprises:

[0037] The starting unit is used to start cache preheating monitoring before obtaining the historical collision results collected by the local cache cluster, traverse and read the original data to be collided, assemble the intelligence data that has been collided in the original data to be collided, and insert it into the local cache cluster. The original data to be collided is the original data extracted for real-time consumption.

[0038] Furthermore, the startup unit is specifically used to start cache preheating monitoring, traverse and read the original data to be collided, and put the original data to be collided into the first queue; extract collision elements from the original data to be collided through the parsing thread, and after the collision preprocessing operation on the collision elements, assemble the missed collision elements into an intelligence collection object and put it into the second queue; pull the intelligence collection object from the second queue through the collision thread, use the intelligence collection object as the data to be collided to batch call the collision interface to obtain the collision results of the data to be collided, and store the collision results in the local cache cluster.

[0039] Furthermore, the query unit is specifically used to respond to the intelligence collision instruction and extract the intelligence data to be collided from the original data according to the preset configuration; parse the valid parameter field in the intelligence data to be collided, and query whether there is hit information of the valid parameter field in the historical collision results based on the valid parameter field; if so, it is determined that the intelligence data to be collided has undergone a collision; otherwise, it is determined that the intelligence data to be collided has not undergone a collision.

[0040] Furthermore, the device further comprises:

[0041] an extraction unit configured to, after querying, in response to the intelligence collision instruction, whether the pending collision intelligence data has undergone a collision based on the historical collision results, extract, if the pending collision intelligence data has undergone a collision, hit information of a valid parameter field from the historical cache results based on a valid parameter field in the pending collision intelligence data;

[0042] The determination unit is used to determine the collision result of the intelligence data according to the hit information of the valid parameter field.

[0043] Furthermore, the device further comprises:

[0044] a judgment unit, configured to, after updating the hit information of different parameter fields in the intelligence data to be collided as the collision result of the intelligence data into the local cache cluster, judge whether the parameter field hit by the intelligence data satisfies the alarm detection rule based on the historical collision results in the local cache cluster;

[0045] The generating unit is used to generate alarm information according to the parameter field of the intelligence hit in the alarm detection rule.

[0046] Furthermore, the alarm detection rule records at least one intelligence hit judgment logic, and the judgment unit is specifically used to determine the parameter field hit in the historical collision verification process based on the historical collision results in the local cache cluster; and judge whether the parameter field hit in the historical collision verification process meets the alarm detection rule through the at least one intelligence hit judgment logic.

[0047] According to a third aspect of the present application, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0048] According to a fourth aspect of the present application, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0049] By means of the above technical solution, the present application provides a data processing method, device and equipment in the process of intelligence collision. Compared with the existing method of frequently calling the intelligence system for data processing using the intelligence collision service in the existing technology, the present application obtains the historical collision results collected by the local cache cluster. The historical collision results record the hit information of the intelligence data on different parameter fields during the historical collision verification process. In response to the intelligence collision instruction, the historical collision results are used to query whether the intelligence data to be collided has been collided. If the intelligence data to be collided has not been collided, the intelligence interface is called to perform collision verification on different parameter fields in the intelligence data to be collided, and the hit information of different parameter fields in the intelligence data to be collided is obtained. Finally, the hit information of different parameter fields in the intelligence data to be collided is updated to the local cache cluster as the collision result of the intelligence data. The whole process does not rely on the collision interface of the intelligence system, uses the historical collision results to pre-judge the intelligence collision of the intelligence data to be collided, reduces the number of calls to the external intelligence system, and reduces the resource usage of the intelligence system. If the intelligence data to be collided has not been collided, the intelligence interface is called for collision verification, shortens the interface request call time of the intelligence service, avoids data backlog, and improves the data processing capability during the intelligence collision process.

[0050] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] FIG1 is a flow chart of a data processing method in an intelligence collision process according to an embodiment of the present application;

[0053] FIG2 is a flow chart of a data processing method in an intelligence collision process according to another embodiment of the present application;

[0054] FIG3 is a flow chart of a specific implementation of step 202 in FIG2 ;

[0055] FIG4 is a flow chart of a specific implementation of step 102 in FIG1 ;

[0056] FIG5 is a flow chart of a data processing method in an intelligence collision process according to another embodiment of the present application;

[0057] FIG6 is a flow chart of a data processing method in an intelligence collision process according to another embodiment of the present application;

[0058] FIG7 is a framework diagram of data processing during an intelligence collision process according to an embodiment of the present application;

[0059] FIG8 is a flow chart of data processing during an intelligence collision process according to an embodiment of the present application;

[0060] FIG9 is a structural diagram of a data processing device during an intelligence collision process according to an embodiment of the present application;

[0061] FIG10 is a schematic diagram of the device structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present application will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the present application, rather than to imply any limitation on the scope of the present application.

[0063] As used herein, the term "including" and variations thereof are to be interpreted as open-ended terms meaning "including, but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The terms "one embodiment" and "an embodiment" are to be interpreted as meaning "at least one embodiment." The term "another embodiment" is to be interpreted as meaning "at least one other embodiment."

[0064] In related technologies, intelligence collision services use raw data to directly collide and compare with the original intelligence interface. The collision function operator combines with other rule operators to detect anomalies and generate alerts when anomaly rules are met. However, when the data volume is large, the intelligence collision service generates a large number of request interface calls. Frequent request interface calls increase the response latency of the intelligence collision service, lengthening the interface call time, resulting in a large data backlog and affecting the data processing capacity during the intelligence collision process.

[0065] To solve this problem, this embodiment provides a data processing method during an intelligence collision process. As shown in FIG1 , the method is applied to a server corresponding to an intelligence collision service and includes the following steps:

[0066] 101. Obtain historical collision results collected by the local cache cluster.

[0067] The historical collision results record hit information for different parameter fields of intelligence data during the historical collision verification process. Intelligence data can be collected, managed, and queried through the intelligence management module, which covers four types of intelligence: IP intelligence, URL intelligence, domain name intelligence, and file intelligence, encompassing information on network security and risk assessment. Intelligence collision is a service within the intelligence management module. By comparing local intelligence with intelligence in a known threat intelligence library, this service can effectively identify potential threats hidden within local intelligence, helping users promptly identify and respond to various network threats and improving network security defense capabilities.

[0068] Assets, a type of local intelligence data, can be divided into two categories: physical assets and virtual assets. Physical assets include various IoT assets such as network devices, security equipment, and terminal devices. Virtual assets include commonly used IP addresses, port services, topology positioning, and other data-related management and supply chain information. The relationships between collected assets are then extracted from different dimensions to form key resources, which are then processed to generate intelligence data.

[0069] Considering that different types of intelligence data have different parameter fields, the parameter fields used in historical collision verification for different types of intelligence data are also different. For example, URL intelligence primarily uses the URL parameter field for collision verification, while IP intelligence primarily uses the IP address parameter field for collision verification. Accordingly, the collision results generated by historical collision verification will also obtain hit information in the corresponding parameter fields, for example, an IP address hits an IP address of a malicious family, and a domain name is registered by a malicious domain name.

[0070] Specifically, during the historical collision verification process, the intelligence data to be collided can trigger the collision logic by calling the intelligence collision interface, calling the corresponding third-party database interface for intelligence collision according to the type of the intelligence data to be collided, and returning the hit information of the parameter field in the intelligence data to be collided.

[0071] In actual application, the intelligence data to be collided can be placed in a queue. By pulling data from the queue, the intelligence data to be collided can be parsed, the intranet IP can be filtered, duplicates can be removed, and the IPs that have collided can be filtered. The data is encapsulated into a collection object and placed in another queue. The collection object is pulled through multiple collision threads, and the batch collision interface is called synchronously to obtain the collision result. If the collision is successful, the IP information of the successful collision is updated to the buffer cluster of the collided IP as a historical collision result.

[0072] The execution subject of this embodiment can be a data processing device or equipment in the intelligence collision process. The server corresponding to the data processing system can be configured to obtain historical collision results collected in the local cache cluster through the intelligence collision service, and perform pre-collision judgment on the intelligence data to be collided based on the hit information of the parameter field in the intelligence data in the historical collision results. If the intelligence data to be collided has already collided, the parameter field hit in the intelligence data to be collided will be recorded in the historical collision record, and then the intelligence data to be collided will be verified based on the collision result that has already collided successfully. There is no need to call the resources of the intelligence system, thereby reducing the number of intelligence collisions.

[0073] 102. In response to the intelligence collision instruction, query whether the pending collision intelligence data has experienced a collision based on the historical collision results.

[0074] Among them, the intelligence collision instruction can be generated through the request of the collision service interface, and the collision intelligence data can transmit the corresponding collision parameters in the collision service interface. The collision parameters may include the intelligence source manufacturer, intelligence type and intelligence value, etc.

[0075] Specifically, you can perform parameter queries on the intelligence data to be collided, pre-process the query parameters, traverse the intelligence source manufacturers and intelligence values, split the intelligence values, traverse each intelligence value, and use each intelligence value as a key to query the local cache cluster. If the intelligence value is queried in the historical collision results, it means that the intelligence data has collided, and the intelligence details are returned accordingly. If the intelligence value is not queried in the historical collision results, it means that the intelligence data has not collided.

[0076] In actual application scenarios, the historical collision results of intelligence data are equivalent to data independently stored in the local cache cluster. After the collision instruction of the intelligence data is triggered, the collision parameters are extracted from the intelligence data to be collided and sent to the intelligence collision module. When the intelligence collision module receives the collision parameters, the parameter preprocessing module preprocesses the collision parameters, traverses the intelligence source manufacturers, and uses the intelligence values ​​and the intelligence source manufacturers to collide the data in the local cache cluster. If the data collides, the data is assembled and the assembled data is returned. If there is no collision, the data is requested according to the third-party interface corresponding to the source manufacturer, and the data is assembled according to the return of the third-party interface.

[0077] 103. If the pending collision intelligence data has not experienced a collision, call an intelligence interface to perform collision verification on different parameter fields in the pending collision intelligence data to obtain hit information of different parameter fields in the pending collision intelligence data.

[0078] Specifically, in the process of calling the intelligence interface to perform collision verification on different parameter fields in the pending collision intelligence data, if the pending collision intelligence data has not experienced a collision, it means that there is no historical collision record for the pending collision intelligence data. The pending collision intelligence data can be added to the pending collision queue, and multi-threaded collision processing can be used to process the pending collision queue in real time. The parameter fields in the pending collision intelligence data are transmitted to the external intelligence interface, and the parameter fields are used to perform collision verification with the external intelligence interface to obtain the collision result returned by the external interface. The collision result is the hit information of the parameter fields in the pending collision intelligence data.

[0079] Specifically, the collision verification process involves using specialized computer software to collide and compare two or more databases / datasets, and conducting in-depth analysis of the resulting overlapping and intersecting data. It should be noted that Peng Zhuan's datasets must contain similar parameter fields. For example, collisions can be performed on different ages or IP addresses. These data are unique, and using identification data as a medium for collisions often makes it easier to quickly and accurately obtain target information. For example, quickly colliding IP addresses can reveal data with security risks.

[0080] It is understandable that in order to effectively prevent malicious attacks and illegal access and improve the security and reliability of the system, the intelligence collision service can provide an encrypted public key by enabling the authentication function. The external intelligence interface needs to add an identification parameter to the request header when making a request. The parameter value can be a string encrypted with the public key using the current request time.

[0081] 104. Update the hit information of different parameter fields in the intelligence data to be collided into the local cache cluster as the collision result of the intelligence data.

[0082] In this embodiment, the local cache cluster can use distributed cache. Considering the high availability requirements of intelligence management for cache components, redis has better support in persistence and clustering, which can maximize the stability of the distributed cache cluster and achieve self-recovery of faults to a certain extent, reducing operation and maintenance costs.

[0083] To ensure the stability and reliability of the Redis cluster, a master-slave replication architecture can be used. Each piece of intelligence data will generate a collision result after a collision. Each collision result will have a manually set cache expiration time when it is added to the cache. The expiration time can default to a preset time after the cache insertion time, for example, 10 days, 5 days, etc. The default expiration time can be adjusted by adjusting the intelligence management service configuration file. Furthermore, to fully utilize the storage resources in the cache cluster, a memory eviction strategy can be used. When the cache cluster's memory is insufficient to accommodate newly written data, the least recently used key in the key space is removed.

[0084] Typically, the id parameter field in the pending collision intelligence data can be used as the primary key of the table in the cache cluster, which is equivalent to a candidate key field for unique identification. The id parameter field is used to uniquely constrain other parameter fields in the pending collision intelligence data.

[0085] The data processing method in the intelligence collision process provided by the embodiment of the present application is compared with the method of frequently calling the intelligence system for data processing using the intelligence collision service in the existing technology. The present application obtains the historical collision results collected by the local cache cluster. The historical collision results record the hit information of the intelligence data on different parameter fields during the historical collision verification process. In response to the intelligence collision instruction, it queries whether the intelligence data to be collided has been collided based on the historical collision results. If the intelligence data to be collided has not been collided, the intelligence interface is called to perform collision verification on the different parameter fields in the intelligence data to be collided, and the hit information of the different parameter fields in the intelligence data to be collided is obtained. Finally, the hit information of the different parameter fields in the intelligence data to be collided is updated to the local cache cluster as the collision result of the intelligence data. The whole process does not rely on the collision interface of the intelligence system, uses the historical collision results to pre-judge the intelligence collision of the intelligence data to be collided, reduces the number of calls to the external intelligence system, and reduces the resource usage of the intelligence system. If the intelligence data to be collided has not been collided, the intelligence interface is called for collision verification, shortens the interface request call time of the intelligence service, avoids data backlog, and improves the data processing capability during the intelligence collision process.

[0086] In the above embodiment, after the intelligence collision service is started, cache preheating monitoring is triggered. Through cache preheating monitoring, the read data can be better assembled and stored in the distributed cache. Furthermore, as shown in FIG2 , before step 101, the method further includes the following steps:

[0087] 105. Start cache preheating monitoring, traverse and read the original data to be collided, assemble the collided intelligence data in the original data to be collided, and insert it into the local cache cluster.

[0088] It is understandable that in order to facilitate the collision comparison of intelligence data, the collision service can be monitored through cache preheating. When the collision service is turned on, it is determined whether cache preheating is required based on the collision service parameters. If cache preheating is required, the original data to be collided is traversed and read. Here, the original data to be collided is the original data extracted for real-time consumption, and the read data is assembled and stored in the distributed cache. The original data includes data that requires collision intelligence in original practices, logs, traffic and other data.

[0089] Specifically, after starting the preheating monitoring, the collision service parameters can be used to determine whether to preheat the local cache cluster. If so, the original data to be collided is traversed and read, and the intelligence data that has collided in the local intelligence data is assembled and inserted into the local cache cluster. Among them, the collision service parameters may include the collision request time, the collision object, etc., and the collision service is determined based on the collision service parameters whether the collision request is triggered. If the collision request is triggered, the abnormal operation is started to query the collision result of the intelligence data to be collided, and the collision result is inserted into the local cache cluster according to the collision type.

[0090] Specifically, in the above embodiment, as shown in FIG3 , step 105 includes the following steps:

[0091] 201. Start cache preheating monitoring, traverse and read original data to be collided, and put the original data to be collided into a first queue.

[0092] 202. Extract collision elements from the raw data to be collided through a parsing thread, perform collision pre-processing on the collision elements, assemble the missed collision elements into an intelligence collection object and put it into a second queue.

[0093] 203. Pull the intelligence collection object from the second queue through the collision thread, use the intelligence collection object as the data to be collided to batch call the collision interface to obtain the collision result of the data to be collided, and store the collision result in the local cache cluster.

[0094] In this embodiment, the data in Kafka can be pulled in batches by starting a Kafka consumer thread, and then the data pulled each time is placed in a memory queue of limited length, where the number of consumer threads is less than or equal to the number of partitions. Then, data is pulled from the queue through multiple data parsing threads, and the field information of the data is parsed in turn to obtain collision elements, where the collision elements can be the original address, destination address, destination domain name, file hash, etc., and then the intranet IP is filtered, duplicates are removed, and the IP operations that have collided are filtered, and the unhit collision elements are encapsulated into a collection object, which is placed in another memory queue of limited length as the data to be collided, and then the data to be collided is pulled through multiple collision threads, and the batch collision interface is called synchronously to obtain the collision result. If the collision is successful, the IP information of the successful collision is updated to filter the IP that has collided and store it in the local cache cluster.

[0095] Specifically, in the above embodiment, as shown in FIG4 , step 102 includes the following steps:

[0096] 301. In response to an intelligence collision instruction, extract pending collision intelligence data from raw data according to a preset configuration.

[0097] 302 : Analyze the valid parameter fields in the pending collision intelligence data, and query whether there is hit information of the valid parameter fields in the historical collision results based on the valid parameter fields.

[0098] 303a. If so, determine that the pending collision intelligence data has undergone a collision; 303b. Otherwise, determine that the pending collision intelligence data has not undergone a collision.

[0099] Among them, the preset configuration can be performed through the database configuration file, which may include configuration items of the database and configuration items of the intelligence data in the database. The configuration items of the database may include but are not limited to the database docking host address, cache usage information, etc., and the configuration items of the intelligence data may include but are not limited to the location of the intelligence data in the database, creation time and operation, etc.

[0100] Taking into account that the pending collision intelligence data may come from multiple manufacturers, specifically, the validity parameter field in the pending collision intelligence data can be parsed according to the intelligence type, and the intelligence value corresponding to the valid parameter field can be traversed. The intelligence value is a valuable information value obtained by using an algorithm to process the valid parameter field. The intelligence value is used as a key to query whether there is intelligence details in the local cache cluster. If so, it means that the pending collision intelligence data has experienced a collision, and the intelligence details obtained by the query are further returned as the query result. Otherwise, it means that the pending collision intelligence data has not experienced a collision.

[0101] Furthermore, in the above embodiment, as shown in FIG5 , after step 102, the method further includes the following steps:

[0102] 401. If the pending collision intelligence data has experienced a collision, extract hit information of the valid parameter field from the historical cache result according to the valid parameter field in the pending collision intelligence data.

[0103] 402. Determine a collision result of the intelligence data based on the hit information of the valid parameter field.

[0104] It is understandable that if the intelligence data to be collided has undergone a collision, the historical collision results can be obtained directly from the local cache cluster, that is, the hit information of the valid parameter field can be extracted from the historical cache results. The process of obtaining the collision results through the historical cache results has basically no delay, which meets the requirements of large-scale data collisions. The intelligence data that hits the cache is data from the real-time synchronized intelligence collision results.

[0105] The local cache cluster can be used to determine whether the intelligence data to be collided has experienced a collision. Furthermore, the collision results in the local cache cluster can be synchronized to the alarm rule processing process in real time. Based on the collision results, early warning analysis can be performed on the hit intelligence data. Specifically, in the above embodiment, as shown in FIG6 , after step 104, the method further includes the following steps:

[0106] 501. Determine whether the parameter field hit by the intelligence data satisfies an alarm detection rule based on historical collision results in the local cache cluster.

[0107] 502. If yes, generate alarm information according to the parameter fields that meet the intelligence hit in the alarm detection rule.

[0108] Among them, the alarm detection rule records at least one intelligence hit judgment logic. Specifically, the parameter field hit in the historical collision verification process can be determined based on the historical collision results in the local cache cluster, and then the parameter field hit in the historical collision verification process can be judged through at least one intelligence hit judgment logic to determine whether the alarm detection rule is met.

[0109] Figure 7 is a framework diagram of data processing during the intelligence collision process. The intelligence collision service in Figure 7 mainly includes three parts in the real-time collision process: the intelligence collision module, the alarm real-time analysis module, and the external intelligence. The intelligence collision module reads the collision data to be collided for real-time consumption from Kafka, caches it in the intelligence real-time cache collision module, and further determines whether the data to be collided has undergone a collision through the cache hit intelligence data. If it has undergone a collision, the collision result can be obtained through the cache hit intelligence data. If it has not undergone a collision, the interface collision data is called to update the collision result to the cache hit intelligence data, and the real-time alarm analysis module program is further started. The hit data is fully loaded in the cache hit intelligence data, and the hit intelligence data is verified through the real-time alarm rules. When the alarm conditions are met, an alarm is issued.

[0110] In actual application scenarios, combined with the framework diagram in Figure 7, the specific data processing flow is shown in Figure 8. Kafka raw data can be divided into two processing flows: the real-time intelligence collision flow and the real-time intelligence alarm rule processing flow. In the real-time intelligence collision flow, the independent intelligence collision module consumes the raw data from Kafka to obtain the latest raw data in real time. It extracts the data that requires collision intelligence from the raw data through configuration. After extracting the data that requires collision, it verifies whether the data has already collided with intelligence through the local cache. The data that has not collided is placed in the queue for collision intelligence. The data processing module for the queue for collision intelligence is a multi-threaded processing module. It obtains the data to be collided in the queue and verifies the data collision with the third-party intelligence interface. It extracts the verification results and the collision results of the data to be collided, updates the local cache, and synchronizes the collision results of the data in real time to other intelligence collision modules and the real-time alarm rule processing module. In addition, in the real-time intelligence alarm rule processing process, the original data in Kafka is obtained in real time and pushed to each enabled alarm rule analysis module for rule comparison and analysis. The alarm rule real-time analysis module extracts the data that needs to be collided from the original data according to the rules. The alarm rule real-time analysis module collides the extracted collision data with the local intelligence result cache that has collided (the cache data source is the result of the real-time intelligence collision, and is updated synchronously in real time). If the collision is successful and meets the comparison logic of other rules, an alarm is generated.

[0111] Furthermore, as a specific implementation of the method of Figures 1-6, an embodiment of the present application provides a data processing device in the intelligence collision process, as shown in Figure 9, which includes: an acquisition unit 61, a query unit 62, a verification unit 63, and an update unit 64.

[0112] An acquisition unit 61 is configured to acquire historical collision results collected by a local cache cluster, wherein the historical collision results record hit information of intelligence data on different parameter fields during the historical collision verification process;

[0113] A query unit 62 is configured to query, in response to the intelligence collision instruction, whether the pending collision intelligence data has experienced a collision based on the historical collision results;

[0114] a verification unit 63 configured to, if the pending collision intelligence data has not experienced a collision, call an intelligence interface to perform collision verification on different parameter fields in the pending collision intelligence data to obtain hit information for different parameter fields in the pending collision intelligence data;

[0115] The updating unit 64 is configured to update the hit information of different parameter fields in the to-be-collided intelligence data as the collision result of the intelligence data into the local cache cluster.

[0116] The data processing device in the intelligence collision process provided by the embodiment of the present application is compared with the method of frequently calling the intelligence system for data processing using the intelligence collision service in the existing technology. The present application obtains the historical collision results collected by the local cache cluster. The historical collision results record the hit information of the intelligence data on different parameter fields during the historical collision verification process. In response to the intelligence collision instruction, it queries whether the intelligence data to be collided has experienced a collision based on the historical collision results. If the intelligence data to be collided has not experienced a collision, the intelligence interface is called to perform collision verification on different parameter fields in the intelligence data to be collided, and the hit information of different parameter fields in the intelligence data to be collided is obtained. Finally, the hit information of different parameter fields in the intelligence data to be collided is updated to the local cache cluster as the collision result of the intelligence data. The whole process does not rely on the collision interface of the intelligence system, uses historical collision results to pre-judge the intelligence collision of the intelligence data to be collided, reduces the number of calls to the external intelligence system, and reduces the resource usage of the intelligence system. If the intelligence data to be collided has not experienced a collision, the intelligence interface is called for collision verification, shortens the interface request call time of the intelligence service, avoids data backlog, and improves the data processing capability during the intelligence collision process.

[0117] In a specific application scenario, the device further includes:

[0118] The starting unit is used to start cache preheating monitoring before obtaining the historical collision results collected by the local cache cluster, traverse and read the original data to be collided, assemble the intelligence data that has been collided in the original data to be collided, and insert it into the local cache cluster. The original data to be collided is the original data extracted for real-time consumption.

[0119] In a specific application scenario, the startup unit is specifically used to start cache preheating monitoring, traverse and read the original data to be collided, and put the original data to be collided into the first queue; extract collision elements from the original data to be collided through the parsing thread, and after the collision preprocessing operation on the collision elements, assemble the missed collision elements into an intelligence collection object and put it into the second queue; pull the intelligence collection object from the second queue through the collision thread, use the intelligence collection object as the data to be collided to batch call the collision interface to obtain the collision results of the data to be collided, and store the collision results in the local cache cluster.

[0120] In a specific application scenario, the query unit 72 is specifically used to respond to the intelligence collision instruction and extract the intelligence data to be collided from the original data according to the preset configuration; parse the valid parameter field in the intelligence data to be collided, and query whether there is hit information of the valid parameter field in the historical collision results based on the valid parameter field; if so, it is determined that the intelligence data to be collided has undergone a collision; otherwise, it is determined that the intelligence data to be collided has not undergone a collision.

[0121] In a specific application scenario, the device further includes:

[0122] an extraction unit configured to, after querying, in response to the intelligence collision instruction, whether the pending collision intelligence data has undergone a collision based on the historical collision results, extract, if the pending collision intelligence data has undergone a collision, hit information of a valid parameter field from the historical cache results based on a valid parameter field in the pending collision intelligence data;

[0123] The determination unit is used to determine the collision result of the intelligence data according to the hit information of the valid parameter field.

[0124] In a specific application scenario, the device further includes:

[0125] a judgment unit, configured to, after updating the hit information of different parameter fields in the intelligence data to be collided as the collision result of the intelligence data into the local cache cluster, judge whether the parameter field hit by the intelligence data satisfies the alarm detection rule based on the historical collision results in the local cache cluster;

[0126] The generating unit is used to generate alarm information according to the parameter field of the intelligence hit in the alarm detection rule.

[0127] In a specific application scenario, the alarm detection rule records at least one intelligence hit judgment logic, and the judgment unit is specifically used to determine the parameter field hit in the historical collision verification process based on the historical collision results in the local cache cluster; and judge whether the parameter field hit in the historical collision verification process meets the alarm detection rule through the judgment logic of the at least one intelligence hit.

[0128] It should be noted that for other corresponding descriptions of the functional units involved in the data processing device in the intelligence collision process provided in this embodiment, reference can be made to the corresponding descriptions in Figures 1 to 6, which will not be repeated here.

[0129] Based on the above-mentioned method as shown in Figures 1-6, accordingly, an embodiment of the present application also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the data processing method in the intelligence collision process as shown in Figures 1-6.

[0130] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0131] Based on the above-mentioned method as shown in Figures 1-6, and the virtual device embodiment shown in Figure 9, in order to achieve the above-mentioned purpose, the embodiment of the present application also provides a physical device for data processing during the intelligence collision process, which can be specifically a computer, a smart phone, a tablet computer, a smart watch, a server, or a network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above-mentioned data processing method during the intelligence collision process as shown in Figures 1-6.

[0132] Optionally, the physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a Wi-Fi module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface), etc.

[0133] In an exemplary embodiment, as shown in FIG10 , the physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor is configured to execute the program stored in the memory and perform the data processing method for the intelligence collision process described in the aforementioned embodiment.

[0134] Those skilled in the art will understand that the physical device structure for data processing during an intelligence collision process provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0135] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device that processes data during the intelligence collision process, supporting the execution of information processing programs and other software and / or programs. The network communication module is used to enable communication between components within the storage medium, as well as with other hardware and software within the physical device that processes information.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that this application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the technical solution of this application, compared with the current existing methods, this application uses historical collision results to pre-judge the collision intelligence data, reduce the number of calls to the external intelligence system, and reduce the resource usage of the intelligence system. If the collision intelligence data has not collided, the intelligence interface is called for collision verification, shortening the interface request call time of the intelligence service, avoiding data backlog, and improving the data processing capability during the intelligence collision process.

[0137] Those skilled in the art will understand that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required to implement the present application. Those skilled in the art will understand that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0138] The serial numbers of the above application are for descriptive purposes only and do not represent the advantages or disadvantages of the implementation scenarios. The above disclosure only discloses several specific implementation scenarios of the present application, but the present application is not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present application.

Claims

1. A data processing method in an intelligence collision process, wherein: include: Obtaining historical collision results collected by the local cache cluster, wherein the historical collision results record hit information of intelligence data on different parameter fields during the historical collision verification process; In response to the intelligence collision instruction, querying whether the intelligence data to be collided has experienced a collision according to the historical collision result; If the intelligence data to be collided has not been collided, calling the intelligence interface to perform collision verification on different parameter fields in the intelligence data to be collided, and obtaining hit information of different parameter fields in the intelligence data to be collided; The hit information of different parameter fields in the intelligence data to be collided is updated in the local cache cluster as the collision result of the intelligence data.

2. The method according to claim 1, wherein: Before obtaining the historical collision results collected by the local cache cluster, the method further includes: Start cache preheating monitoring, traverse and read the original data to be collided, assemble the intelligence data that has been collided in the original data to be collided, and insert it into the local cache cluster. The original data to be collided is the original data extracted by real-time consumption.

3. The method according to claim 2, wherein: The starting cache preheating monitoring, traversing and reading the original data to be collided, assembling the intelligence data that has been collided in the original data to be collided and inserting it into the local cache cluster, includes: Start cache preheating monitoring, traverse and read the original data to be collided, and put the original data to be collided into the first queue; Extract collision elements from the original data to be collided by parsing thread, assemble the collision elements that missed into intelligence collection objects and put them into the second queue after collision preprocessing operation on the collision elements; Pull the intelligence collection object from the second queue through the collision thread, use the intelligence collection object as the data to be collided to batch call the collision interface to obtain the collision result of the data to be collided, and store the collision result in the local cache cluster.

4. The method according to claim 1, wherein: In response to the intelligence collision instruction, querying whether the intelligence data to be collided has been collided according to the historical collision result includes: In response to the intelligence collision instruction, extracting the intelligence data to be collided from the original data according to the preset configuration; Parsing the valid parameter field in the pending collision intelligence data, and querying whether there is hit information of the valid parameter field in the historical collision results according to the valid parameter field; If so, it is determined that the pending collision intelligence data has undergone a collision; otherwise, it is determined that the pending collision intelligence data has not undergone a collision.

5. The method according to claim 1, wherein: After responding to the intelligence collision instruction and querying whether the intelligence data to be collided has been collided according to the historical collision result, the method further includes: If the intelligence data to be collided has been collided, then according to the valid parameter field in the intelligence data to be collided, the hit information of the valid parameter field is extracted from the historical cache result; The collision result of the intelligence data is determined according to the hit information of the valid parameter field.

6. The method according to any one of claims 1 to 5, wherein: After updating the hit information of different parameter fields in the intelligence data to be collided as the collision result of the intelligence data into the local cache cluster, the method further includes: According to the historical collision results in the local cache cluster, determining whether the parameter field hit by the intelligence data satisfies the alarm detection rule; If so, an alarm message is generated based on the parameter fields that meet the intelligence hits in the alarm detection rules.

7. The method according to claim 6, wherein: The alarm detection rule records at least one intelligence hit judgment logic, and judging whether the parameter field hit by the intelligence data satisfies the alarm detection rule according to the historical collision results in the local cache cluster specifically includes: Determine the parameter field hit during the historical collision verification process according to the historical collision results in the local cache cluster; The judgment logic of the at least one intelligence hit is used to determine whether the parameter field hit during the historical collision verification process meets the alarm detection rules.

8. A data processing device in an intelligence collision process, wherein: include: An acquisition unit, used to acquire historical collision results collected by a local cache cluster, wherein the historical collision results record hit information of intelligence data on different parameter fields during the historical collision verification process; A query unit, for responding to the intelligence collision instruction and querying whether the intelligence data to be collided has been collided according to the historical collision result; A verification unit, configured to call an intelligence interface to perform collision verification on different parameter fields in the intelligence data to be collided if the intelligence data to be collided has not been collided, so as to obtain hit information of different parameter fields in the intelligence data to be collided; The updating unit is used to update the hit information of different parameter fields in the intelligence data to be collided as the collision result of the intelligence data into the local cache cluster.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for processing data in the intelligence collision process described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method for processing data in an intelligence collision process as claimed in any one of claims 1 to 7 are implemented.

Images