Industrial control network big data storage method and equipment based on kafka system

Abstract

The application provides an industrial control network big data storage method and equipment based on a Kafka system. The method comprises the following steps: step one, constructing an industrial control network data acquisition layer; step two, constructing an industrial control network data cache layer; step three, constructing an industrial control network data forwarding layer; and step four, constructing an industrial control network data storage, retrieval and display layer. The application selects Kafka cache messages, meets the subscription and forwarding of hundreds of thousands of messages per second of industrial control network data with huge throughput, avoids the data loss phenomenon, has the characteristics of high throughput, persistence and distributed storage, and can match the accurate, fast and safe forwarding and storage of large-scale industrial control network real-time streaming data.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the field of information technology, in particular to a kind of industrial control network big data storage method and equipment based on Kafka system. BACKGROUND

[0002] Early industrial control network is always isolated from external network, and obtains "concealed security" by running proprietary closed software and protocol, so as to protect it from various network attacks. But with the deep integration of two and the large-scale application of network communication technology in industrial control network, the traditional industrial control network isolated from external network has been unable to meet the production demand, and with the gradual opening of industrial control network, various network attacks come in a flood. Because industrial control network and traditional information network have great difference, the existing network security measures cannot play the same effect in the maintenance of industrial control network security. In the existing industrial control network security product, although some intelligent information processing technology has been adopted, the deployment of the existing security product is usually scattered, the management is too independent, lacks a complete overall framework, is easy to form information island, makes various security information difficult to be effectively shared and comprehensively used, and in addition, due to the lack of efficient multi-source multi-dimensional data collection technology, the number of security threat samples is insufficient and the quality is not high. Therefore, developing an industrial control network big data storage method and equipment based on Kafka system can effectively overcome the defects in the above-mentioned related technologies, and has become a technical problem to be solved in the industry. SUMMARY

[0003] In view of the above problems existing in the prior art, the embodiment of the present application provides an industrial control network big data storage method and equipment based on Kafka system.

[0004] In the first aspect, the embodiment of the present application provides an industrial control network big data storage method based on Kafka system, comprising: step one, constructing an industrial control network data collection layer; step two, constructing an industrial control network data cache layer; step three, constructing an industrial control network data forwarding layer; and step four, constructing an industrial control network data storage, retrieval and display layer.

[0005] On the basis of the above method embodiment content, the industrial control network big data storage method based on Kafka system provided in the embodiment of the present application, the step one specifically includes: data collection adopts bypass mode, and the data collection system is connected in parallel with the network for data collection. The broadcast mechanism of the protocol in the industrial control network is adopted, and the Wireshark network traffic data packet capture tool is used. On the premise of not affecting the normal operation of the industrial control system, the session log, network application log and original communication data of the industrial control network are collected, and the corresponding filtering, cleaning, splicing and restoration are completed. The original session data is realized by disk based on the big data storage architecture, and the high-performance full collection of network original traffic is completed.

[0006] On the basis of the above method embodiment content, the method for storing big data of industrial control network based on Kafka system provided in the embodiment of the application, the step two specifically comprises: adding a data cache layer between the data collection layer and the forwarding layer, using a Kafka distributed subscription message system, when writing the multi-dimensional heterogeneous data collected from different data sources by the collection layer into Kafka, data fusion needs to be performed, the multi-source data contains various uncertain information, the DS evidence theory is used to improve the reliability of data fusion during fusion, the DS evidence theory defines a space For the identification framework, any proposition A in the problem domain should be contained in the power set , define a mapping m: , a basic probability assignment function on the identification framework , if the evidence is not completely conflicting, the belief function under the combined action of the evidence is calculated by using the DS combination rule. Then the organic fusion of multi-source data is completed by using the Dempster combination rule.

[0007] On the basis of the above method embodiment content, the method for storing big data of industrial control network based on Kafka system provided in the embodiment of the application, the definition mapping m: , a basic probability assignment function on the identification framework , comprises: the DS evidence theory provides the Dempster combination rule, assuming that two independent evidence bodies on the unified identification framework, the corresponding mass functions are and , the focal elements are , ,…, and , ,…, , the combined mapping m is defined as:

[0008]

[0009] , wherein is a conflict item, reflecting the degree of evidence conflict.

[0010] On the basis of the above method embodiment content, the industrial control network big data storage method based on the Kafka system provided in the embodiment of the application specifically comprises that: the industrial control network data forwarding layer pays attention to data compression, this layer has extensibility, when the Kafka consumer consumes data, different algorithms are added for different needs to analyze and process the data, the forwarding layer sends the related data to the persistent storage medium, the data forwarding layer writes each flow data processed into the persistent storage medium in the storage layer by writing the Kafka consumer program, consuming the industrial control network flow data corresponding to the topic, and structuring the data; the Kafka consumer subscribes to one or more topics and reads according to the order of message generation, the consumer distinguishes the messages that have been read by checking the offset of the messages, the offset is a kind of metadata, which is an integer value that is constantly increasing, when the message is created, the kafka adds the offset to the message, in a given partition, the offset of each message is unique, the consumer saves the last read message offset of each partition in the zookeeper or the kafka, if the consumer is closed or restarted, the reading state will not be lost, the consumer is part of a consumer group, and one or more consumers will read a topic together, the consumer group ensures that each partition can only be used by one consumer in the same group, if a consumer fails, other consumers in the group will take over the work of the failed consumer, and the consumer programs in different consumer groups of different nodes in the cluster consume the same information of the same topic.

[0011] On the basis of the above method embodiment content, the industrial control network big data storage method based on the Kafka system provided in the embodiment of the application specifically comprises that: the forwarding layer writes the data required for subsequent display into the permanent storage medium after the industrial control network flow data is consumed by the Kafka consumer program through program structuring processing, and builds a retrieval function, so that the data requester can call the structured data stored in the database to train the statistical learning algorithm and perform big data analysis; the purpose of setting the data retrieval and display layer is to provide an interface for subsequent information query and analysis, the display layer displays the related conditions of the industrial control network flow data through the development of a web interface, the web interface links the back-end database, provides a query retrieval function, and allows users to supervise the security situation of the industrial control network in real time and make decisions.

[0012] In a second aspect, embodiments of the present application provide a big data storage device for an industrial control network based on a Kafka system, comprising: a first main module configured to implement step one of constructing an industrial control network data collection layer; a second main module configured to implement step two of constructing an industrial control network data cache layer; a third main module configured to implement step three of constructing an industrial control network data forwarding layer; and a fourth main module configured to implement step four of constructing an industrial control network data storage, retrieval and display layer.

[0013] In a third aspect, embodiments of the present application provide an electronic device, comprising:

[0014] at least one processor; and

[0015] at least one memory in communication with the processor, wherein:

[0016] The memory stores program instructions executable by the processor, and the processor invoking the program instructions is capable of executing the big data storage method for an industrial control network based on a Kafka system provided in any one of the various implementation manners of the first aspect.

[0017] In a fourth aspect, embodiments of the present application provide a non-transitory computer readable storage medium storing computer instructions, and the computer instructions cause a computer to execute the big data storage method for an industrial control network based on a Kafka system provided in any one of the various implementation manners of the first aspect.

[0018] The big data storage method for an industrial control network based on a Kafka system and the device provided by the embodiments of the present application select Kafka cache messages, meet the subscription forwarding of hundreds of thousands of messages per second of industrial control network data with huge throughput, avoid data loss, have high throughput, persistence and distributed storage characteristics, and can meet the accurate, fast and safe forwarding and storage of large-scale industrial control network real-time streaming data. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 The big data storage method for an industrial control network based on a Kafka system provided by the embodiments of the present application is shown in the flowchart;

[0021] Figure 2 The big data storage device for an industrial control network based on a Kafka system provided by the embodiments of the present application is shown in the structural schematic diagram;

[0022] Figure 3 The physical structure schematic diagram of the electronic equipment provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application. In addition, the technical features in each embodiment or in a single embodiment provided by the present application can be combined with each other at will to form a feasible technical scheme, and this combination is not restricted by the order of steps and / or structure mode, but it must be based on the implementation by those skilled in the art, when the combination of technical schemes appears contradictory or cannot be implemented, it should be considered that the combination of technical schemes does not exist, and is not within the protection scope required by the present application.

[0024] In order to effectively cope with the increasingly severe industrial control network security situation, in the face of large-scale and various protocol industrial control network, the multi-dimensional data compression and multi-source data fusion of the whole network are researched, and the efficient collection and comprehensive use of multi-source multi-dimensional data of the whole network are realized on the premise of greatly reducing the storage and transmission overhead. Due to the large scale and fast generation speed of industrial control network flow data, the traditional data processing platform cannot meet the storage and calculation of large-scale data, and the data acquisition and storage need to use the method of big data. Based on this idea, the embodiment of the present application provides an industrial control network big data storage method based on Kafka system, which is shown in Figure 1 The method comprises the following steps: step one, constructing an industrial control network data acquisition layer; step two, constructing an industrial control network data cache layer; step three, constructing an industrial control network data forwarding layer; and step four, constructing an industrial control network data storage, retrieval and display layer.

[0025] Based on the content of the above method embodiment, as an optional embodiment, the industrial control network big data storage method based on Kafka system provided in the embodiment of the present application specifically comprises: the data acquisition adopts a bypass mode, the data acquisition system is connected in parallel with the network for data acquisition, the broadcast mechanism of the protocol in the industrial control network is used, the Wireshark network flow data packet capture tool is used, the session log, network application log and original communication data of the industrial control network are collected on the premise of not affecting the normal operation of the industrial control system, and the corresponding filtering, cleaning, splicing and restoration are completed, the original session data is realized based on the big data storage architecture, and the high-performance full-amount collection of the original network flow is completed.

[0026] Specifically, the data collection layer is responsible for obtaining traffic data from the industrial control network in the form of stream data, which is cached in Kafka through the producer program of Kafka. For actual industrial control network scenarios, this layer uses traffic mirroring to access the industrial control network terminal to collect traffic data without affecting the normal operation of the system and the industrial production business. One data collection method is to use WireShark or other packet capture tools to capture traffic data packets in the network. Or write a script in Python to read the industrial control network data set in the form of stream data, simulate the collection of stream data in a real scenario, and expand the data set to generate stream data continuously. Bypassing access does not change the network structure, is easy to deploy, and does not affect the stability and reliability of the network. For various network forms, the system device uses a network hardware-independent working mode that can adapt to various hardware configurations and network drivers. In a typical deployment, switch mirroring or access is used. The collection system has high availability and high reliability, single-point deployment and cluster deployment, and supports elastic expansion of system storage, retrieval and analysis capabilities, and avoids data loss due to single-point hardware downtime.

[0027] Based on the content of the above method embodiment, as an optional embodiment, the industrial control network big data storage method based on the Kafka system provided in the embodiment of the application comprises the following steps: adding a data cache layer between the data collection layer and the forwarding layer, and using a Kafka distributed subscription message system, when writing multi-dimensional heterogeneous data collected from different data sources by the collection layer into Kafka, data fusion is required, multi-source data contains various uncertain information, and DS evidence theory is used to improve the reliability of data fusion during fusion. The DS evidence theory defines a space For the identification framework, any proposition A in the problem domain should be included in the power set The mapping m is defined as: The basic probability assignment function on the identification framework If the evidence is not completely conflicting, the belief function under the joint action of the evidence is calculated by using the DS combination rule. Then, the organic fusion of multi-source data is completed by using the Dempster combination rule.

[0028] Based on the content of the above method embodiment, as an optional embodiment, the industrial control network big data storage method based on the Kafka system provided in the embodiment of the application comprises the following steps: adding a data cache layer between the data collection layer and the forwarding layer, and using a Kafka distributed subscription message system, when writing multi-dimensional heterogeneous data collected from different data sources by the collection layer into Kafka, data fusion is required, multi-source data contains various uncertain information, and DS evidence theory is used to improve the reliability of data fusion during fusion. The DS evidence theory defines a space The basic probability assignment function on the identification framework includes: the DS evidence theory provides the Dempster combination rule, and two independent evidence bodies on the unified identification framework are set, and the corresponding mass functions are and , the focal elements are , ,…, and , ,…, , the combined mapping m is defined as:

[0029] (1)

[0030] wherein, is the conflict item, reflecting the degree of evidence conflict.

[0031] Specifically, due to the huge data throughput of large-scale industrial control networks, when the throughput exceeds the data processing capacity, data loss may occur. In order to avoid this phenomenon as much as possible, a data cache layer is added between the data collection layer and the forwarding layer. This layer uses the Kafka distributed publish-subscribe message system to cache streaming data, and uses the Kafka producer program to forward messages from the data collection layer to Kafka. Ubuntu 18.04.6 operating system is used to build a Kafka cluster. Since Kafka depends on Zookeeper, Zookeeper and Kafka on each virtual machine are set. The steps of building the big data environment of the application include:

[0032] (1) Configure JDK, Zookeeper and Kafka software required by the application.

[0033] (2) Create a Kafka topic named ics for publishing and subscribing to the industrial control network traffic data of the application.

[0034] (3) Write a Kakfa producer program to read industrial control network data and produce it to the corresponding partition of Kafka in the form of streaming data.

[0035] (4) Write a Kafka consumer program to subscribe and forward industrial control network data.

[0036] When writing multi-dimensional heterogeneous data collected from different data sources in the collection layer into Kafka, data fusion is required. Multi-source data contains various uncertain information, and DS evidence theory is used to improve the reliability of data fusion during fusion. DS evidence theory provides Dempster combination rule, assuming two independent evidence bodies on a unified recognition framework, their corresponding mass functions are and , the focal elements are , ,…, and , ,…, The combined mapping m is then defined as in (1). The coefficient 1 / (1-k) is a normalizing factor that is included to avoid assigning a non-zero probability to the empty set. The DS combination rule reflects the conjunctive effect of the evidence and gives the belief functions based on different evidence in the same recognition framework. If the evidence is not completely conflicting, the belief function under the conjunctive effect of the evidence is calculated by the DS combination rule. The organic fusion of multi-source data is completed by using the Dempster combination rule.

[0037] Based on the content of the method embodiment, as an optional embodiment, the method for storing big data of an industrial control network based on a Kafka system provided in the embodiment of the application specifically comprises the following steps: the industrial control network data forwarding layer pays attention to data compression, the layer has extensibility, when a Kafka consumer consumes data, different algorithms are added according to different requirements to analyze and process the data, the forwarding layer sends relevant data to a persistent storage medium, the data forwarding layer writes each piece of processed flow data into the persistent storage medium in the storage layer by writing a Kafka consumer program, consuming the industrial control network flow data corresponding to a topic, and performing structured processing on the data; the Kafka consumer subscribes to one or more topics and reads according to the order in which the messages are generated, the consumer distinguishes the messages that have been read by checking the offset of the messages, the offset is metadata, which is an integer value that is constantly increasing, when a message is created, the Kafka adds the offset to the message, the offset of each message in a given partition is unique, the consumer saves the last read message offset of each partition in the Zookeeper or the Kafka, if the consumer is closed or restarted, the reading state will not be lost, the consumer is part of a consumer group, and one or more consumers in the group read a topic together, the consumer group ensures that each partition can only be used by one consumer in the same group, if a consumer fails, other consumers in the group take over the work of the failed consumer, and the consumer programs in different consumer groups of different nodes in the cluster consume the same information of the same topic.

[0038] Specifically, the data forwarding and storage layer consumes data from Kafka by running a Kafka consumer program, the consumer program pulls traffic data in real time, and performs compression, statistical learning analysis operations on the pulled traffic data. The forwarding layer finally sends the data to the persistent storage medium. The data forwarding layer writes the processed traffic data to the persistent storage medium by writing a Kafka consumer program, consuming industrial control network traffic data corresponding to the topic, and structuring the data. The storage layer serves the subsequent retrieval and display layer, and opens the database for remote reading, allowing data requesters to call the structured data stored in the database for statistical learning algorithm training and big data analysis. The forwarding layer deploys consumer programs on multiple nodes in the big data cluster to read the same messages in the Kafka topic for different functions, such as structured processing and writing to the database for web display; bringing in small sample enhancement algorithms to expand the data after writing to the data collection layer to expand the data; and bringing in neural network or machine learning model to learn attack behavior of various attacks on industrial control network to maintain the security of industrial control network.

[0039] Based on the above method embodiment, as an optional embodiment, the industrial control network big data storage method based on the Kafka system provided in the embodiment of the application specifically includes the following steps: after the forwarding layer consumes industrial control network traffic data through the Kafka consumer program, the subsequent data required for display is structured and written to the permanent storage medium through the program, and a retrieval function is constructed to allow data requesters to call the structured data stored in the database for statistical learning algorithm training and big data analysis; the purpose of setting the data retrieval and display layer is to provide an interface for subsequent information query and analysis, and the display layer displays the relevant situation of industrial control network traffic data through the development of a web interface, the web interface links the backend database, provides query and retrieval functions, and allows users to supervise the security situation of the industrial control network in real time and make decisions.

[0040] Specifically, the storage layer selects a persistent storage medium such as a MySQL database for constructing a retrieval table, and provides a data interface for the presentation layer to read to form a webpage display. The purpose of the data retrieval and presentation layer is to provide an interface for subsequent information query and analysis. It is difficult to find the rules implied in massive data from a single piece of information. The data retrieval and presentation layer provides a unified query portal for all information in the data storage layer, and performs statistical analysis on the information to present the results in a visual form. Network situation visualization converts complex and abstract log information into chart forms, making it easier for users to understand the relationship and changes between various information, and facilitating users to make better decisions. The retrieval layer provides remote database connection to data requesters through the remote access permission of the database adopted by the storage layer, and retrieves and uses the industrial control network flow data through database-related operations. The presentation layer displays the relevant conditions of the industrial control network data through the development of a web interface, and displays the network situation data of the industrial control system in real time. The web interface is attached with historical flow data query, dangerous flow data query, administrator background management, flow monitoring, and network situation prediction display functions.

[0041] The industrial control network big data storage method based on the Kafka system provided in the embodiments of the present application selects Kafka cache messages, meets the subscription and forwarding of hundreds of thousands of messages per second of the industrial control network data with huge throughput, avoids data loss, has high throughput, persistence and distributed storage characteristics, and can accurately, quickly and safely forward and store large-scale industrial control network real-time streaming data.

[0042] The implementation basis of each embodiment of the present application is the programmed processing of a device with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application can be packaged into various modules. Based on this actual situation, on the basis of the above embodiments, the embodiments of the present application provide an industrial control network big data storage device based on a Kafka system, which is used to execute the industrial control network big data storage method based on the Kafka system in the above method embodiments. Referring to Figure 2 , the device comprises: a first main module for implementing step one: constructing an industrial control network data acquisition layer; a second main module for implementing step two: constructing an industrial control network data cache layer; a third main module for implementing step three: constructing an industrial control network data forwarding layer; and a fourth main module for implementing step four: constructing an industrial control network data storage, retrieval and presentation layer.

[0043] The industrial control network big data storage device based on the Kafka system provided in the embodiments of the present application adopts Figure 2The Kafka cache message is selected in several modules in the method, the subscription forwarding of the industrial control network data of the huge throughput of several hundred thousand messages per second is met, the data loss phenomenon is avoided, the high throughput, the persistence and the distributed storage characteristics are possessed, and the accurate, fast, safe forwarding and storage of the large-scale industrial control network real-time streaming data can be matched.

[0044] It should be noted that the device in the device embodiment provided by the application can be used to implement the method in the method embodiment, and can also be used to implement the method in other method embodiments provided by the application. The difference is only that the corresponding function modules are set, and the principle is basically the same as that of the above-mentioned device embodiment. As long as the person skilled in the art can obtain the corresponding technical means by combining technical features on the basis of the above-mentioned device embodiment, and the technical solution formed by these technical means, on the premise of ensuring the practicability of the technical solution, the device in the above-mentioned device embodiment can be improved to obtain the corresponding device embodiment, and the method in other method embodiments can be implemented. For example:

[0045] Based on the content of the above device embodiment, as an optional embodiment, the industrial control network big data storage device based on the Kafka system provided in the embodiment of the application further comprises: a first sub-module, configured to implement the step one and specifically comprising: the data acquisition system is connected in parallel with the network for data acquisition in a bypass mode, the broadcast mechanism of the protocol in the industrial control network is adopted, the Wireshark network traffic data packet capture tool is used, the session log, the network application log and the original communication data of the industrial control network are collected without affecting the normal operation of the industrial control system, and corresponding filtering, cleaning, splicing and restoration are completed, the original session data is written based on the big data storage architecture, and high-performance full-quantity acquisition of the network original traffic is completed.

[0046] Based on the content of the above device embodiment, as an optional embodiment, the industrial control network big data storage device based on the Kafka system provided in the embodiment of the application further comprises: a second sub-module, configured to implement the step two and specifically comprising: a data cache layer is added between the data acquisition layer and the forwarding layer of the system, when the multi-dimensional heterogeneous data collected from different data sources in the acquisition layer is written into the Kafka distributed subscription message system, data fusion is required, the multi-source data contains various uncertain information, the DS evidence theory is used to improve the reliability of data fusion during fusion, the DS evidence theory defines the space For the identification framework, any proposition A in the problem domain should be contained in the power set The mapping m is defined as: For the identification framework The basic probability assignment function on the unified recognition framework is calculated, and if the evidence is not completely conflicting, the belief function under the combined action of the evidence is calculated by using a DS combination rule.

[0047] Based on the above device embodiment, as an optional embodiment, the industrial control network big data storage device based on the Kafka system provided in the embodiment of the application further includes a third sub-module for realizing the definition of the mapping m: For the recognition framework The basic probability assignment function on the unified recognition framework is calculated, and if the evidence is not completely conflicting, the belief function under the combined action of the evidence is calculated by using a DS combination rule. and The focus element is , …, and , …, The combined mapping m is defined as:

[0048]

[0049] Wherein, is a conflict item, reflecting the degree of evidence conflict.

[0050] Based on the content of the above device embodiment, as an optional embodiment, the industrial control network big data storage device based on the Kafka system provided in the embodiment of the application further includes: a fourth sub-module, configured to implement the step three and specifically include: the industrial control network data forwarding layer pays attention to data compression, the layer has extensibility, when the Kafka consumer consumes data, different algorithms are added for different needs to analyze and process the data, the forwarding layer sends the related data to the persistent storage medium, the data forwarding layer writes each flow data processed into the persistent storage medium by writing the Kafka consumer program, consuming the industrial control network flow data corresponding to the topic, and performing structured processing on the data; the Kafka consumer subscribes to one or more topics and reads in the order of message generation, the consumer distinguishes the messages that have been read by checking the offset of the messages, the offset is a kind of metadata, which is an integer value that is constantly increasing, when the message is created, the Kafka adds the offset to the message, in a given partition, the offset of each message is unique, the consumer saves the last read message offset of each partition in the zookeeper or the Kafka, if the consumer is closed or restarted, the reading state will not be lost, the consumer is part of a consumer group, and one or more consumers in the group read a topic together, the consumer group ensures that each partition can only be used by one consumer in the same group, if a consumer fails, other consumers in the group take over the work of the failed consumer, and the consumer programs in different consumer groups of different nodes in the cluster consume the same information of the same topic.

[0051] Based on the content of the above device embodiment, as an optional embodiment, the industrial control network big data storage device based on the Kafka system provided in the embodiment of the application further includes: a fifth sub-module, configured to implement the step four and specifically include: after the forwarding layer consumes the industrial control network flow data through the Kafka consumer program, the subsequent data required for display is structured and written into the permanent storage medium through the program, and a retrieval function is constructed, so that the data requester can call the structured data stored in the database to train the statistical learning algorithm and perform big data analysis; the purpose of setting the data retrieval and display layer is to provide an interface for subsequent information query and analysis, the display layer displays the related situation of the industrial control network flow data through the development of a web interface, the web interface links the back-end database to provide a query retrieval function, so that the user can supervise the security situation of the industrial control network in real time and make decisions.

[0052] The method of the embodiment of the application is implemented by relying on an electronic device, and therefore it is necessary to introduce the related electronic device. Based on this purpose, the embodiment of the application provides an electronic device, such as Figure 3As shown, the electronic device includes at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory complete mutual communication through the communications bus. The at least one processor can invoke a logical instruction in the at least one memory to execute all or part of the steps of the method provided by the foregoing various method embodiments.

[0053] In addition, the logical instruction in the at least one memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various method embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0054] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0055] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0056] The computer program product of the present application can be a computer program implemented on one or more various apparatuses, e.g., general purpose computers or dedicated computers. When implemented on a general purpose computer, the computer program can be stored in a tangible computer usable medium, e.g., a compact diskette, a compact disk, a magnetic disk, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), etc., of the general purpose computer. Therefore, the computer program can be loaded onto such a general purpose computer in a straight forward manner by placing the computer program in the tangible computer usable medium and then placing the computer program into the general purpose computer.

[0057] It should be noted that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0058] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limiting the present application; although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions recorded in the above-mentioned embodiments can be modified or equivalent replacements can be made to some technical features; and the modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An industrial control network big data storage method based on a Kafka system, characterized in that, The application relates to an industrial control network data collection method. Step one: constructing an industrial control network data collection layer; step two: constructing an industrial control network data cache layer; Step three: constructing an industrial control network data forwarding layer; and step four: constructing an industrial control network data storage, retrieval and display layer. The step two specifically comprises: the system adds a data cache layer between the data collection layer and the forwarding layer, adopts a Kafka distributed subscription message system, and writes the multi-dimensional heterogeneous data collected from different data sources in the collection layer into the Kafka, and needs to perform data fusion. The multi-source data contains various uncertain information. In the fusion, the DS evidence theory is used to improve the reliability of data fusion. The DS evidence theory defines a space For the identification framework, any proposition A in the problem domain should be included in the power set , define the mapping m: The basic probability assignment function on the identification framework is m. If the evidence is not completely conflicting, the belief function under the joint action of the evidence is calculated by using the DS combination rule; then the organic fusion of multi-source data is completed by using the Dempster combination rule. The definition of the mapping m: is a basic probability assignment function on the recognition frame , including: DS evidence theory provides Dempster combination rule, set two independent evidence bodies on the unified recognition frame, and the corresponding mass functions are and , the focal elements are , , and , , , then the combined mapping m is defined as: wherein, is a conflict item, reflecting the degree of evidence conflict.

2. The method according to claim 1, wherein, The step one specifically comprises the following steps: a bypass mode is adopted for data collection, a data collection system is connected in parallel with a network for data collection, a broadcast mechanism of an industrial control network protocol is adopted, a Wireshark network flow data packet capturing tool is used, session logs, network application logs and original communication data of the industrial control network are collected under the premise of not affecting the normal operation of the industrial control system, corresponding filtering, cleaning, splicing and restoration are completed, original session data is stored based on a big data storage architecture, and high-performance full-amount collection of network original flow is completed. 3.The method of claim 1, wherein, The step three specifically comprises the following steps: the industrial control network data forwarding layer focuses on data compression, the layer has extensibility, different algorithms are added for data analysis and processing according to different requirements when a Kafka consumer consumes data, the forwarding layer sends related data to a persistent storage medium, the data forwarding layer writes each flow data after processing into the persistent storage medium by writing a Kafka consumer program, subscribing to one or more topics, reading in the order of message generation, distinguishing the read messages by checking the message offset, the offset is a kind of metadata, is a continuously increasing integer value, and is added to the message when the message is created, the offset of each message is unique in a given partition, the consumer saves the last read message offset of each partition in the zookeeper or the kafka, and the read state is not lost when the consumer is closed or restarted, the consumer is part of a consumer group, one or more consumers read a topic together, the consumer group ensures that each partition can only be used by one consumer in the same group, and other consumers in the group take over the work of the failed consumer, and the consumer programs of different consumers in different nodes in the cluster consume the same information of the same topic.

4. The method according to claim 3, wherein, The step four specifically comprises the following steps: after the forwarding layer consumes the industrial control network flow data through the Kafka consumer program, the data required for subsequent display is structured and written into the permanent storage medium through the program, a retrieval function is constructed, structured data stored in the database is called by a data requester to train a statistical learning algorithm and perform big data analysis, and the data retrieval and display layer is set to provide an interface for subsequent information query and analysis, the display layer displays the related conditions of the industrial control network flow data through the development of a web interface, the web interface links a back-end database, provides a query and retrieval function, and enables users to supervise the security situation of the industrial control network in real time and make decisions.

5. An industrial control network big data storage device based on a Kafka system, characterized in that, The device for implementing the method of any one of claims 1-4 comprises: a first main module for implementing step one of constructing an industrial control network data acquisition layer; a second main module for implementing step two of constructing an industrial control network data cache layer; a third main module for implementing step three of constructing an industrial control network data forwarding layer; and a fourth main module for implementing step four of constructing an industrial control network data storage, retrieval and display layer.

6. An electronic device, comprising: Comprise: at least one processor, at least one memory and a communication interface; wherein, the processor, memory and communication interface communicate with each other; the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the method of any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 4.

Images