Markov chain-based data analysis method and device

Abstract

The invention provides a Markov chain-based data analysis method and device, and the method comprises the steps: obtaining to-be-analyzed data, and carrying out the basic analysis of the to-be-analyzed data, so as to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the data to be analyzed and an analysis result of the unencrypted plaintext data; calling a pre-stored secondary analysis program, and performing secondary analysis on the encrypted file in the encrypted database to obtain an analysis result of the encrypted file; and performing data summarization on the analysis result of the unencrypted plaintext data and the analysis result of the encrypted file to obtain a summarized result. The device can perform effective data analysis on common data and encrypted data, thereby providing effective data support for intranet data security.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network data security, and in particular to a data analysis method and device based on Markov chain. BACKGROUND

[0002] In the information age, with the rapid development of computer science and network technology, the use of network has become an indispensable part of various industries. The dangerous viruses and incorrect network traffic in the Internet require analysis of the data in the intranet, and the abnormal data traffic needs to be analyzed and warned.

[0003] However, the analysis of data is a complex thing. Each protocol in the data traffic needs to be parsed to analyze the data content in the protocol. Each protocol type is extremely complex, and most of the data is encrypted, which makes it impossible to know the data content and effectively analyze the encrypted data.

[0004] Therefore, how to effectively analyze ordinary data and encrypted data to provide effective data support for intranet data security has become a problem to be solved by those skilled in the art. SUMMARY

[0005] The present application provides a data analysis method and device based on Markov chain, which can effectively analyze ordinary data and encrypted data to provide effective data support for intranet data security.

[0006] The present application provides a data analysis method based on Markov chain, which comprises:

[0007] obtaining data to be analyzed;

[0008] performing basic analysis on the data to be analyzed to obtain a plaintext database and an encrypted database; the plaintext database includes unencrypted plaintext data in the data to be analyzed and analysis results of the unencrypted plaintext data;

[0009] calling a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files;

[0010] performing data aggregation on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain an aggregated result.

[0011] In some embodiments, obtaining the aggregated result further comprises:

[0012] visually displaying the aggregated result.

[0013] In some embodiments, the data to be analyzed is acquired, specifically including:

[0014] The target network file is selected for uploading;

[0015] The uploaded network file is fragmented, and the current incoming data and the number of fragments are recorded;

[0016] The incoming data and the number of fragments are saved, and if the data saving process is interrupted, the file hash value is used for judgment;

[0017] If the saving fails, the breakpoint resume is performed, and if the saving succeeds, the data is merged, and the merged file is stored in the formal file.

[0018] In some embodiments, the data to be analyzed is analyzed, specifically including:

[0019] The packet file is read and fragmented;

[0020] The protocol information of the unencrypted plaintext that can be acquired is parsed, and the parsing result is stored in the plaintext database in the form of JSON;

[0021] The encrypted data requiring secondary analysis is stored in the encrypted database.

[0022] In some embodiments, the encrypted file in the encrypted database is analyzed, specifically including:

[0023] The ciphertext is acquired, and the encryption type of the ciphertext is judged based on the Markov chain adjusted based on the data, the Markov chain predicts the content according to t+1 based on known X1 characteristics and s characteristics;

[0024] After the algorithm analysis, the results are summarized to obtain a result set, the result set is analyzed again to obtain the purpose of the encrypted data packet, and the results are output to the JSON file, and the key information and the communication process are also collected and stored in the confidential database.

[0025] In some embodiments, the analysis results of the unencrypted plaintext data and the analysis results of the encrypted file are summarized, specifically including:

[0026] According to the arrival sequence algorithm, the sorting is performed again, and the corresponding JSON value is stored in the MongoDB database;

[0027] The old data after the basic analysis and the secondary analysis is cleaned.

[0028] The application also provides a data analysis device based on Markov chain, the device comprises:

[0029] a data acquisition unit configured to acquire data to be analyzed;

[0030] a basic analysis unit configured to perform basic analysis on the data to be analyzed to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the data to be analyzed and analysis results of the unencrypted plaintext data;

[0031] a secondary analysis unit configured to call a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files;

[0032] a result summarizing unit configured to summarize the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain a summary result.

[0033] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as described above when executing the program.

[0034] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the method as described above.

[0035] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the method as described above.

[0036] The application provides a Markov chain-based data analysis method, which comprises the following steps: acquiring data to be analyzed, performing basic analysis on the data to be analyzed to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the data to be analyzed and analysis results of the unencrypted plaintext data; calling a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files; and summarizing the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain a summary result. The method and device can effectively analyze both normal data and encrypted data, thereby providing effective data support for internal network data security. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0038] Figure 1 is one of the flowcharts of the data analysis method based on Markov chain provided by the present application;

[0039] Figure 2 is another flowchart of the data analysis method based on Markov chain provided by the present application;

[0040] Figure 3 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0042] In one specific embodiment, as shown in Figure 1 the present application provides a data analysis method based on Markov chain, which comprises the following steps:

[0043] S110: obtaining the data to be analyzed, i.e. importing the data needed for analysis;

[0044] S120: performing basic analysis on the data to be analyzed to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the data to be analyzed and analysis results of the unencrypted plaintext data; and performing basic analysis on the imported data;

[0045] S130: calling a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files; that is, using Markov chain to perform secondary behavior analysis on encrypted data;

[0046] S140: performing data summarization on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain a summary result; that is, summarizing the data;

[0047] S150: performing visual display on the summary result, i.e. performing display.

[0048] Overall, as shown in Figure 1 the overall flow of the data analysis method comprises importing the data needed for analysis, first performing basic analysis, then using Markov chain to perform secondary analysis on encrypted data which cannot be analyzed by the basic analysis, then summarizing the data, and finally performing display.

[0049] In step S110, the data to be analyzed is acquired, specifically including:

[0050] The target network file is selected for uploading;

[0051] The uploaded network file is fragmented, and the current incoming data and the number of fragments are recorded;

[0052] The incoming data and the number of fragments are saved, and if the process is interrupted, the file hash value is used for judgment;

[0053] If the saving fails, the breakpoint resume is performed, and if the saving succeeds, the data is merged, and the merged file is stored in the formal file.

[0054] Specifically, as shown in Figure 2 During the data import process, the user selects a network file for uploading through a web browser, supports pcap, pacapng, cap, and other network packet formats, and when the uploading starts, the web interface fragments the data, transmits only 10MB at a time, records the current incoming data and the number of fragments, and then saves them. If the process is interrupted, the file hash value can be used for judgment. If it is a failed file, the breakpoint resume is performed. If it is successful, the data is merged, and the merged file is stored in the formal file, thereby completing the data import step.

[0055] In step S120, the data to be analyzed is analyzed, specifically including:

[0056] The packet file is read and fragmented;

[0057] The unencrypted plaintext protocol information that can be obtained is parsed, and the parsing result is stored in the plaintext database in JSON format;

[0058] The encrypted data that needs to be analyzed again is stored in the encrypted database.

[0059] Specifically, when the basic analysis starts, the packet file is read and fragmented, which is to analyze faster. The unencrypted plaintext protocol information that can be obtained is parsed, and the parsing result is stored in JSON format. The required data is stored in the ES database. The TCP, UDP, ISAKMP key interaction, HTTP, SSH key interaction, and HTTPS key interaction data information can be effectively obtained and parsed, and the communication relationship, certificate information, and signature information can be effectively obtained and parsed. Finally, it is displayed through the web interface.

[0060] In step S130, the encrypted file in the encrypted database is analyzed again, specifically including:

[0061] Obtaining ciphertext, judging the encryption type of the ciphertext by adjusting the Markov chain based on the data, the Markov chain predicts the content according to t+1 by known X1 characteristics and s characteristics;

[0062] After using algorithm analysis, the results are summarized to obtain a result set, and the result set is analyzed again to obtain the purpose of the encrypted data packet, and the results are output to the JSON file, and the key information and the communication process are also collected and stored in the confidential database.

[0063] Specifically, after the basic analysis is completed, the basic analysis will arrange the encrypted data into a pcap file, and call the secondary analysis program for analysis. When starting secondary analysis, taking a single data packet as an example, first judge the large encryption type, such as HTTPS, SSH, VPN and other commonly used encryption information. Taking HTTPS as an example, first obtain the ciphertext, and after obtaining the ciphertext, judge by Markov chain, and the details of the Markov chain are changed:

[0064] P(X t+1 =s t+1 |X t =s t , X t-1 =s t-1 ,..., X0=s0) = P(X t+1 =s t+1 |X t =s t )

[0065] Where: s t represents the time interval between the tth data packet, s t-1 represents the time interval between the t-1th data packet;

[0066] X t represents the length of the tth data packet, X t-1 represents the length of the t-1th data packet;

[0067] P represents the probability that the data packet time interval is s t and the data packet length is X t .

[0068] The adjustment of Markov algorithm for data analysis, for example, by known X1 feature and s feature, according to t+1 predict his content, for example, a certain APP or application and protocol of https encrypted data has specific encryption algorithm and length, can be according to algorithm+length+IP+communication times, speculate what this request requests, what application, although the plaintext cannot be known, also can be according to specific algorithm, draw what he is doing, as follows:

[0069]

[0070] Wherein, p ij Indicates the probability of time interval i and data packet length j;

[0071] P represents the probability of being a certain application.

[0072] After using algorithm analysis, the results are summarized to obtain a result set, and the result set is analyzed to obtain the purpose of the encrypted data packet, and the results are output to the JSON file, and the key information and communication process are also collected and stored in the ES database, and finally summarized.

[0073] In step S140, the analysis results of the unencrypted plaintext data and the encrypted file are summarized, specifically including:

[0074] According to the arrival sequence algorithm, the sorting is carried out again, and the corresponding JSON value is stored in the MongoDB database;

[0075] The old data after basic analysis and secondary analysis is cleaned.

[0076] Specifically, through the basic analysis and secondary analysis, the details of the basic data and the details of the secondary prediction data are obtained, which are JSON data and key data stored in ES, respectively. We need to merge, according to The arrival sequence algorithm is sorted again, and the corresponding JSON value is stored in the MongoDB database. Because the types of protocols are too many, the document type database is used to store this part of data, and the old data after basic analysis and secondary analysis is cleaned, and the storage resources are reduced and wasted. After the summary is completed, the query can be carried out according to different query conditions.

[0077] In step S150, in the process of display, the display is carried out through a web interface, the display is divided into several parts, including protocol details, communication process details, IP communication pairs, deep analysis and the like, and mainly displays time, local IP, opposite IP, protocol type, size and details. In the communication process, the communication times, communication ports, communication times and communication IP are displayed. The protocol details display detailed information, such as the overall process of key exchange of ssl protocol and the like, and the analysis data can be analyzed. For the display of deep analysis, the results analyzed by using the inertia algorithm are mainly displayed. As long as the encrypted data is referenced.

[0078] In the above specific embodiments, the data analysis method based on Markov chain provided by the application comprises the following steps: obtaining to-be-analyzed data, performing basic analysis on the to-be-analyzed data to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the to-be-analyzed data and analysis results of the unencrypted plaintext data; calling a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files; and performing data summarization on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain a summary result. The method can effectively analyze ordinary data and encrypted data, thereby providing effective data support for internal network data security.

[0079] In addition to the above method, the application further provides a data analysis device based on Markov chain, which comprises:

[0080] a data acquisition unit configured to acquire to-be-analyzed data;

[0081] a basic analysis unit configured to perform basic analysis on the to-be-analyzed data to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the to-be-analyzed data and analysis results of the unencrypted plaintext data;

[0082] a secondary analysis unit configured to call a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files;

[0083] a result summarization unit configured to perform data summarization on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain a summary result.

[0084] In some embodiments, obtaining the summary result further comprises:

[0085] visually displaying the summary result.

[0086] In some embodiments, acquiring the to-be-analyzed data specifically comprises:

[0087] Selecting a target network file for uploading;

[0088] Fragmenting the uploaded network file, and recording the current incoming data and the number of fragments;

[0089] Saving the incoming data and the number of fragments, and if the data saving process is interrupted, judging according to the file hash value;

[0090] If the saving fails, resuming the breakpoint, and if the saving succeeds, merging the data, and storing the merged file into the formal file.

[0091] In some embodiments, the data to be analyzed is subjected to basic analysis, specifically including:

[0092] Reading and splitting the data packet file;

[0093] Parsing the protocol information of the unencrypted plaintext that can be obtained, and storing the parsing result in the plaintext database in the form of JSON;

[0094] Storing the encrypted data that needs to be analyzed again into the encrypted database.

[0095] In some embodiments, the encrypted file in the encrypted database is subjected to secondary analysis, specifically including:

[0096] Obtaining the ciphertext, and judging the encryption type of the ciphertext by a Markov chain adjusted based on data, wherein the Markov chain predicts the content of t+1 according to known X1 characteristics and s characteristics;

[0097] After algorithm analysis, the results are summarized to obtain a result set, the result set is analyzed again to obtain the purpose of the encrypted data packet, and the results are output to a JSON file, and the key information and the communication process are also collected and stored in the confidential database.

[0098] In some embodiments, the analysis results of the unencrypted plaintext data and the analysis results of the encrypted file are data summarized, specifically including:

[0099] Sorting according to the arrival sequence algorithm, and then sorting uniformly, and storing the corresponding JSON value into the MongoDB database;

[0100] Cleaning the old data after basic analysis and secondary analysis.

[0101] In the foregoing specific embodiments, the Markov chain-based data analysis device provided by the application obtains to-be-analyzed data, performs basic analysis on the to-be-analyzed data to obtain a plaintext database and an encrypted database; the plaintext database includes unencrypted plaintext data in the to-be-analyzed data and analysis results of the unencrypted plaintext data; a pre-stored secondary analysis program is called to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files; and the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files are data-aggregated to obtain an aggregation result. The device can effectively analyze both common data and encrypted data, thereby providing effective data support for intranet data security.

[0102] Figure 3 An example of a schematic diagram of a physical structure of an electronic device is shown in FIG. 1. Figure 3 As shown in FIG. 1, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 can communicate with each other through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute the method described above.

[0103] In addition, the logical instruction in the memory 330 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0104] On the other hand, the application further provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the methods described above.

[0105] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above method.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0107] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data analysis based on Markov chain, characterized in that, The method comprises: obtaining data to be analyzed; performing basic analysis on the data to be analyzed to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the data to be analyzed and analysis results of the unencrypted plaintext data; calling a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files; performing data aggregation on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain an aggregated result.

2. The Markov chain-based data analysis method according to claim 1, wherein, After obtaining the aggregated result, the method further comprises: performing visual display on the aggregated result.

3. The Markov chain-based data analysis method of claim 1, wherein, Obtaining data to be analyzed specifically comprises: selecting a target network file for uploading; fragmenting the uploaded network file and recording the current incoming data and the number of fragments; performing data saving on the incoming data and the number of fragments, and if the data saving is interrupted in the middle, judging according to a file hash value; if the saving fails, performing breakpoint continuation, and if the saving succeeds, performing data merging and storing the merged file into a formal file.

4. The Markov chain-based data analysis method according to claim 3, wherein, Performing basic analysis on the data to be analyzed specifically comprises: performing packet reading and packet splitting on a packet file; performing analysis on protocol information of the unencrypted plaintext that can be obtained, and storing the analysis results in the plaintext database in the form of JSON; storing encrypted data that needs secondary analysis into the encrypted database.

5. The Markov chain-based data analysis method according to claim 4, wherein, Performing secondary analysis on the encrypted files in the encrypted database specifically comprises: obtaining ciphertext, judging the encryption type of the ciphertext by a Markov chain adjusted based on data, the Markov chain predicting content according to t+1 based on known X1 characteristics and s characteristics; after algorithm analysis, aggregating the results to obtain a result set, analyzing the result set to obtain the purpose of the encrypted packet, and outputting the result to a JSON file, collecting key information and communication processes, and storing them into a confidential database.

6. The Markov chain-based data analysis method according to claim 5, wherein, Performing data aggregation on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files specifically comprises: sorting according to a reach sequence algorithm, then uniformly sorting, and storing corresponding JSON values into a MongoDB database; cleaning up old data after basic analysis and secondary analysis.

7. A Markov chain-based data analysis device, characterized by comprising: The device comprises: a data acquisition unit configured to obtain data to be analyzed; a basic analysis unit configured to perform basic analysis on the data to be analyzed to obtain a plaintext database and an encrypted database; the plaintext database comprises unencrypted plaintext data in the data to be analyzed and analysis results of the unencrypted plaintext data; a secondary analysis unit configured to call a pre-stored secondary analysis program to perform secondary analysis on encrypted files in the encrypted database to obtain analysis results of the encrypted files; a result aggregation unit configured to perform data aggregation on the analysis results of the unencrypted plaintext data and the analysis results of the encrypted files to obtain an aggregated result.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method according to any one of claims 1 to 6 when executing the program. The processor implements the method according to any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which when executed by a processor, implements the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor, implements the method of any one of claims 1 to 6.

Images