Data information collection and analysis method based on big data

Abstract

The present invention belongs to the field of data processing technology, and discloses a data information collection and analysis method based on big data, the method comprising: storing collected network big data; selecting data from the stored network big data, and performing semantic retrieval on the selected data through a semantic retrieval system to determine relevant data with semantics close to the selected data; comparing the number of relevant data with a preset data number threshold, if the number of relevant data is greater than or equal to the preset data number threshold, the generation time of the selected data and the generation time of the relevant data are compared, with time as the axis, and the time on the time axis is respectively associated with the selected data and the relevant data and marked as event formation data, and an event corresponding to the event formation data is generated; collecting coverage information of the event formation data, generating an event impact coefficient according to the coverage information, and generating a corresponding impact mark for the corresponding event according to the size of the event impact coefficient.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a data information collection and analysis method based on big data. Background Art

[0002] Big data has the following characteristics: large volume, the data scale is very large, usually measured in TB, PB, or even EB. The data volume far exceeds the processing capacity of traditional data processing technology, low value density, there is a lot of useless data in big data, and data cleaning and screening are needed to improve the value density of the data, diversity, big data comes from various sources, including structured data, semi-structured data and unstructured data, such as text, images, videos, etc., with diversity characteristics, timeliness: big data is real-time and immediate, and data needs to be processed and analyzed in a timely manner to make timely decisions.

[0003] In today's era of high-speed data transmission, when network events generate data that spreads rapidly, the data has the characteristics of large volume, diversity, and value density. In existing technologies, relevant functional departments will only issue warnings after the data has been widely disseminated and caused a certain impact. The processing time is short, resulting in poor processing effects on the events corresponding to the data.

[0004] In view of this, the inventors of the present application invented a data information collection and analysis method based on big data. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a data information collection and analysis method based on big data.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A data information collection and analysis method based on big data, the method comprising:

[0008] Storing the collected network big data; network big data includes articles, documents, videos and other network data carriers with information dissemination functions;

[0009] Select data from the stored network big data, and perform semantic retrieval on the selected data through a semantic retrieval system to determine related data with similar semantics to the selected data;

[0010] The amount of relevant data is compared with a preset data amount threshold. If the amount of relevant data is greater than or equal to the preset data amount threshold, the time when the selected data is generated and the time when the relevant data is generated are associated with the selected data and the relevant data on the time axis, and marked as event formation data, and an event corresponding to the event formation data is generated;

[0011] Collect coverage information of event-generated data, generate event impact coefficients based on the coverage information, and generate corresponding impact tags for corresponding events based on the magnitude of the event impact coefficients;

[0012] Collect comment data corresponding to each event-forming data corresponding to the event within a unit time, obtain i fields in the comment data, where i is an integer greater than or equal to 1, generate field property coefficients based on the properties corresponding to different fields, and determine the nature of the event based on the field property coefficients;

[0013] Obtain the impact tags and properties corresponding to t events, determine whether to generate an early warning instruction, and send the early warning instruction to the relevant functional departments.

[0014] Furthermore, the event impact coefficient generation step includes:

[0015] The coverage information includes the number of covered areas and the number of people covered. The number of covered areas is marked as R t , the number of people covered is marked as C t , preset coverage area number R' t and the preset number of people covered C' t Substitute into the calculation formula:

[0016] SJ t is the event impact coefficient, γ1 and γ2 are the weight factors corresponding to the number of covered areas and the number of covered people, respectively, and their values ​​are both greater than 0. t is the number of corresponding events, t = 1, 2, 3...t, where t is an integer greater than or equal to 1.

[0017] Furthermore, the impact markers include a large impact marker, a medium impact marker, and a small impact marker. The specific generation process includes:

[0018] Set the event impact coefficient gradient threshold, and set the event impact coefficient SJ t Substitute the event impact coefficient gradient threshold, if the event impact coefficient SJ t If the event impact coefficient SJ is greater than or equal to the maximum value of the gradient threshold, a high impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the event impact coefficient SJ is less than the maximum value of the gradient threshold and greater than the minimum value of the gradient threshold, a medium impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the value is less than or equal to the minimum value of the event impact coefficient gradient threshold, a low impact mark is generated for the corresponding event.

[0019] Furthermore, the number of covered people is determined according to the data reading volume generated by the event corresponding to the event, and the number of covered areas is determined according to the corresponding covered administrative areas to which the IP addresses of the covered people belong.

[0020] Furthermore, the field property coefficient generation step includes: analyzing the i fields in the bag-of-words database to determine the i field properties, and determining the corresponding event formation data properties based on the i field properties;

[0021] The properties include benign, neutral and malignant;

[0022] The number of benign fields in the field corresponding to each event data is marked as Z1, the number of neutral fields in the field corresponding to the event data is marked as Z2, and the number of malignant fields in the field corresponding to the event data is marked as Z3;

[0023] Calculated ZD in the formula t is the field property coefficient, Z4 is the total reading volume of the data generated by the event corresponding to the event; α1, α2, and α3 are all preset proportional coefficients.

[0024] Furthermore, the step of determining the nature of the event according to the nature of the data generated by each event includes:

[0025] Set the field property coefficient threshold, and set the field property coefficient ZD t Substitute the field property coefficient threshold, if the field property coefficient ZD t If the field property coefficient ZD is greater than or equal to the field property coefficient threshold, the corresponding event property is marked as a malicious event; if the field property coefficient ZD t If the value is less than the field property coefficient threshold, the corresponding event property is marked as a general event.

[0026] Furthermore, the early warning instruction generation logic includes:

[0027] The early warning instructions include first-level early warning instructions and second-level early warning instructions; if the same event has both a large impact mark and a malicious event mark, a first-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a medium impact mark, a second-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a small impact mark, a general event and a small impact mark, a general event and a large impact mark, and a general event and a medium impact mark, no early warning instruction is generated for the corresponding event.

[0028] Furthermore, the event impact coefficient SJ corresponding to the unit time after the early warning instruction is generated is obtained. t and field property coefficient ZD t , the event impact coefficient SJ obtained t Establish an influence coefficient set and obtain the field property coefficient ZD t Establish a set of field property coefficients;

[0029] Calculate the average impact coefficient of events in the impact coefficient set With the dispersion coefficient Q,

[0030]

[0031] Where m = {1, 2, 3...m}, m represents the event impact coefficient SJ in the impact coefficient set t Quantity, E n Indicates the different event impact coefficients SJ in the impact coefficient set t , Indicates the event impact coefficient SJ within the impact coefficient set t average value;

[0032] Calculate the average value of the field property coefficient set With the coefficient of dispersion P,

[0033]

[0034] Where s={1,2,3……s}, s represents the field property coefficient ZD in the field property coefficient set t Quantity, W j Indicates the different field property coefficients ZD in the field property coefficient set t , Indicates the field property coefficient ZD in the field property coefficient set t average value;

[0035] If the average If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is less than the discrete coefficient Q threshold, the event is marked as an impact expansion mark; if the average value If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is greater than the discrete coefficient Q threshold, the event is marked as an impact mitigation mark; if the average value If the value is less than the maximum value of the event impact coefficient gradient threshold, the event is marked as an impact fading mark.

[0036] If the average If the value is greater than or equal to the field property coefficient threshold, and the discrete coefficient P is less than the discrete coefficient P threshold, then the event is marked as a property expansion mark; if the average value If the value is less than the field property coefficient threshold, the event is marked as a property improvement mark;

[0037] If any event has both the impact expansion flag and the nature expansion flag, an early warning upgrade instruction will be generated for the event;

[0038] If any event has both the impact alleviation flag and the property improvement flag and the impact disappearance flag and the property improvement flag, a processing success instruction is generated for the event.

[0039] The technical effects and advantages of the data information collection and analysis method based on big data of the present invention are as follows:

[0040] By collecting and analyzing relevant big data during the development of an incident, we can provide data support for the decision-making of relevant functional departments before the incident deteriorates and spreads on a large scale, and generate corresponding early warning instructions to relevant functional departments in a timely manner, so that relevant functional departments have ample time to make handling plans and reduce the impact of the incident.

[0041] By continuously analyzing the real-time data generated by events after being handled by relevant functional departments, relevant functional departments can quickly grasp the development of events after handling, which is timely and convenient for relevant functional departments to improve and formulate more appropriate handling strategies, and further avoid the expansion of the scope of the deterioration of events. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of a data information collection and analysis system for big data in the first embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of a data information collection and analysis system for big data in the second embodiment of the present invention;

[0044] Figure 3 Schematic diagram of a data information collection and analysis system for big data in another embodiment of the present invention;

[0045] Figure 4 Schematic diagram of the data information collection and analysis method of big data of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] Example 1

[0048] See also Figure 1 As shown, the data information collection and analysis system based on big data described in this embodiment includes a big data collection module 1, a semantic retrieval module 2, an event establishment module 3, an event impact analysis module 4, an event nature analysis module 5, and an event processing module 6.

[0049] The big data collection module 1 is used to collect network big data and store the collected network big data. In order to facilitate the analysis of big data, it can be stored in a network storage server. Network big data includes data on portal servers, which include web forum servers, short video servers, etc.; network big data includes articles, documents, videos and other network data carriers with information dissemination functions.

[0050] Semantic retrieval module 2 is used to select data from the stored network big data and perform semantic retrieval on the selected data through the semantic retrieval system to determine related data with similar semantics to the selected data. The selected data can be documents, video tags, fields, such as title, abstract, text, author information, etc. This embodiment takes the selected data as an article as an example;

[0051] Event establishment module 3 compares the amount of relevant data with the preset data amount threshold. If the amount of relevant data is greater than or equal to the preset data amount threshold, the selected data generation time and the relevant data generation time are associated with the selected data and the relevant data on the time axis, with time as the axis, and marked as event formation data, and an event corresponding to the event formation data is generated.

[0052] The event impact analysis module 4 collects coverage information of event-generated data, generates event impact coefficients according to the coverage information, and generates corresponding impact tags for corresponding events according to the size of the event impact coefficients.

[0053] Impact markers include large impact markers, medium impact markers, and small impact markers. The specific generation process includes:

[0054] The coverage range information includes the number of covered areas and the number of covered people. The number of covered areas is the number of corresponding covered administrative areas, which can be the number of provincial administrative regions or the number of municipal administrative regions, etc. The number of covered areas is marked as R t , the number of people covered is marked as C t , preset coverage area number R' t and the preset number of people covered C' t Substitute into the calculation formula:

[0055] SJ t is the event impact coefficient, γ1 and γ2 are the weight factors corresponding to the number of covered areas and the number of covered people, respectively, and their values ​​are both greater than 0. t is the number of corresponding events, t = 1, 2, 3...t, t is an integer greater than or equal to 1. The weight factor is used to balance the proportion of each data in the formula, thereby promoting the accuracy of the calculation results.

[0056] It should be noted that the event impact coefficient SJ tThe larger the performance value is, the greater the impact range caused by the time is.

[0057] The number of people covered can be determined based on the data reading volume generated by the event corresponding to the event, and the number of covered areas can be determined based on the corresponding covered administrative areas according to the IP addresses of the covered people.

[0058] Set the event impact coefficient gradient threshold, and set the event impact coefficient SJ t Substitute the event impact coefficient gradient threshold, if the event impact coefficient SJ t If the event impact coefficient SJ is greater than or equal to the maximum value of the gradient threshold, a high impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the event impact coefficient SJ is less than the maximum value of the gradient threshold and greater than the minimum value of the gradient threshold, a medium impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the value is less than or equal to the minimum value of the event impact coefficient gradient threshold, a low impact mark is generated for the corresponding event.

[0059] The event nature analysis module 5 collects the comment data corresponding to each event-forming data corresponding to the event within the unit time, obtains i fields in the comment data, i is an integer greater than or equal to 1, and the field is a component of the comment data, which can be a word or a phrase. The field nature coefficient is generated according to the nature corresponding to different fields, and the event nature is determined according to the field nature coefficient.

[0060] The nature of an event can be benign, neutral, or malignant. The process for determining the nature of an event is as follows:

[0061] Analyze i fields in the bag-of-words database to determine the properties of i fields, determine the data properties of the corresponding events based on the properties of i fields, and determine the properties of the events based on the data properties of each event. The properties of the fields in the bag-of-words database can be annotated manually or by related recognition algorithms;

[0062] The number of benign fields in the field corresponding to each event data is marked as Z1, the number of neutral fields in the field corresponding to the event data is marked as Z2, and the number of malignant fields in the field corresponding to the event data is marked as Z3.

[0063] Calculated according to the formula ZD in the formula t is the field property coefficient, Z4 is the total reading volume of the data corresponding to the event; α1, α2, and α3 are all preset proportional coefficients. It should be noted that the field property coefficient ZD t The larger the value is, the worse the impact of the corresponding event is, and vice versa.

[0064] Set the field property coefficient threshold, and set the field property coefficient ZDt Substitute the field property coefficient threshold, if the field property coefficient ZD t If the field property coefficient ZD is greater than or equal to the field property coefficient threshold, the corresponding event property is marked as a malicious event; if the field property coefficient ZD t If the value is less than the field property coefficient threshold, the corresponding event property is marked as a general event.

[0065] The event processing module 6 obtains the impact tags and properties corresponding to the t events, determines whether to generate an early warning instruction, and sends the early warning instruction to the relevant functional departments.

[0066] The early warning instruction generation logic includes:

[0067] The early warning instructions include first-level early warning instructions and second-level early warning instructions; if the same event has both a large impact mark and a malicious event mark, a first-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a medium impact mark, a second-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a small impact mark, a general event and a small impact mark, a general event and a large impact mark, and a general event and a medium impact mark, no early warning instruction is generated for the corresponding event.

[0068] It should be noted that the generation of first-level warning instructions and second-level warning instructions is from high to low according to the severity of the impact of the event. When a first-level warning instruction is issued, it means that the event has reached the stage that requires immediate processing. If it is not processed in time, there is a risk of further deterioration. When a second-level warning instruction appears, if the number of processing personnel is insufficient, the frequency of event data collection corresponding to the event can be increased, that is, the monitoring of the event can be strengthened, and its subsequent development status can be monitored, that is, attention should be paid to the changes in the field property coefficient and the event impact coefficient corresponding to the event. Between the upgrade of the corresponding warning instruction to and the warning instruction, the processing strategy should be specified in time to avoid the expansion of the scope of the event deterioration.

[0069] By collecting and analyzing relevant big data during the development of an incident, we can provide data support for the decision-making of relevant functional departments before the incident deteriorates and spreads on a large scale, and generate corresponding early warning instructions to relevant functional departments in a timely manner, so that relevant functional departments have ample time to make handling plans and reduce the impact of the incident.

[0070] Example 2

[0071] See also Figure 2 As shown, this embodiment further improves the design based on the first embodiment. The data information collection and analysis system based on big data provided by this embodiment also includes an event development trend analysis module 7, which is used to obtain the event impact coefficient SJ corresponding to the unit time after the early warning instruction is generated. t and field property coefficient ZDt , the event impact coefficient SJ obtained t Establish an influence coefficient set and obtain the field property coefficient ZD t Establish a set of field property coefficients.

[0072] First calculate the average value of the event impact coefficient in the impact coefficient set With the dispersion coefficient Q,

[0073]

[0074] Where m = {1, 2, 3...m}, m represents the event impact coefficient SJ in the impact coefficient set t Quantity, E n Indicates the different event impact coefficients SJ in the impact coefficient set t , Indicates the event impact coefficient SJ within the impact coefficient set t average value.

[0075] Calculate the average value of the field property coefficient set With the coefficient of dispersion P,

[0076]

[0077] Where s={1,2,3……s}, s represents the field property coefficient ZD in the field property coefficient set t Quantity, W j Indicates the different field property coefficients ZD in the field property coefficient set t , Indicates the field property coefficient ZD in the field property coefficient set t average value.

[0078] If the average If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is less than the discrete coefficient Q threshold, the event is marked as an impact expansion mark; if the average value If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is greater than the discrete coefficient Q threshold, the event is marked as an impact mitigation mark; if the average value If the value is less than the maximum value of the event impact coefficient gradient threshold, the event is marked as an impact fading mark.

[0079] If the average If the value is greater than or equal to the field property coefficient threshold, and the discrete coefficient P is less than the discrete coefficient P threshold, then the event is marked as a property expansion mark; if the average value If the value is less than the field property coefficient threshold, the event is marked as a property improvement mark.

[0080] If any event has both an impact expansion mark and a nature expansion mark, an early warning upgrade instruction will be generated for the event, indicating that the event is developing in an unfavorable direction and that the handling strategy needs to be changed in a timely manner. The development of the event after handling needs to be understood in a timely manner, so that relevant functional departments can form data analysis based on the event and formulate more appropriate handling strategies to further avoid the expansion of the scope of the event.

[0081] If any event has both the impact alleviation flag and the property improvement flag, as well as the impact fading flag and the property improvement flag, a successful processing instruction is generated for the event, indicating that time is moving in the direction of improvement.

[0082] By continuously analyzing the real-time data generated by events after being handled by relevant functional departments, relevant functional departments can quickly grasp the development of events after handling, which is timely and convenient for relevant functional departments to improve and formulate more appropriate handling strategies, and further avoid the expansion of the scope of the deterioration of events.

[0083] Example 3

[0084] See also Figure 4 As shown, for parts not described in detail in this embodiment, please refer to the description of the above embodiment. A data information collection and analysis method based on big data is provided, and the method includes:

[0085] Storing the collected network big data; network big data includes articles, documents, videos and other network data carriers with information dissemination functions;

[0086] Select data from the stored network big data, and perform semantic retrieval on the selected data through a semantic retrieval system to determine related data with similar semantics to the selected data;

[0087] The amount of relevant data is compared with a preset data amount threshold. If the amount of relevant data is greater than or equal to the preset data amount threshold, the time when the selected data is generated and the time when the relevant data is generated are associated with the selected data and the relevant data on the time axis, and marked as event formation data, and an event corresponding to the event formation data is generated;

[0088] Collect coverage information of event-generated data, generate event impact coefficients based on the coverage information, and generate corresponding impact tags for corresponding events based on the magnitude of the event impact coefficients;

[0089] Collect comment data corresponding to each event-forming data corresponding to the event within a unit time, obtain i fields in the comment data, where i is an integer greater than or equal to 1, generate field property coefficients based on the properties corresponding to different fields, and determine the nature of the event based on the field property coefficients;

[0090] Obtain the impact tags and properties corresponding to t events, determine whether to generate an early warning instruction, and send the early warning instruction to the relevant functional departments.

[0091] Furthermore, the event impact coefficient generation step includes:

[0092] The coverage information includes the number of covered areas and the number of people covered. The number of covered areas is marked as R t , the number of people covered is marked as C t , preset coverage area number R' t and the preset number of people covered C' t Substitute into the calculation formula:

[0093] SJ t is the event impact coefficient, γ1 and γ2 are the weight factors corresponding to the number of covered areas and the number of covered people, respectively, and their values ​​are both greater than 0. t is the number of corresponding events, t = 1, 2, 3...t, where t is an integer greater than or equal to 1.

[0094] Furthermore, the impact markers include a large impact marker, a medium impact marker, and a small impact marker. The specific generation process includes:

[0095] Set the event impact coefficient gradient threshold, and set the event impact coefficient SJ t Substitute the event impact coefficient gradient threshold, if the event impact coefficient SJ t If the event impact coefficient SJ is greater than or equal to the maximum value of the gradient threshold, a high impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the event impact coefficient SJ is less than the maximum value of the gradient threshold and greater than the minimum value of the gradient threshold, a medium impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the value is less than or equal to the minimum value of the event impact coefficient gradient threshold, a low impact mark is generated for the corresponding event.

[0096] Furthermore, the number of covered people is determined according to the data reading volume generated by the event corresponding to the event, and the number of covered areas is determined according to the corresponding covered administrative areas to which the IP addresses of the covered people belong.

[0097] Furthermore, the field property coefficient generation step includes: analyzing the i fields in the bag-of-words database to determine the i field properties, and determining the corresponding event formation data properties based on the i field properties;

[0098] The properties include benign, neutral and malignant;

[0099] The number of benign fields in the field corresponding to each event data is marked as Z1, the number of neutral fields in the field corresponding to the event data is marked as Z2, and the number of malignant fields in the field corresponding to the event data is marked as Z3;

[0100] Calculated ZD in the formula t is the field property coefficient, Z4 is the total reading volume of the data generated by the event corresponding to the event; α1, α2, and α3 are all preset proportional coefficients.

[0101] Furthermore, the step of determining the nature of the event according to the nature of the data generated by each event includes:

[0102] Set the field property coefficient threshold, and set the field property coefficient ZD t Substitute the field property coefficient threshold, if the field property coefficient ZD t If the field property coefficient ZD is greater than or equal to the field property coefficient threshold, the corresponding event property is marked as a malicious event; if the field property coefficient ZD t If the value is less than the field property coefficient threshold, the corresponding event property is marked as a general event.

[0103] Furthermore, the early warning instruction generation logic includes:

[0104] The early warning instructions include first-level early warning instructions and second-level early warning instructions; if the same event has both a large impact mark and a malicious event mark, a first-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a medium impact mark, a second-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a small impact mark, a general event and a small impact mark, a general event and a large impact mark, and a general event and a medium impact mark, no early warning instruction is generated for the corresponding event.

[0105] Furthermore, the event impact coefficient SJ corresponding to the unit time after the early warning instruction is generated is obtained. t and field property coefficient ZD t , the event impact coefficient SJ obtained t Establish an influence coefficient set and obtain the field property coefficient ZD t Establish a set of field property coefficients;

[0106] Calculate the average impact coefficient of events in the impact coefficient set With the dispersion coefficient Q,

[0107]

[0108] Where m = {1, 2, 3...m}, m represents the event impact coefficient SJ in the impact coefficient set t Quantity, E nIndicates the different event impact coefficients SJ in the impact coefficient set t , Indicates the event impact coefficient SJ within the impact coefficient set t average value;

[0109] Calculate the average value of the field property coefficient set With the coefficient of dispersion P,

[0110]

[0111] Where s={1,2,3……s}, s represents the field property coefficient ZD in the field property coefficient set t Quantity, W j Indicates the different field property coefficients ZD in the field property coefficient set t , Indicates the field property coefficient ZD in the field property coefficient set t average value;

[0112] If the average If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is less than the discrete coefficient Q threshold, the event is marked as an impact expansion mark; if the average value If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is greater than the discrete coefficient Q threshold, the event is marked as an impact mitigation mark; if the average value If the value is less than the maximum value of the event impact coefficient gradient threshold, the event is marked as an impact fading mark.

[0113] If the average If the value is greater than or equal to the field property coefficient threshold, and the discrete coefficient P is less than the discrete coefficient P threshold, then the event is marked as a property expansion mark; if the average value If the value is less than the field property coefficient threshold, the event is marked as a property improvement mark;

[0114] If any event has both the impact expansion flag and the nature expansion flag, an early warning upgrade instruction will be generated for the event;

[0115] If any event has both the impact alleviation flag and the property improvement flag and the impact disappearance flag and the property improvement flag, a processing success instruction is generated for the event.

[0116] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0117] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only one type. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0123] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0125] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data information collection and analysis method based on big data, characterized in that: The method comprises: Store the collected network big data; Select data from the stored network big data, and perform semantic retrieval on the selected data through a semantic retrieval system to determine related data with similar semantics to the selected data; The amount of relevant data is compared with a preset data amount threshold. If the amount of relevant data is greater than or equal to the preset data amount threshold, the time when the selected data is generated and the time when the relevant data is generated are associated with the selected data and the relevant data on the time axis, and marked as event formation data, and an event corresponding to the event formation data is generated; Collect coverage information of event-generated data, generate event impact coefficients based on the coverage information, and generate corresponding impact tags for corresponding events based on the magnitude of the event impact coefficients; The steps for generating the event impact coefficient include: The coverage information includes the number of covered areas and the number of people covered. The number of covered areas is marked as R t , the number of people covered is marked as C t , preset coverage area number R' t and the preset number of people covered C' t Substitute into the calculation formula: SJ t is the event impact coefficient, 、 are the weight factors corresponding to the number of covered areas and the number of covered people, both of which are greater than 0; t is the number of corresponding events, and t is an integer greater than or equal to 1; The number of people covered is determined based on the data reading volume of the event corresponding to the event, and the number of covered areas is determined based on the corresponding covered administrative areas according to the IP addresses of the covered people; Collect the comment data corresponding to each event in the unit time, obtain i fields in the comment data, generate field property coefficients according to the properties corresponding to different fields, and determine the nature of the event according to the field property coefficients; The step of generating the field property coefficient includes: analyzing the i fields in the bag-of-words database, determining the i field properties, and determining the data properties of the corresponding event based on the i field properties; The properties include benign, neutral and malignant; The number of benign fields in the field corresponding to each event data is marked as Z1, the number of neutral fields in the field corresponding to the event data is marked as Z2, and the number of malignant fields in the field corresponding to the event data is marked as Z3; Calculated ZD in the formula t is the field property coefficient, Z4 is the total reading volume of the data generated by the event corresponding to the event; α1, α2, and α3 are all preset proportional coefficients; Obtain the impact tags and properties corresponding to t events, determine whether to generate an early warning instruction, and send the early warning instruction to the relevant functional departments.

2. The data information collection and analysis method based on big data according to claim 1 is characterized in that: Impact markers include large impact markers, medium impact markers, and small impact markers. The specific generation process includes: Set the event impact coefficient gradient threshold, and set the event impact coefficient SJ t Substitute the event impact coefficient gradient threshold, if the event impact coefficient SJ t If the event impact coefficient SJ is greater than or equal to the maximum value of the gradient threshold, a high impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the event impact coefficient SJ is less than the maximum value of the gradient threshold and greater than the minimum value of the gradient threshold, a medium impact mark is generated for the corresponding event; if the event impact coefficient SJ t If the value is less than or equal to the minimum value of the event impact coefficient gradient threshold, a low impact mark is generated for the corresponding event.

3. The data information collection and analysis method based on big data according to claim 2 is characterized in that: The steps to determine the nature of an event based on the nature of the data generated by each event include: Set the field property coefficient threshold, and set the field property coefficient ZD t Substitute the field property coefficient threshold, if the field property coefficient ZD t If the field property coefficient ZD is greater than or equal to the field property coefficient threshold, the corresponding event property is marked as a malicious event; if the field property coefficient ZD t If the value is less than the field property coefficient threshold, the corresponding event property is marked as a general event.

4. The data information collection and analysis method based on big data according to claim 3 is characterized in that: The early warning instruction generation logic includes: The early warning instructions include first-level early warning instructions and second-level early warning instructions; if the same event has both a large impact mark and a malicious event mark, a first-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a medium impact mark, a second-level early warning instruction is generated for the corresponding event; if the same event has both a malicious event and a small impact mark, a general event and a small impact mark, a general event and a large impact mark, and a general event and a medium impact mark, no early warning instruction is generated for the corresponding event.

5. The data information collection and analysis method based on big data according to claim 4 is characterized in that: Get the event impact coefficient SJ corresponding to the unit time after the early warning instruction is generated t and field property coefficient ZD t , the event impact coefficient SJ obtained t Establish an influence coefficient set and obtain the field property coefficient ZD t Establish a set of field property coefficients; Calculate the average impact coefficient of events in the impact coefficient set With the dispersion coefficient Q, Where m is an integer greater than or equal to 2, and m represents the event impact coefficient SJ in the impact coefficient set t Quantity, E n Indicates the different event impact coefficients SJ in the impact coefficient set t , Indicates the event impact coefficient SJ within the impact coefficient set t average value; Calculate the average value of the field property coefficient set With the coefficient of dispersion P, Where s is an integer greater than or equal to 2, and s represents the field property coefficient ZD in the field property coefficient set. t Quantity, W j Indicates the different field property coefficients ZD in the field property coefficient set t , Indicates the field property coefficient ZD in the field property coefficient set t average value; If the average If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is less than the discrete coefficient Q threshold, the event is marked as an impact expansion mark; If the average If the event impact coefficient gradient threshold is greater than or equal to the maximum value, and the discrete coefficient Q is greater than the discrete coefficient Q threshold, the event is marked as an impact mitigation mark; If the average If the value is less than the maximum value of the event impact coefficient gradient threshold, the event is marked as an impact fading mark; If the average If the value is greater than or equal to the field property coefficient threshold, and the discrete coefficient P is less than the discrete coefficient P threshold, then the event is marked as a property expansion mark; if the average value If the value is less than the field property coefficient threshold, the event is marked as a property improvement mark; If any event has both the impact expansion flag and the nature expansion flag, an early warning upgrade instruction will be generated for the event; If any event has both an impact alleviation flag and a property improvement flag or an impact fading flag and a property improvement flag, a successful processing instruction is generated for the event.

Images