Information collection and analysis system based on big data

Abstract

The invention relates to the technical field of information collection and analysis, and discloses a big data-based information collection and analysis system, which comprises a data preprocessing system, a data analysis system, a data feedback module and a cloud database, the data preprocessing system comprises an information parameter data extraction module, an information parameter data transmission module and an information parameter data screening module. In the using process, the information parameter data screening module is adopted for preliminarily screening information parameter data of big data; the information parameter data can be confirmed again through the target information confirmation module, and the confirmed information parameter data is automatically verified through the index information verification module, so that the processing efficiency of a large amount of information parameter data can be improved through a step-by-step batch processing mode; and effective information data can be accurately and timely extracted.

Description

Technical Field

[0001] The present invention relates to the technical field of information collection and analysis, and in particular to an information collection and analysis system based on big data. Background Art

[0002] A network is a graph consisting of several nodes and links connecting these nodes, representing many objects and their connections. The network has the characteristics of resource sharing, fast information transmission, improved system reliability, easy distributed processing and integrated information services. The network is an important invention in the history of human development, which brings people wonderful enjoyment and promotes the development of science and technology and human society. The main function of the network is resource sharing. Shared resources include software resources, hardware resources and various data resources stored in public databases. Online users can share these resources partially or completely, so that resources in the network can be shared and divided into work and cooperate, thereby greatly improving the utilization rate of system resources.

[0003] In the prior art, with the advent of the big data era, the amount of information in the network has shown an exponential growth. In the collection, analysis and processing of big data generated by a certain part of the information, there will be poor results in extracting valuable information data, resulting in slow processing of large amounts of data.

[0004] After searching, it is found that a Chinese patent document discloses an information collection and analysis system based on big data (publication number: CN114817817A). The invention discloses an information collection and analysis system based on big data, which belongs to the field of information collection and analysis technology, and is specifically an information collection and analysis system based on big data. The server is connected with a user terminal, a data acquisition module, a data storage module, a data analysis management module, a retrieval module, a data combing module and a data transmission module. The data acquisition module is used to collect basic data information of the website, and the data transmission module is used to transmit the basic data information of the website collected by the data acquisition module to the server. The basic data information of the website is collected by the data acquisition module and transmitted to the server. The data combing module classifies and integrates the basic data information of the website collected, and stores it through the data storage module. The stored data information is analyzed and managed through the data analysis management module. When the user applies to the server, the corresponding interface and information pop up, and the user can operate the interface, which is convenient for the online training platform to use. However, there are still the following defects:

[0005] Although the above-mentioned information collection and analysis system based on big data realizes storage through the data storage module, analyzes and manages the stored data information through the data analysis management module, and pops up the corresponding interface and information through the user's application to the server, the user can operate the interface to facilitate the use of the online training platform, but it still has the problem of poor effect in extracting valuable information data in the big data collection and analysis processing generated by a certain part of the information, resulting in slow processing of large amounts of data. Summary of the invention

[0006] The purpose of the present invention is to provide an information collection and analysis system based on big data, which solves the problem in the prior art that with the advent of the big data era, the amount of information in the network shows an exponential growth, and in the big data collection and analysis processing generated for a certain part of the information, there will be poor results in extracting valuable information data, resulting in slow processing of large amounts of data.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] An information collection and analysis system based on big data, comprising an information processing system, wherein the information processing system comprises a data preprocessing system, a data analysis system, a data feedback module, and a cloud database;

[0009] The data preprocessing system includes an information parameter data extraction module, an information parameter data transmission module and an information parameter data screening module, wherein the parameter data extraction module is electrically connected to the information parameter data transmission module and the information parameter data screening module, and the information parameter data screening module is electrically connected to the data analysis system;

[0010] The data analysis system comprises an indicator target information confirmation module and an indicator information verification module, wherein the target information confirmation module is electrically connected to the indicator information verification module, and the indicator information verification module is electrically connected to the data feedback module.

[0011] Preferably, the function of the information parameter data extraction module is to extract information data of the same type, and determine three elements of each information data during the extraction process, wherein the three elements include time parameter data of the information data, identity parameter data of the information data, and security parameter data of the information data;

[0012] The function of the information parameter data transmission module is to receive the generated information parameter data in real time, and write the information parameter data to the information parameter data extraction module through the set writing number threshold method and writing time threshold method.

[0013] Preferably, the information parameter data transmission module writes the information parameter data into the information parameter data extraction module in the following specific steps:

[0014] S1: setting corresponding number thresholds and time thresholds for the write number threshold mode and the write time threshold;

[0015] S2.1.1: When the information parameter data transmission module stores information parameter data and reaches the write times threshold;

[0016] S2.1.2: writing the information parameter data stored in the transmission module into the information parameter data extraction module;

[0017] S2.2.1: When the information parameter data transmission module stores the information parameter data and reaches the write time threshold;

[0018] S2.2.2: Write the information parameter data stored in the transmission module into the information parameter data extraction module.

[0019] Preferably, the function of the information parameter data screening module is to perform preliminary screening of the information parameter data collected in the information parameter data extraction module, and the specific screening indicators are:

[0020] (1) Eliminate information parameter data that is missing from information parameter data, time parameter data, and identity parameter data;

[0021] (2) Eliminate information parameter data whose content does not meet the basic standards for such information data;

[0022] (3) Eliminate information parameter data that does not meet the specified indicators;

[0023] The specified indicators include at least one of text parameter data, image parameter data, data parameter data and audio parameter data.

[0024] Preferably, the function of the target information confirmation module is to determine the target information parameters of the information parameter data after preliminary screening by the information parameter data screening module by inputting at least one indicator information and using the indicator information as a screening indicator, and to form a target information parameter data set for the acquired target information parameter data set.

[0025] Preferably, the specific steps of the indicator target information confirmation module obtaining the target information parameter data set are:

[0026] Step 1: Determine the categories of text parameter data, image parameter data, data parameter data and audio parameter data;

[0027] Step 2: Setting at least one target information parameter for different information parameter data;

[0028] Step 3: Screening the text parameter data and the data parameter data with at least one determined target information parameter to determine the target information parameter data;

[0029] Step 4: construct a comparison model for the image parameter data, standardize the image parameter data, compare it with at least one target information, and determine the target information parameter data for the image parameter data with high adaptability;

[0030] Step 5: for the audio parameter data, perform standardization processing on the audio parameter data, compare it with at least one target information, obtain the audio parameter data that has been successfully compared, and determine the target information parameter data;

[0031] Step 6: After determining the target information parameter data, aggregate them to obtain the target information parameter data set.

[0032] Preferably, the function of the indicator information verification module is to receive the target information parameter data set obtained by the target information confirmation module, extract the target information parameter data in the target information parameter data set according to the set extraction threshold, and perform comparison and verification through the verification and comparison model according to the text parameter data, image parameter data, data parameter data and audio parameter data categories, and transmit the verified target information parameter data set to the data feedback module.

[0033] Preferably, the function of the data feedback module is to receive the target information parameter data set after verification by the indicator information verification module, compress the target information parameter data set, and transmit the compressed target information parameter data set to the interactive terminal through one or more of the 2G, 3G, 4G, 5G networks, WIFI, and Bluetooth.

[0034] Preferably, the cloud database functions to store the information parameter data acquired by the information parameter data screening module in the information processing system.

[0035] Preferably, the specific steps of the information processing system to complete information collection and analysis are:

[0036] (1) First, the information parameter data transmission module receives the information parameter data;

[0037] (2) the information parameter data transmission module writes the information parameter data to the information parameter data extraction module by means of a write count threshold method and a write time threshold method;

[0038] (3) The information parameter data extraction module extracts information data of the same type and transmits it to the information parameter data screening module;

[0039] (4) The information parameter data screening module performs preliminary screening of the information parameter data collected by the information parameter data extraction module;

[0040] (5) The target information confirmation module receives the information parameter data after screening, confirms it, and obtains the target information parameter data set;

[0041] (6) The indicator information verification module receives the target information parameter data set and completes verification of the target information parameter data set;

[0042] (7) The data feedback module exchanges the verified target information parameter data set to the interactive terminal.

[0043] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0044] 1. During use, the present invention first uses an information parameter data screening module to perform preliminary screening on the information parameter data of big data, and can confirm the information parameter data again through the target information confirmation module. At the same time, the confirmed information parameter data is automatically verified through the indicator information verification module. In this way, through the gradual batch processing method, the efficiency of a large amount of information parameter data can be improved, and effective information data can be extracted accurately and timely.

[0045] 2. During use, the present invention adopts a data preprocessing system to write information parameter data into an information parameter data extraction module using a write count threshold method and a write time threshold method. In this way, it can cope with a large amount of data processing, and then when writing information parameter data, it can reduce the system load by reducing the number of writes, further improving the timeliness of the information collection process of big data and reducing the loss of information parameter data.

[0046] 3. During use, the present invention can respond to the text parameter data, image parameter data, data parameter data and audio parameter data generated, and can automatically analyze and process the corresponding information parameter data to obtain valid information parameter data. At the same time, this method can more accurately obtain valid information parameter data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the structure of the information processing system in the present invention;

[0048] Figure 2 A schematic diagram of specific steps for the information parameter data transmission module of the present invention to write the information parameter data into the information parameter data extraction module;

[0049] Figure 3A schematic diagram of specific steps for the indicator target information confirmation module in the present invention to obtain a target information parameter data set;

[0050] Figure 4 It is a schematic diagram of the specific operation flow of the information processing system in the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Example 1

[0053] See also Figure 1-4 , an information collection and analysis system based on big data, including an information processing system, the information processing system includes a data preprocessing system, a data analysis system, a data feedback module, and a cloud database;

[0054] The data preprocessing system includes an information parameter data extraction module, an information parameter data transmission module and an information parameter data screening module, the parameter data extraction module is electrically connected to the information parameter data transmission module and the information parameter data screening module, and the information parameter data screening module is electrically connected to the data analysis system;

[0055] The data analysis system includes an indicator target information confirmation module and an indicator information verification module. The target information confirmation module is electrically connected to the indicator information verification module, and the indicator information verification module is electrically connected to the data feedback module.

[0056] The function of the information parameter data extraction module is to extract information data of the same type and determine the three elements of each information data during the extraction process. The three elements include the time parameter data of the information data, the identity parameter data of the information data, and the security parameter data of the information data.

[0057] When there is an abnormality in the information data security parameter data, the information parameter data will be deleted.

[0058] The function of the information parameter data transmission module is to receive the generated information parameter data in real time, and write the information parameter data to the information parameter data extraction module through the set writing number threshold method and writing time threshold method.

[0059] The function of the target information confirmation module is to determine the target information parameters of the information parameter data after preliminary screening by the information parameter data screening module by inputting at least one indicator information and using the indicator information as a screening indicator, and to form a target information parameter data set for the acquired target information parameter data set.

[0060] The function of the information parameter data screening module is to complete the preliminary screening of the information parameter data collected in the information parameter data extraction module. The specific screening indicators are:

[0061] (1) Eliminate information parameter data that is missing from information parameter data, time parameter data, and identity parameter data;

[0062] (2) Eliminate information parameter data whose content does not meet the basic standards for such information data;

[0063] (3) Information parameter data that does not meet the specified indicators will be eliminated.

[0064] The basic standard of this type of information data at least includes searching for information data of uneven basic feature points and associated feature points diffused from the basic feature points based on the basic feature points of this type of information data.

[0065] The prescribed index includes at least one of text parameter data, image parameter data, data parameter data and audio parameter data.

[0066] The function of the indicator information verification module is to receive the target information parameter data set obtained by the target information confirmation module, extract the target information parameter data in the target information parameter data set according to the set extraction threshold, and compare and verify the text parameter data, image parameter data, data parameter data and audio parameter data categories through the verification and comparison model respectively, and transmit the verified target information parameter data set to the data feedback module.

[0067] The verification model comparison and verification indicator is to set a definite indicator for completing the comparison task. The extraction is performed according to the set extraction threshold. If the verification accuracy of the extracted information parameter is greater than the set verification threshold, the verification is completed. On the contrary, if the verification accuracy of the extracted information parameter is less than the set verification threshold, all information parameter data in the target information parameter data set are compared and verified through the comparison model, and the unqualified information parameter data are eliminated.

[0068] The function of the data feedback module is to receive the target information parameter data set after verification by the indicator information verification module, compress the target information parameter data set, and transmit the compressed target information parameter data set to the interactive terminal through one or more of the 2G, 3G, 4G, 5G networks, WIFI, and Bluetooth.

[0069] The cloud database and the data feedback module both have a query function, and the query function has a retrieval function.

[0070] The function of the cloud database is to store the information parameter data obtained by the information parameter data screening module in the information processing system, and the storage standard in the cloud database is that the same data category has the information parameter data collection time and method, etc.

[0071] Example 2

[0072] See also Figure 2 , an information collection and analysis system based on big data, including an information parameter data transmission module, the information parameter data transmission module writes the information parameter data to the information parameter data extraction module. The specific steps are:

[0073] S1: setting corresponding number thresholds and time thresholds for the write number threshold mode and the write time threshold;

[0074] S2.1.1: When the information parameter data transmission module stores information parameter data and reaches the write times threshold;

[0075] S2.1.2: writing the information parameter data stored in the transmission module into the information parameter data extraction module;

[0076] S2.2.1: When the information parameter data transmission module stores the information parameter data and reaches the write time threshold;

[0077] S2.2.2: Write the information parameter data stored in the transmission module into the information parameter data extraction module.

[0078] In S2.2.2, after the information parameter data transmission module writes the information parameter data into the information parameter data extraction module by writing the time threshold, the write count threshold is reset.

[0079] Example 3

[0080] See also Figure 3 , an information collection and analysis system based on big data, including an indicator target information confirmation module, the indicator target information confirmation module obtains the target information parameter data set in the following specific steps:

[0081] Step 1: Determine the categories of text parameter data, image parameter data, data parameter data and audio parameter data;

[0082] Step 2: Setting at least one target information parameter for different information parameter data;

[0083] Step 3: Screening the text parameter data and the data parameter data with at least one determined target information parameter to determine the target information parameter data;

[0084] Step 4: construct a comparison model for the image parameter data, standardize the image parameter data, compare it with at least one target information, and determine the target information parameter data for the image parameter data with high adaptability;

[0085] Step 5: for the audio parameter data, perform standardization processing on the audio parameter data, compare it with at least one target information, obtain the audio parameter data that has been successfully compared, and determine the target information parameter data;

[0086] Step 6: After determining the target information parameter data, aggregate them to obtain the target information parameter data set.

[0087] In step 4, the image parameter data is standardized by cutting the image parameter data into a number of grids of the same size, and removing the grids from which features cannot be extracted.

[0088] In step five, the standardization process of the audio parameter data is to perform noise processing on the audio parameter data, and extract text information parameters from the audio parameter data after the noise processing.

[0089] Example 4

[0090] See also Figure 4 , an information collection and analysis system based on big data, including an information processing system, the specific steps of the information processing system to complete the information collection and analysis are:

[0091] (1) First, the information parameter data transmission module receives the information parameter data;

[0092] (2) the information parameter data transmission module writes the information parameter data to the information parameter data extraction module by means of a write count threshold method and a write time threshold method;

[0093] (3) The information parameter data extraction module extracts information data of the same type and transmits it to the information parameter data screening module;

[0094] (4) The information parameter data screening module performs preliminary screening of the information parameter data collected by the information parameter data extraction module;

[0095] (5) The target information confirmation module receives the information parameter data after screening, confirms it, and obtains the target information parameter data set;

[0096] (6) The indicator information verification module receives the target information parameter data set and completes verification of the target information parameter data set;

[0097] (7) The data feedback module exchanges the verified target information parameter data set to the interactive terminal.

[0098] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit thereof, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An information collection and analysis system based on big data, including an information processing system, characterized in that: The information processing system includes a data preprocessing system, a data analysis system, a data feedback module, and a cloud database; The data preprocessing system includes an information parameter data extraction module, an information parameter data transmission module and an information parameter data screening module, wherein the parameter data extraction module is electrically connected to the information parameter data transmission module and the information parameter data screening module, and the information parameter data screening module is electrically connected to the data analysis system; The data analysis system comprises an indicator target information confirmation module and an indicator information verification module, wherein the target information confirmation module is electrically connected to the indicator information verification module, and the indicator information verification module is electrically connected to the data feedback module.

2. The information collection and analysis system based on big data according to claim 1, characterized in that: The function of the information parameter data extraction module is to extract information data of the same type, and determine three elements of each information data during the extraction process, wherein the three elements include time parameter data of the information data, identity parameter data of the information data, and security parameter data of the information data; The function of the information parameter data transmission module is to receive the generated information parameter data in real time, and write the information parameter data to the information parameter data extraction module through the set writing number threshold method and writing time threshold method.

3. The information collection and analysis system based on big data according to claim 2 is characterized in that: The specific steps of the information parameter data transmission module writing the information parameter data to the information parameter data extraction module are as follows: S1: setting corresponding number thresholds and time thresholds for the write number threshold mode and the write time threshold; S2.1.1: When the information parameter data transmission module stores information parameter data and reaches the write times threshold; S2.1.2: writing the information parameter data stored in the transmission module into the information parameter data extraction module; S2.2.1: When the information parameter data transmission module stores the information parameter data and reaches the write time threshold; S2.2.2: Write the information parameter data stored in the transmission module into the information parameter data extraction module.

4. The information collection and analysis system based on big data according to claim 1 is characterized by: The function of the information parameter data screening module is to complete the preliminary screening of the information parameter data collected in the information parameter data extraction module. The specific screening indicators are: (1) Eliminate information parameter data that is missing from information parameter data, time parameter data, and identity parameter data; (2) Eliminate information parameter data whose content does not meet the basic standards for such information data; (3) Eliminate information parameter data that does not meet the specified indicators; The specified indicators include at least one of text parameter data, image parameter data, data parameter data and audio parameter data.

5. The information collection and analysis system based on big data according to claim 1 is characterized by: The function of the target information confirmation module is to determine the target information parameters of the information parameter data preliminarily screened by the information parameter data screening module by inputting at least one indicator information and using the indicator information as a screening indicator, and to form a target information parameter data set for the acquired target information parameter data set.

6. The information collection and analysis system based on big data according to claim 1 is characterized by: The specific steps for the indicator target information confirmation module to obtain the target information parameter data set are: Step 1: Determine the categories of text parameter data, image parameter data, data parameter data and audio parameter data; Step 2: Setting at least one target information parameter for different information parameter data; Step 3: Screening the text parameter data and the data parameter data with at least one determined target information parameter to determine the target information parameter data; Step 4: construct a comparison model for the image parameter data, standardize the image parameter data, compare it with at least one target information, and determine the target information parameter data for the image parameter data with high adaptability; Step 5: for the audio parameter data, perform standardization processing on the audio parameter data, compare it with at least one target information, obtain the audio parameter data that has been successfully compared, and determine the target information parameter data; Step 6: After determining the target information parameter data, aggregate them to obtain the target information parameter data set.

7. The information collection and analysis system based on big data according to claim 1 is characterized by: The function of the indicator information verification module is to receive the target information parameter data set obtained by the target information confirmation module, extract the target information parameter data in the target information parameter data set according to the set extraction threshold, and compare and verify the text parameter data, image parameter data, data parameter data and audio parameter data categories through the verification and comparison model respectively, and transmit the verified target information parameter data set to the data feedback module.

8. The information collection and analysis system based on big data according to claim 1 is characterized by: The function of the data feedback module is to receive the target information parameter data set after verification by the indicator information verification module, compress the target information parameter data set, and transmit the compressed target information parameter data set to the interactive terminal through one or more of the 2G, 3G, 4G, 5G networks, WIFI, and Bluetooth.

9. The information collection and analysis system based on big data according to claim 1, characterized in that: The cloud database is used to store the information parameter data acquired by the information parameter data screening module in the information processing system.

10. The information collection and analysis system based on big data according to claim 1, characterized in that: The specific steps of the information processing system to complete information collection and analysis are: (1) First, the information parameter data transmission module receives the information parameter data; (2) the information parameter data transmission module writes the information parameter data to the information parameter data extraction module by means of a write count threshold method and a write time threshold method; (3) The information parameter data extraction module extracts information data of the same type and transmits it to the information parameter data screening module; (4) The information parameter data screening module performs preliminary screening of the information parameter data collected by the information parameter data extraction module; (5) The target information confirmation module receives the information parameter data after screening, confirms it, and obtains the target information parameter data set; (6) The indicator information verification module receives the target information parameter data set and completes verification of the target information parameter data set; (7) The data feedback module exchanges the verified target information parameter data set to the interactive terminal.

Images